Data processing method and device of data service system based on read-write separation architecture

Abstract

The embodiment of the invention provides a data processing method and device of a data service system based on a read-write separation architecture, and the method comprises the steps: receiving a data reading request sent by a client, constructing a query plan corresponding to the data reading request, and generating a data direct reading instruction, sending the query plan and the data direct reading indication to a query server of the data service system, wherein the data direct reading indication is used for indicating to read data in a write-in server of the data service system; and responding to the data reading request according to the data obtained by executing the query plan and the data direct reading instruction by the query server. According to the invention, the data real-time performance of the data service system based on the read-write separation architecture can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and device for a data service system based on a read-write separation architecture. Background Art

[0002] In traditional data service systems based on read-write separation architecture, data will only be actually written to the data service system when the amount of data to be written reaches the threshold amount of data, or the time from the last write time reaches the threshold time. The user's latest written data often needs to wait for a period of time before it can be read. Therefore, traditional data service systems based on read-write separation architecture cannot meet users' requirements for real-time data in high-speed read and write scenarios. Summary of the invention

[0003] In view of this, an embodiment of the present application provides a data processing method and device for a data service system based on a read-write separation architecture, which is used to improve the data real-time performance of the data service system based on the read-write separation architecture.

[0004] In order to achieve the above purpose, the embodiments of the present application provide the following technical solutions:

[0005] In a first aspect, an embodiment of the present application provides a data processing method of a data service system based on a read-write separation architecture, comprising:

[0006] Receive a data read request sent by a client, construct a query plan corresponding to the data read request and generate a data direct read instruction, and send the query plan and the data direct read instruction to a query server of the data service system, wherein the data direct read instruction is used to instruct to read data in a write server of the data service system;

[0007] The data read request is responded to according to the data obtained by the query server executing the query plan and executing the data direct read instruction.

[0008] As an optional implementation of the embodiment of the present application, the direct data read indication is used to indicate reading data in the variable memory table of the write server; the variable memory table is a memory table in the write server used to store data to be written to the storage server of the data service system; the method also includes: executing the direct data read indication through the query server, and reading the target data from the variable memory table in the write server.

[0009] As an optional implementation of the embodiment of the present application, the method further includes:

[0010] When the data service system fails and restarts, the variable memory table of the write server is restored, and before the restoration of the variable memory table of the write server is completed, data reading service is prohibited to be provided to the client.

[0011] As an optional implementation of an embodiment of the present application, the direct data read indication is used to indicate reading data in the immutable memory table of the write server; the immutable memory table is a memory table used to store data that has been written to the storage server of the data service system; the method also includes: executing the direct data read indication through the query server to read the target data from the immutable memory table in the write server.

[0012] As an optional implementation of the embodiment of the present application, the direct data read indication carries identification information of the target metadata, which is used to indicate reading the data in the immutable memory table of the write server that is written to the storage server of the data service system after the data corresponding to the target metadata; the immutable memory table is a memory table used to store data written to the storage server of the data service system; the target metadata is the metadata with the latest generation time in the local metadata cache; the method also includes: executing the direct data read indication through the query server, and reading the target data written to the storage server of the data service system after the data corresponding to the target metadata from the immutable memory table in the write server.

[0013] As an optional implementation of the embodiment of the present application, the method further includes:

[0014] Receiving, by the write server, low-water mark metadata sent by the metadata server of the data service system, and deleting data corresponding to the low-water mark metadata in the immutable memory table and data written to the storage server before the data corresponding to the low-water mark metadata;

[0015] The low-water mark metadata is metadata with the latest generation time in the invalid metadata set, and the invalid metadata set is a set consisting of metadata acquired by each data reading server of the data service system.

[0016] As an optional implementation of the embodiment of the present application, constructing a query plan corresponding to the data read request includes:

[0017] Querying a local metadata cache to obtain metadata, wherein the metadata cache is used to store metadata obtained by periodically polling a metadata server in the data service system;

[0018] A corresponding query plan is constructed according to the metadata and the data read request.

[0019] As an optional implementation of the embodiment of the present application, the method further includes:

[0020] Receive the data to be written sent by the client;

[0021] Writing the data to be written into the variable memory table of the write server;

[0022] After the writing is successful, a writing success notification message is sent to the client.

[0023] As an optional implementation of the embodiment of the present application, the method further includes:

[0024] When the data in the mutable memory table meets the conditions for writing into the storage server of the data service system, converting the mutable memory table into an immutable memory table, writing the data in the immutable memory table into the storage server, and obtaining metadata corresponding to the data in the immutable memory table;

[0025] The metadata corresponding to the data in the immutable memory table is saved to the metadata server of the data service system.

[0026] In a second aspect, an embodiment of the present application provides a data processing device of a data service system based on a read-write separation architecture, comprising:

[0027] A receiving unit, used for receiving a data reading request sent by a client;

[0028] A processing unit, configured to construct a query plan corresponding to the data read request and generate a data direct read instruction, wherein the data direct read instruction is used to instruct reading data in a write server of the data service system;

[0029] A sending unit, configured to send the query plan and the data direct reading instruction to a query server of the data service system;

[0030] A response unit is used to respond to the data read request according to the data obtained by the query server executing the query plan and executing the data direct read instruction.

[0031] As an optional implementation of the embodiment of the present application, the data direct read indication is used to indicate reading data in a variable memory table of the write server; the variable memory table is a memory table in the write server for storing data to be written to the storage server of the data service system;

[0032] The processing unit is further configured to execute the data direct read instruction through the query server to read the target data from the variable memory table in the write server.

[0033] As an optional implementation of the embodiment of the present application, the processing unit is also used to restore the variable memory table of the write server when the data service system fails and restarts, and prohibit providing data reading service to the client before the recovery of the variable memory table of the write server is completed.

[0034] As an optional implementation of the embodiment of the present application, the data direct read indication is used to indicate reading the data in the immutable memory table of the write server; the immutable memory table is a memory table for storing data that has been written to the storage server of the data service system;

[0035] The processing unit is further configured to execute the direct data read instruction through the query server to read target data from the immutable memory table in the write server.

[0036] As an optional implementation of the embodiment of the present application, the data direct read indication carries identification information of the target metadata, which is used to indicate reading the data in the immutable memory table of the write server that is written to the storage server of the data service system after the data corresponding to the target metadata; the immutable memory table is a memory table for storing data written to the storage server of the data service system; the target metadata is the metadata with the latest generation time in the local metadata cache;

[0037] The processing unit is further used to execute the data direct read instruction through the query server, and read the target data written to the storage server of the data service system after the data corresponding to the target metadata from the immutable memory table in the write server.

[0038] As an optional implementation of the embodiment of the present application, the processing unit is further used to receive the low water mark metadata sent by the metadata server of the data service system through the write server, and delete the data corresponding to the low water mark metadata in the immutable memory table and the data written to the storage server before the data corresponding to the low water mark metadata;

[0039] The low-water mark metadata is metadata with the latest generation time in the invalid metadata set, and the invalid metadata set is a set consisting of metadata acquired by each data reading server of the data service system.

[0040] As an optional implementation of the embodiment of the present application, the processing unit is specifically used to query the local metadata cache to obtain metadata, and build a corresponding query plan according to the metadata and the data reading request;

[0041] The metadata cache is used to store metadata obtained by periodically polling a metadata server in the data service system.

[0042] As an optional implementation of the embodiment of the present application,

[0043] The receiving unit is further used to receive the data to be written sent by the client;

[0044] The processing unit is further used to write the data to be written into the variable memory table of the write server;

[0045] The sending unit is further configured to send write success notification information to the client after the write is successful.

[0046] As an optional implementation of the embodiment of the present application, the processing unit is also used to convert the variable memory table into an immutable memory table when the data in the variable memory table meets the conditions for writing to the storage server of the data service system, write the data in the immutable memory table to the storage server, obtain metadata corresponding to the data in the immutable memory table, and save the metadata corresponding to the data in the immutable memory table to the metadata server of the data service system.

[0047] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to enable the electronic device to implement the data processing method of the data service system based on the read-write separation architecture described in any of the above embodiments when executing the computer program.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which, when the computer program is executed by a computing device, enables the computing device to implement any of the above-mentioned data processing methods of the data service system based on the read-write separation architecture.

[0049] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to implement any of the above-mentioned data processing methods of a data service system based on a read-write separation architecture.

[0050] When receiving a data read request sent by a client, the data processing method of the data service system based on the read-write separation architecture provided in the embodiment of the present application constructs a query plan corresponding to the data read request, and generates a data direct read instruction for indicating reading the data in the write server of the data service system; then responds to the data read request according to the data obtained by executing the query plan and the data direct read instruction on the query server. Since the data processing method of the data service system based on the read-write separation architecture provided in the embodiment of the present application generates a data direct read instruction when receiving a data read request sent by a client, and responds to the data read request according to the data obtained by executing the data direct read instruction, and the data direct read instruction is used to indicate reading the data in the write server of the data service system, as long as the data written to the write server of the data service system can be read by the client, there is no need to wait for the data to be written to the data storage server of the data service system before it can be read by the client. Therefore, the embodiment of the present application uses the write service as the real source of information, and responds to the data read request based on the data situation of the write server, thereby improving the real-time performance of the data of the data service system based on the read-write separation architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0053] Figure 1 A schematic diagram of the structure of a data service system provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of the time nodes of the data writing process of the data service system provided in an embodiment of the present application;

[0055] Figure 3 One of the step flow charts of the data processing method of the data service system based on the read-write separation architecture provided in an embodiment of the present application;

[0056] Figure 4 The second step flow chart of the data processing method of the data service system based on the read-write separation architecture provided in the embodiment of the present application;

[0057] Figure 5The third step flow chart of the data processing method of the data service system based on the read-write separation architecture provided in the embodiment of the present application;

[0058] Figure 6 A fourth step flow chart of a data processing method of a data service system based on a read-write separation architecture provided in an embodiment of the present application;

[0059] Figure 7 A schematic diagram of the structure of a data processing device based on a read-write separation data service system provided in an embodiment of the present application;

[0060] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0062] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein. Obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0063] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. In addition, in the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more.

[0064] The embodiments of the present application relate to a data service system based on a read-write separation architecture. The data service system may be a distributed data system for hybrid serving and analytical processing (HSAP), or may be other read-write separation architecture systems based on LSM-Tree. The following first describes a data service system based on a read-write separation architecture in some embodiments. Figure 1As shown, the data service system based on the read-write separation architecture includes: a write server 11, a metadata server 12, a query server 13, a read server 14 and a storage server 15. Among them, the write server 11, the query server 13, and the read server 14 may include one or more to jointly perform data writing tasks and data reading tasks.

[0065] The write server 11 is responsible for the real-time writing of data. The internal implementation of the write server 11 is similar to the Log Structured Merge Tree (LSM Tree). The data to be written sent by the client is first written into the Write Ahead Log (WAL) for persistence, and then the persisted data to be written is written into the mutable memtable. When the size of the mutable memtable reaches the threshold size, or the existence time of the mutable memtable reaches the threshold time, the data in the mutable memtable is converted into a storage file in a preset format, for example: a rowset file, the mutable memtable is converted into an immutable memtable, the data in the immutable memtable is written to the storage server 15, and the metadata corresponding to the data in the immutable memtable is obtained, and the metadata corresponding to the data in the immutable memtable is written to the metadata server 12. The write server 11 also receives a direct data read request sent by the query server 13, and sends the data in the mutable memtable and / or immutable memtable to the query server 13 in response to the direct data read request. The write server 11 also receives the low water mark metadata sent by the metadata server 12, and cleans up the data in the immutable memory table according to the low water mark metadata.

[0066] The metadata server 12 is responsible for the metadata (Meta) information of the data service system. The metadata information includes: the schema of the library table, the storage path and routing information of the data, the operating status of each device of the data service system, etc. The metadata server 12 is used to receive the metadata sent by the write server 11, and save the metadata sent by the write server 11. The metadata server 12 is also used to receive the polling request sent by the read server 14, and in response to the polling request sent by the read server 14, send the saved metadata to the read server 14. The metadata server 12 is also used to obtain the low water mark metadata according to the metadata currently held by each write server 11, and send the low water mark metadata to each write server 11.

[0067] The read server 14 is used to construct a query plan (Query Plan) corresponding to the data read request and generate a direct data read instruction. The read server 14 will poll the metadata stored in the metadata server 12 at a preset time period, and generate a metadata cache (MetaCache) based on the metadata obtained from the metadata server 12. When receiving a data read request sent by the client, the read server 14 will parse the data read request, build a corresponding query plan based on the metadata cache, and send the data query plan to the query server 13. In addition, the read server 14 will also send a direct data read instruction to the query server 13 to indicate the reading of data in the write server 11.

[0068] The query server 13 is used to provide data reading services. The query server 13 is stateless and executes the query plan and data direct reading instructions sent by the read server 14, and obtains the target data that the client wants to read according to the data obtained from the storage server 15 by executing the query plan and the data obtained from the write server 11 by executing the data direct reading instructions, and sends the target data that the client wants to read to the read server 14, and sends the data that the client wants to read to the read server 14.

[0069] Reference Figure 2 As shown in the figure, the data writing process of the data service system based on the read-write separation architecture and the time nodes corresponding to each step of the process include:

[0070] Time node A: Data writing starts. In some embodiments, the data writing method is to write data by writing a remote procedure call (RPC), and the start of data writing is the start of writing the RPC.

[0071] Time node B: Write the data to be written into the WAL cache.

[0072] Time node C: Transfer the data to be written from the WAL to the mutable memory table and return a write success notification.

[0073] Time node D: The mutable memory table is converted to an immutable memory table, the data in the immutable memory table is refreshed as a storage file, the data in the immutable memory table is written to the storage server, the metadata of the data written to the storage server is obtained, and the metadata of the data written to the storage server is registered with the metadata server.

[0074] Time node E: The read server polls and reads the latest metadata written to the metadata server. Since the read server uses polling to synchronize metadata from the metadata server, it may happen that the metadata of some data has been written to the metadata server, but the read server has not yet synchronized this part of metadata.

[0075] Time node F: After the data in the immutable content table is written to the storage server, the immutable memory table will still exist for a period of time to provide read servers.

[0076] Time node G: The data service system has limited memory for writing data to the server, and cannot permanently save the immutable memory table. Therefore, after each read server has obtained the metadata corresponding to the data in the immutable memory table, the memory occupied by the data in the immutable memory table can be released. At this time, the query server can read the data that has been released from the immutable memory table from the storage server according to the corresponding metadata.

[0077] The present application embodiment also provides a data processing method for a data service system based on a read-write separation architecture. The execution subject of the method can be a data service system based on a read-write separation architecture. Figure 3 As shown, the data processing method of the data service system based on the read-write separation architecture includes the following steps:

[0078] S301: Receive a data reading request sent by a client.

[0079] In some embodiments, identification information of the data to be read may be carried in the data read request, so that the data service system can determine the data to be read according to the identification information of the data.

[0080] In other embodiments, the client may carry condition information of the data to be read in the data read request, so that the data service system can determine the data to be read according to the condition information of the data.

[0081] S302: Construct a query plan corresponding to the data read request and generate a data direct read instruction.

[0082] The data direct reading instruction is used to instruct reading data written into the server of the data service system.

[0083] The query plan corresponding to a data read request refers to the data read solution generated by the data service system based on the data read request. The query plan describes a series of steps taken by the data service system to execute the data read request, which steps may include which tables to read data from, which indexes to use, how to perform join operations, how to sort, etc.

[0084] In some embodiments, the data service system includes multiple write servers, and different data will be allocated to different write servers for writing according to an allocation algorithm. When the data service system includes multiple write servers, the target write server for writing the data that the data read request wants to read can be determined first, and the data direct read indication carries the identification information of the target write server to indicate the data in the target write server used to indicate reading the data of the data service system.

[0085] In some embodiments, the data direct read instruction is used to instruct reading data in a variable memory table of the write server, wherein the variable memory table is a memory table in the write server for storing data to be written into the storage server of the data service system.

[0086] When the data direct read instruction is used to instruct reading data in the variable memory table of the write server, the method further includes: executing the data direct read instruction through the query server to read target data from the variable memory table in the write server.

[0087] The direct data read indication described in the above embodiment is used to indicate reading the data in the variable memory table of the write server, and the data processing method provided in the embodiment of the present application will also execute the direct data read indication through the query server to read the target data from the variable memory table in the write server. Therefore, even if the data is only written to the variable memory table of the write server and has not yet been written to the storage server of the data service system, this part of the data can also be read by the client. Therefore, the above embodiment can improve the real-time performance of data in the data service system based on the read-write separation architecture.

[0088] In some embodiments, the data direct read instruction is used to instruct reading data in the immutable memory table of the write server, wherein the immutable memory table is a memory table for storing data written into the storage server of the data service system.

[0089] When the direct data read indication is used to indicate reading data in the immutable memory table of the write server, the method further includes: executing the direct data read indication through the query server to read target data from the immutable memory table in the write server.

[0090] The direct data read indication described in the above embodiment is used to indicate the reading of data in the immutable memory table of the write server, and the data processing method provided in the embodiment of the present application will also execute the direct data read indication through the query server to read the target data from the immutable memory table in the write server. Therefore, even if the metadata of the target data has not yet been synchronized to the metadata cache of the read server, this part of the data can be obtained by the data query server and then read by the client. Therefore, the above embodiment can further improve the real-time performance of data in the data service system based on the read-write separation architecture.

[0091] In some embodiments, the data direct read instruction carries identification information of the target metadata, which is used to indicate reading the data in the immutable memory table of the write server that is written to the storage server of the data service system after the data corresponding to the target metadata. The immutable memory table is a memory table used to store data written to the storage server of the data service system; the target metadata is the metadata with the latest generation time in the local metadata cache.

[0092] When the data direct read indication is used to indicate reading the data in the immutable memory table of the write server that is written to the storage server of the data service system after the data corresponding to the target metadata, the method further includes: executing the data direct read indication through the query server, reading the target data from the immutable memory table in the write server and writing the target data to the storage server of the data service system after the data corresponding to the target metadata.

[0093] Since the data corresponding to the target metadata and the target data written to the storage server before the data corresponding to the target metadata can be obtained from the storage server according to the query plan, compared to directly reading all the data in the immutable memory table in the write server, the above embodiment only needs to read the target data written to the storage server after the data corresponding to the target metadata from the immutable memory table in the write server. Therefore, the embodiment of the present application can reduce the data read from the write server while ensuring the reading of all target data, thereby saving the data transmission overhead between the query server and the data write server of the data service system.

[0094] In some embodiments, the data direct read instruction is used to indicate reading data in a variable memory table and data in an immutable memory table of the write server. The variable memory table is a memory table in the write server used to store data to be written to the storage server of the data service system; the immutable memory table is a memory table used to store data that has been written to the storage server of the data service system.

[0095] When the data direct read indication is used to indicate reading data in the variable memory table and the immutable memory table of the write server, the method further includes: executing the data direct read indication through the query server to read target data from the variable memory table and the immutable memory table in the write server.

[0096] Based on the above-mentioned technical effects of respectively reading the target data from the variable memory table and the immutable memory table in the write server, reading the target data from the variable memory table and the immutable memory table in the write server can not only enable the data that has only been written to the variable memory table of the write server and has not yet been written to the storage server of the data service system to be read by the client, but also enable the data whose corresponding metadata has not yet been synchronized to the read server to be read by the client. Therefore, the above-mentioned embodiment can more comprehensively improve the real-time performance of data in the data service system based on the read-write separation architecture.

[0097] S303: Send the query plan and the data direct reading instruction to the query server of the data service system.

[0098] Correspondingly, the query server of the data service system receives the query plan and the data direct reading instruction.

[0099] S304: Respond to the data read request according to the data obtained by the query server executing the query plan and executing the data direct read instruction.

[0100] In some embodiments, the query server executes the direct data read indication, including: the query server sends a direct data read request to the write server in response to the direct data read indication, and receives the requested data sent by the write server in the direct data read request.

[0101] When receiving a data read request sent by a client, the data processing method of the data service system based on the read-write separation architecture provided in the embodiment of the present application constructs a query plan corresponding to the data read request, and generates a data direct read instruction for indicating reading the data in the write server of the data service system; then responds to the data read request according to the data obtained by executing the query plan and the data direct read instruction on the query server. Since the data processing method of the data service system based on the read-write separation architecture provided in the embodiment of the present application generates a data direct read instruction when receiving a data read request sent by a client, and responds to the data read request according to the data obtained by executing the data direct read instruction, and the data direct read instruction is used to indicate reading the data in the write server of the data service system, as long as the data written to the write server of the data service system can be read by the client, there is no need to wait for the data to be written to the data storage server of the data service system before it can be read by the client. Therefore, the embodiment of the present application uses the write service as the real source of information, and responds to the data read request based on the data situation of the write server, thereby improving the real-time performance of the data of the data service system based on the read-write separation architecture.

[0102] The present application also provides a data processing method for a data service system based on a read-write separation architecture. The data processing method is used to write data into the data service system. Figure 4 As shown, the data processing method of the data service system based on the read-write separation architecture includes the following steps:

[0103] S401: Receive data to be written sent by the client.

[0104] S402: Write the data to be written into the variable memory table of the write server.

[0105] In some embodiments, writing the data to be written into the variable memory table of the write server includes: first writing the data to be written into the write-ahead log (WAL) of the write server, and then transferring the data in the write-ahead log to the variable memory table of the write server.

[0106] S403: After the writing is successful, a writing success notification message is sent to the client.

[0107] Compared to the related art, after the data is persisted in the write-ahead log, the write server will send a confirmation message to the client to indicate that the writing of the data has been completed. However, writing data from the write-ahead log to the variable memory table is an asynchronous process. The above embodiment will send a write success notification message to the client only after the data to be written is successfully written to the variable memory table, and the data direct read indication is used to indicate reading the data in the variable memory table of the write server. Therefore, the above embodiment can provide a guarantee that data can be read as soon as it is written.

[0108] S404: When the data in the variable memory table meets the condition of being written into the storage server of the data service system, convert the variable memory table into an immutable memory table.

[0109] In some embodiments, the condition that the data in the variable memory table satisfies the requirement of being written into the storage server of the data service system is that the amount of data in the variable memory table reaches a threshold amount of data.

[0110] In some other embodiments, the condition that the data in the variable memory table satisfies the writing into the storage server of the data service system is that the length of time that the data exists in the variable memory table reaches a threshold length of time.

[0111] S405: Write the data in the immutable memory table into the storage server, and obtain metadata corresponding to the data in the immutable memory table.

[0112] In some embodiments, the data in the immutable memory table is written to the storage server, and metadata corresponding to the data in the immutable memory table is obtained, including: the write server sends the data in the immutable memory table to the storage server, the storage server saves the data sent by the write server, and returns the metadata of the saved data.

[0113] S406: Save the metadata corresponding to the data in the immutable memory table to the metadata server of the data service system.

[0114] The present application also provides a data processing method for a data service system based on a read-write separation architecture. The data processing method is used to write data into the data service system. Figure 5 As shown, the data processing method of the data service system based on the read-write separation architecture includes the following steps:

[0115] S501. The client sends data to be written to a write server of a data service system.

[0116] Correspondingly, the write server of the data service system receives the data to be written sent by the client.

[0117] S502: The write server of the data service system writes the data to be written into a variable memory table.

[0118] S503: After the data to be written is successfully written into the variable memory table, the write server of the data service system sends a write success notification message to the client.

[0119] Correspondingly, the client receives write success notification information sent by the write server of the data service system.

[0120] S504: When the data in the variable memory table meets the condition of being written into the storage server of the data service system, the write server of the data service system converts the variable memory table into an immutable memory table.

[0121] S505. The write server of the data service system sends the data to be written in the immutable memory table to the storage server of the data service system.

[0122] Correspondingly, the storage server of the data service system receives the data to be written sent by the write server of the data service system.

[0123] S506: The storage server of the data service system saves the data to be written and generates metadata corresponding to the data to be written.

[0124] S507: The storage server of the data service system sends metadata corresponding to the data to be written to the write server of the data service system.

[0125] Correspondingly, the write server of the data service system receives the metadata corresponding to the to-be-written data sent by the storage server of the data service system.

[0126] S508: The write server of the data service system sends metadata corresponding to the data to be written to the metadata server of the data service system.

[0127] Correspondingly, the metadata server of the data service system receives the metadata corresponding to the data to be written sent by the write server of the data service system.

[0128] S509: The metadata server of the data service system saves the metadata corresponding to the data to be written.

[0129] The present application also provides a data processing method for a data service system based on a read-write separation architecture. The data processing method is used to read data from a data service system. Figure 6 As shown, the data processing method of the data service system based on the read-write separation architecture includes the following steps:

[0130] S601: The client sends a data read request to a read server of a data service system.

[0131] Correspondingly, the reading server of the data service system receives the data reading request sent by the client.

[0132] S602: The read server of the data service system generates a query plan corresponding to the data read request according to the metadata cache (Meta Cache), and generates a data direct read instruction.

[0133] The data direct reading instruction is used to instruct reading data written into the server of the data service system.

[0134] In some embodiments, the reading server polls the metadata stored in the metadata server at a preset time period, and generates the metadata cache based on the metadata obtained from the metadata server. Therefore, the metadata cache of the reading server is the metadata cache generated based on the latest metadata obtained from the metadata server.

[0135] S603: The read server of the data service system sends the query plan and the data direct read instruction to the query server of the data service system.

[0136] Correspondingly, the query server of the data service system receives the query plan and the data direct reading instruction sent by the reading server of the data service system.

[0137] S604: The query server of the data service system sends a first data read command to the storage server of the data service system in response to the query plan.

[0138] Correspondingly, the storage server of the data service system receives the first data reading command sent by the query server of the data service system.

[0139] S605: The storage server of the data service system sends the data corresponding to the first data reading command to the query server of the data service system.

[0140] Correspondingly, the query server of the data service system receives the data corresponding to the first data reading command sent by the storage server of the data service system.

[0141] S606: The query server of the data service system sends a second data read command to the write server of the data service system in response to the data direct read instruction.

[0142] Correspondingly, the write server of the data service system receives the second data read command sent by the query server of the data service system.

[0143] S607: The write server of the data service system sends the data corresponding to the second data read command to the query server of the data service system.

[0144] Correspondingly, the query server of the data service system receives the data corresponding to the second data read command sent by the write server of the data service system.

[0145] S608. The query server of the data service system acquires target data according to the data obtained by executing the query plan and executing the data direct reading instruction.

[0146] The target data is the data that the data read request wants to read.

[0147] S609: The query server of the data service system sends the target data to the read server of the data service system.

[0148] Correspondingly, the reading server of the data service system receives the target data sent by the query server of the data service system.

[0149] S610: The read server of the data service system sends the target data to the client.

[0150] Correspondingly, the client receives the target data sent by the reading server of the data service system.

[0151] In some embodiments, the data processing method provided in the embodiments of the present application further includes:

[0152] Receiving, by the write server, low-water mark metadata sent by the metadata server of the data service system, and deleting data corresponding to the low-water mark metadata in the immutable memory table and data written to the storage server before the data corresponding to the low-water mark metadata;

[0153] The low-water mark metadata is metadata with the latest generation time in the invalid metadata set, and the invalid metadata set is a set consisting of metadata acquired by each data reading server of the data service system.

[0154] Although the read server will poll the metadata server at a preset period to obtain the metadata stored in the metadata server, the metadata server only stores the metadata of the data submitted by the data write server. From the perspective of data freshness, the read server > metadata server > front-end. Therefore, after the read server synchronizes metadata from the metadata, it can report the version number of the data corresponding to the synchronized metadata to the metadata server through the heartbeat packet, so that the metadata server has a global view at this time, and takes the lowest version of metadata among all read servers as the low-water mark metadata, and sends it to the read server. After the read server obtains the low-water mark metadata from the metadata server, the data below the low-water mark metadata can be directly eliminated to release memory.

[0155] In some embodiments, the metadata server of the data service system may obtain metadata already obtained by each data reading server of the data service system, and determine the low water mark metadata according to the metadata already obtained by each data reading server of the data service system.

[0156] Exemplarily, the following data service system based on the read-write separation architecture includes three read servers (read server 1, read server 2, and read server 3), and the metadata caches of the three read servers are as follows:

[0157] Read server 1: metadata [1, 2];

[0158] Read server 2: metadata [1, 2, 3];

[0159] Read server 2: metadata [1, 2, 3, 4];

[0160] The metadata stored in the metadata server is: metadata [1, 2, 3, 4, 5, 6];

[0161] The storage files stored in the immutable memory table written into the server are: storage files [1, 2, 3, 4, 5, 6, 7] are taken as an example to illustrate the above embodiment.

[0162] After the metadata server determines the metadata in the metadata cache of data read service 1, data read service 2, and data read service 3, it can be determined that the metadata of storage file 1 and storage file 2 have been synchronized to the metadata cache of all write servers of the data service system based on the read-write separation architecture, so it can be determined that the low watermark metadata is metadata 2. After sending metadata 2 to the write server, the write server can delete storage file 1 and storage file 2 in the immutable memory table. After the deletion, the storage files saved in the immutable memory table of the write server are: storage files [3, 4, 5, 6, 7].

[0163] In the above embodiment, if the storage file 3 saved in the immutable memory table written to the server is also deleted, the storage file saved in the immutable memory table written to the server after deletion is: storage file [4, 5, 6, 7]. At this time, if the client reads data by reading server 1, storage file 1 and storage file 2 can be obtained by accessing the storage server based on the metadata cache of reading server 1; storage files 4, 5, 6, and 7 can be obtained by accessing the storage files saved in the immutable memory table written to the server, but storage file 3 cannot be obtained from anywhere, resulting in a data bounce problem.

[0164] In some embodiments, the data processing method of the data service system based on the read-write separation architecture provided in the embodiments of the present application also includes: when the data service system fails and restarts, restoring the variable memory table of the write server, and prohibiting the provision of data reading service to the client before the recovery of the variable memory table of the write server is completed.

[0165] After the write server restarts after a failure, it needs to generate a variable memory table based on the unconsumed data in the WAL during the process of initializing the variable memory table. If data is read during this period, it may be impossible to obtain the data in the variable memory table, resulting in the inability to ensure that the data stored in the variable memory table is read normally. Therefore, during this period, data reading services can be prohibited to the client until the recovery of the variable memory table of the write server is completed.

[0166] In some embodiments, the data processing method of the data service system based on the read-write separation architecture provided in the embodiments of the present application also includes: when the data service system fails and restarts, the immutable memory table of the write server is restored according to the low water mark metadata, and before the restoration of the immutable memory table of the write server is completed, data reading service is prohibited to be provided to the client.

[0167] In some embodiments, restoring the immutable memory table of the write server according to the low water mark metadata includes: obtaining data written after the data corresponding to the low water mark metadata from the storage server, and restoring the immutable memory table according to the data written after the data corresponding to the low water mark metadata.

[0168] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application also provides a data processing device for a data service system based on a read-write separation architecture. This embodiment corresponds to the above method embodiment. For ease of reading, this embodiment will no longer repeat the details of the above method embodiment one by one, but it should be clear that the data processing device in this embodiment can correspond to all the contents in the above method embodiment.

[0169] The present application embodiment provides a data processing device of a data service system based on a read-write separation architecture, Figure 7 is a structural diagram of the data processing device, such as Figure 7 As shown, the data processing device 700 includes:

[0170] A receiving unit 71 is used to receive a data reading request sent by a client;

[0171] A processing unit 72, configured to construct a query plan corresponding to the data read request and generate a data direct read instruction, wherein the data direct read instruction is used to instruct reading data in a write server of the data service system;

[0172] A sending unit 73, configured to send the query plan and the data direct reading instruction to a query server of the data service system;

[0173] The response unit 74 is used to respond to the data read request according to the data obtained by the query server executing the query plan and executing the data direct read instruction.

[0174] As an optional implementation of the embodiment of the present application, the data direct read indication is used to indicate reading data in a variable memory table of the write server; the variable memory table is a memory table in the write server for storing data to be written to the storage server of the data service system;

[0175] The processing unit 72 is further configured to execute the data direct read instruction through the query server to read target data from the variable memory table in the write server.

[0176] As an optional implementation of the embodiment of the present application, the processing unit 72 is also used to restore the variable memory table of the write server when the data service system fails and restarts, and prohibit providing data reading service to the client before the recovery of the variable memory table of the write server is completed.

[0177] As an optional implementation of the embodiment of the present application, the data direct read indication is used to indicate reading the data in the immutable memory table of the write server; the immutable memory table is a memory table for storing data that has been written to the storage server of the data service system;

[0178] The processing unit 72 is further configured to execute the data direct read instruction through the query server to read the target data from the immutable memory table in the write server.

[0179] As an optional implementation of the embodiment of the present application, the data direct read indication carries identification information of the target metadata, which is used to indicate reading the data in the immutable memory table of the write server that is written to the storage server of the data service system after the data corresponding to the target metadata; the immutable memory table is a memory table for storing data written to the storage server of the data service system; the target metadata is the metadata with the latest generation time in the local metadata cache;

[0180] The processing unit 72 is further configured to execute the direct data read instruction through the query server, and read the target data written to the storage server of the data service system after the data corresponding to the target metadata from the immutable memory table in the write server.

[0181] As an optional implementation of the embodiment of the present application, the processing unit 72 is further used to receive the low water mark metadata sent by the metadata server of the data service system through the write server, and delete the data corresponding to the low water mark metadata in the immutable memory table and the data written to the storage server before the data corresponding to the low water mark metadata;

[0182] The low-water mark metadata is metadata with the latest generation time in the invalid metadata set, and the invalid metadata set is a set consisting of metadata acquired by each data reading server of the data service system.

[0183] As an optional implementation of the embodiment of the present application, the processing unit 72 is specifically used to query the local metadata cache to obtain metadata, and build a corresponding query plan according to the metadata and the data read request;

[0184] The metadata cache is used to store metadata obtained by periodically polling a metadata server in the data service system.

[0185] As an optional implementation of the embodiment of the present application,

[0186] The receiving unit 71 is also used to receive the data to be written sent by the client;

[0187] The processing unit 72 is further configured to write the data to be written into the variable memory table of the write server;

[0188] The sending unit 73 is further configured to send a write success notification message to the client after the write is successful.

[0189] As an optional implementation of an embodiment of the present application, the processing unit 72 is also used to convert the variable memory table into an immutable memory table when the data in the variable memory table meets the conditions for writing to the storage server of the data service system, write the data in the immutable memory table to the storage server, obtain metadata corresponding to the data in the immutable memory table, and save the metadata corresponding to the data in the immutable memory table to the metadata server of the data service system.

[0190] The data processing device of the data service system based on the read-write separation architecture provided in the embodiment of the present application can execute the data processing method of the data service system based on the read-write separation architecture provided in any of the above embodiments. Its implementation principle and technical effect are similar and will not be repeated here.

[0191] Based on the same inventive concept, an embodiment of the present application also provides an electronic device. Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the electronic device provided in this embodiment includes: a memory 801 and a processor 802, wherein the memory 801 is used to store a computer program, and the processor 802 is used to execute any data processing method of the data service system based on the read-write separation architecture provided in the above embodiments when executing the computer program.

[0192] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computing device implements any data processing method of a data service system based on a read-write separation architecture provided in the above embodiments.

[0193] Based on the same inventive concept, an embodiment of the present application also provides a computer program product. When the computer program product runs on a computer, the computing device implements any data processing method of a data service system based on a read-write separation architecture provided in the above embodiments.

[0194] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0195] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0196] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0197] Computer readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method for a data service system based on a read-write separation architecture, characterized in that: include: Receive a data read request sent by a client, construct a query plan corresponding to the data read request and generate a data direct read instruction, and send the query plan and the data direct read instruction to a query server of the data service system, wherein the data direct read instruction is used to instruct to read data in a write server of the data service system; The data read request is responded to according to the data obtained by the query server executing the query plan and executing the data direct read instruction.

2. The method according to claim 1, characterized in that The data direct read instruction is used to instruct reading the data in the variable memory table of the write server; The variable memory table is a memory table in the write server for storing data to be written into the storage server of the data service system; The method further includes: executing the direct data read instruction through the query server to read the target data from the variable memory table in the write server.

3. The method according to claim 2, characterized in that The method further comprises: When the data service system fails and restarts, the variable memory table of the write server is restored, and before the restoration of the variable memory table of the write server is completed, data reading service is prohibited to be provided to the client.

4. The method according to claim 1, characterized in that The data direct read instruction is used to instruct reading the data in the immutable memory table of the write server; The immutable memory table is a memory table used to store data written into the storage server of the data service system; the method further comprises: executing the data direct read instruction through the query server to read the target data from the immutable memory table in the write server.

5. The method according to claim 1, characterized in that The data direct reading instruction carries identification information of the target metadata, and is used to instruct to read the data in the immutable memory table of the write server that is written to the storage server of the data service system after the data corresponding to the target metadata; The immutable memory table is a memory table used to store data written into the storage server of the data service system; The target metadata is the metadata with the latest generation time in the local metadata cache; the method also includes: executing the data direct read instruction through the query server, reading from the immutable memory table in the write server the target data written to the storage server of the data service system after the data corresponding to the target metadata.

6. The method according to claim 4 or 5, characterized in that: The method further comprises: Receiving, by the write server, low-water mark metadata sent by the metadata server of the data service system, and deleting data corresponding to the low-water mark metadata in the immutable memory table and data written to the storage server before the data corresponding to the low-water mark metadata; The low-water mark metadata is metadata with the latest generation time in the invalid metadata set, and the invalid metadata set is a set consisting of metadata acquired by each data reading server of the data service system.

7. The method according to claim 1, characterized in that Constructing a query plan corresponding to the data read request, including: Querying a local metadata cache to obtain metadata, wherein the metadata cache is used to store metadata obtained by periodically polling a metadata server in the data service system; A corresponding query plan is constructed according to the metadata and the data read request.

8. The method according to claim 2, characterized in that: The method further comprises: Receive the data to be written sent by the client; Writing the data to be written into the variable memory table of the write server; After the writing is successful, a writing success notification message is sent to the client.

9. The method according to claim 8, characterized in that The method further comprises: When the data in the mutable memory table meets the conditions for being written into the storage server of the data service system, converting the mutable memory table into an immutable memory table, writing the data in the immutable memory table into the storage server, and obtaining metadata corresponding to the data in the immutable memory table; The metadata corresponding to the data in the immutable memory table is saved to the metadata server of the data service system.

10. A data processing device for a data service system based on a read-write separation architecture, characterized in that: include: A receiving unit, used for receiving a data reading request sent by a client; A processing unit, configured to construct a query plan corresponding to the data read request and generate a data direct read instruction, wherein the data direct read instruction is used to instruct reading data in a write server of the data service system; A sending unit, configured to send the query plan and the data direct reading instruction to a query server of the data service system; A response unit is used to respond to the data read request according to the data obtained by the query server executing the query plan and executing the data direct read instruction.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program and the processor is used to enable the electronic device to implement the data processing method of the data service system based on the read-write separation architecture as described in any one of claims 1-9 when executing the computer program.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the data processing method of a data service system based on a read-write separation architecture as described in any one of claims 1-9.

13. A computer program product, characterized in that When the computer program product runs on a computer, the computer implements the data processing method of the data service system based on the read-write separation architecture according to any one of claims 1 to 9.

Images