Data reading and writing method, device and electronic equipment

Abstract

Embodiments of the present application disclose a data read-write method, device and electronic equipment. A specific implementation of the method comprises: receiving a data read-write request for a first database, wherein the first database stores data in a columnar storage format, the data read-write request is used to read and write data in a row store table, the first database comprises a front-end node and a back-end node, the front-end node is used to receive the data read-write request, and the back-end node is used to store data; in the front-end node, the data read-write request is parsed, and a parsing result is sent to the back-end node; in the back-end node, the parsing result is used to perform data read-write in a second database, wherein the first database calls data from the second database, and the second database stores data in a row storage format. The implementation can realize row storage capability in a columnar storage database, and meet high-concurrency point query and high-frequency update capability.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to data reading and writing methods, apparatus, and electronic devices. Background Technology

[0002] OLAP (Online Analytical Processing) systems are a primary application of data warehouse systems. They support complex analytical operations, focusing on decision support for decision-makers and senior management. They can quickly and flexibly perform complex queries on large datasets based on analysts' requirements and provide the results to decision-makers. However, in addition to real-time data warehouse needs, OLAP customers also require high-concurrency point queries and high-frequency updates. Existing data warehouses using columnar storage formats cannot meet these requirements. Summary of the Invention

[0003] This disclosure is provided to briefly introduce the concepts, which will be described in detail in the subsequent Detailed Description section. This disclosure is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, embodiments of this disclosure provide a data read / write method, comprising: receiving a data read / write request for a first database, wherein the first database stores data in a columnar storage format, the data read / write request is used to read and write data in a row-oriented storage table, the first database includes a front-end node and a back-end node, the front-end node is used to receive the data read / write request, and the back-end node is used to store data; in the front-end node, the data read / write request is parsed, and the parsing result is sent to the back-end node; in the back-end node, the parsing result is used to perform data read / write in a second database, wherein the first database retrieves data from the second database, and the second database stores data in a row-oriented storage format.

[0005] Secondly, embodiments of this disclosure provide a data read / write apparatus, comprising: a receiving unit, configured to receive a data read / write request for a first database, wherein the first database stores data in a columnar storage format, the data read / write request is used to read and write data in a row-based storage table, the first database includes a front-end node and a back-end node, the front-end node is used to receive the data read / write request, and the back-end node is used to store data; a sending unit, configured to parse the data read / write request in the front-end node and send the parsing result to the back-end node; and a read / write unit, configured to perform data read / write in a second database in the back-end node using the parsing result, wherein the first database retrieves data from the second database, and the second database stores data in a row-based storage format.

[0006] Thirdly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data read / write method as described in the first aspect.

[0007] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data read / write method as described in the first aspect.

[0008] The data read / write method, apparatus, and electronic device provided in this disclosure receive data read / write requests for a first database storing data in a columnar storage format. The first database includes a front-end node and a back-end node. Then, in the front-end node, the data read / write request is parsed, and the parsing result is sent to the back-end node. Subsequently, in the back-end node, the parsing result is used to perform data read / write operations in a second database storing data in a row-oriented storage format. This method enables row-oriented storage capabilities within a columnar storage database, satisfying high-concurrency point lookups and high-frequency update capabilities. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0010] Figure 1 This is a flowchart of an embodiment of the data read / write method according to this disclosure;

[0011] Figure 2 This is a schematic diagram illustrating an application scenario of updating data in the data read and write method of this disclosure;

[0012] Figure 3 This is a flowchart of yet another embodiment of the data reading and writing method according to this disclosure;

[0013] Figure 4 This is a flowchart of an embodiment of the first read / write path in the data read / write method of this disclosure;

[0014] Figure 5 This is a flowchart of an embodiment of the second read / write path in the data read / write method according to this disclosure;

[0015] Figure 6 This is a schematic diagram of the first read / write path and the second read / write path in the data read / write method of this disclosure;

[0016] Figure 7 This is a schematic diagram illustrating the data reading and writing process performed according to the data reading and writing method disclosed herein;

[0017] Figure 8 This is a schematic diagram of the structure of an embodiment of the data read / write device according to the present disclosure;

[0018] Figure 9 These are exemplary system architecture diagrams to which the various embodiments of this disclosure can be applied;

[0019] Figure 10 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present disclosure. Detailed Implementation

[0020] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0022] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0023] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0024] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0025] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0026] Please refer to Figure 1 The diagram illustrates a flow 100 of an embodiment of a data read / write method according to the present disclosure. The data read / write method includes the following steps:

[0027] Step 101: Receive data read / write requests for the first database.

[0028] In this embodiment, the execution entity of the data read / write method can receive data read / write requests for the first database. Here, the first database typically uses a columnar storage format to store data, and the data read / write requests are typically used to read and write data in a row-based table. These requests are usually SQL (Structured Query Language) statements. SQL is a database language with multiple functions such as data manipulation and data definition. This language is interactive and provides great convenience to users. Database management systems should fully utilize SQL to improve the working quality and efficiency of computer application systems.

[0029] Here, the aforementioned first database typically includes a front-end node (FE) and a back-end node (BE). The front-end node is typically used to receive data read and write requests. In addition, it can also manage metadata, manage client connections, perform query parsing and planning, generate query execution plans, and schedule queries (distributing queries to the BE for execution). The data in the first database is typically stored in the back-end node. Furthermore, the back-end node can also be used for executing query execution plans and managing replicas.

[0030] The first database mentioned above could be Doris, a distributed columnar storage and analysis system that supports real-time querying and analysis.

[0031] Using column-oriented tables (i.e., columnar storage format) for data storage is suitable for inserting large amounts of data at once, statistical analysis queries (e.g., multiple group and join scenarios), and ad-hoc queries (scenarios where the query condition columns are uncertain and the row storage cannot determine the index), thereby providing better data compression ratio and better index performance.

[0032] Using row-based tables (i.e., row-oriented storage format) for data storage is suitable for scenarios with frequent data updates, point queries (returning few records, simple queries based on indexes), or scenarios with many additions, deletions, and modifications, thus providing better point updates and point queries.

[0033] As an example, users can create row storage tables or write data to existing row storage tables. Users need to specify the engine (i.e., database) as row storage, i.e., ENGINE=ROW_STORE. At this time, the aforementioned execution entity can determine from the received data write request that data should be written to the row storage table. After that, the data can be encoded and stored in a second database that stores data in row storage format.

[0034] Step 102: In the front-end node, the data read / write request is parsed, and the parsing result is sent to the back-end node.

[0035] In this embodiment, the execution entity can parse the data read / write request in the front-end node of the first database and send the parsing result to the back-end node of the first database.

[0036] Specifically, the aforementioned front-end node can parse the aforementioned data read / write requests, determine the storage location of the data to be read / written in the data table of the aforementioned back-end node, and generate a data query task, which can then send the data storage location and the data query task to the aforementioned back-end node.

[0037] Step 103: In the backend node, use the parsing results to read and write data in the second database.

[0038] In this embodiment, the aforementioned execution entity can utilize the parsing results in the backend node to perform data read and write operations in the second database. After receiving the data storage location and data query task, the backend node can execute the data query task, retrieve data from the corresponding storage location in the second database for data read and write operations, and also analyze the retrieved data.

[0039] Here, the aforementioned second database typically uses a row-oriented storage format to store data. This second database can include, but is not limited to, at least one of the following: HBase, or a Key-Value database. HBase is a distributed, column-oriented open-source database, a distributed storage system for structured data. A Key-Value database is a database based on key-value pair storage; data retrieval and storage are both accomplished through key-value pairs.

[0040] Here, the first database can retrieve data from the second database, and the second database can provide an additional data storage layer for the first database.

[0041] The method provided in the above embodiments of this disclosure receives data read / write requests for a first database storing data in columnar storage format and performs data read / write in a second database storing data in row-oriented storage format, thereby enabling row-oriented storage capabilities in a columnar storage database and satisfying high-concurrency point lookup and high-frequency update capabilities.

[0042] In some optional implementations, the aforementioned execution entity can parse the data read / write requests in the following way: The execution entity can determine the request type of the data read / write request, and then, based on the request type, determine the read / write paths for the front-end and back-end nodes to read and write data to the second database. The execution entity can store the correspondence between request types and read / write paths. After determining the request type, it can use the read / write path corresponding to that request type to read and write data to the second database. This method improves the flexibility of data read / write by selecting the appropriate read / write path based on the request type.

[0043] In some optional implementations, the aforementioned first database can include a Massively Parallel Processing (MPP) database. As an example, the first database could be StarRocks, a next-generation, high-speed, full-scenario MPP database. StarRocks' vision is to make data analysis simpler and more agile for users. Users can use StarRocks to support rapid analysis across various data analysis scenarios without complex preprocessing. StarRocks has a simple architecture, employs a fully vectorized engine, and is equipped with a newly designed CBO (Cost Based Optimizer), resulting in query speeds (especially for multi-table joins) far exceeding those of similar products. StarRocks effectively supports real-time data analysis and enables efficient querying of real-time updated data. StarRocks also supports modern materialized views, further accelerating queries. Using StarRocks, users can flexibly build various models, including wide tables, star schemas, and snowflake schemas. StarRocks is compatible with the MySQL (open-source relational database management system) protocol, supports standard SQL syntax, is easy to integrate, has no external dependencies, is highly available, and is easy to operate and manage.

[0044] The second database mentioned above can include a distributed, transactional key-value database. As an example, the second database could be TiKV, a highly scalable, low-latency, and easy-to-use key-value database.

[0045] StarRock can meet users' needs for building real-time data warehouses, and data analysis capabilities can achieve sub-second response times. However, StarRock uses a columnar storage format for data storage, which means it cannot handle high-concurrency point queries and high-frequency updates. The solution described in this embodiment uses StarRock to access data in TiKV to achieve row-based data read and write, thereby implementing row-based storage capabilities in the StarRock database, satisfying high-concurrency point queries and high-frequency updates. Furthermore, using a TiKV database for row-based storage reduces costs and improves ease of use.

[0046] In some optional implementations, since TiKV consists of two fields, Key and Value, the table data in the data read / write request needs to be encoded into a Key-Value field and stored in TiKV. Here, the key in the second database can include at least one of the following: a fixed prefix, a data shard identifier, an underscore, and data column data. The fixed prefix can be set by configuration options to distinguish different clusters. When multiple StarRocks clusters share a TiKV, the fixed prefix can distinguish the StarRocks from which the data originates. The data shard identifier (table_id) is typically 4 bytes. The underscore is used to facilitate data decoding. Table data in StarRocks consists of multiple columns, and some of these columns are encoded as the aforementioned data column data. The data column data can be categorized by type; for string types, it is categorized by delimiter.

[0047] The values ​​in the second database mentioned above can include at least one of the following: version number, number of columns, number of null columns, index array of null columns, and data column data. The version number (Version) usually refers to the TiKV version number, typically 4 bytes, used for subsequent compatibility. The number of columns (Col_size) is typically 4 bytes, used for handling inconsistencies between data storage and the schema (a collection of database objects). The number of null columns (null_col_size) is typically 4 bytes, used to indicate the length of the null col index array. The index array of null columns is typically a variable-length data structure, encoded using consecutive integers (int), with each int being 4 bytes. Table data in StarRocks consists of multiple columns, and some of these columns are encoded as the data column data mentioned above. The data column data can be sized by type; for string types, it is separated by a delimiter.

[0048] By setting the composition of the key-value fields in the second database in this way, the value field corresponding to a key field in the table can be found more quickly, thus improving decoding efficiency.

[0049] In some optional implementations, the first database mentioned above can store a collection of pre-compiled request templates, each corresponding to a type of data read / write request. To reduce the overhead of SQL statement parsing and expression evaluation, the StarRocks frontend can provide the PreparedStatement feature, which is fully compatible with the MySQL protocol. A PreparedStatement instance contains a compiled SQL statement, essentially "preparing" the statement. The SQL statement contained in the PreparedStatement object can have one or more IN parameters. The values ​​of the IN parameters are not specified when the SQL statement is created. Instead, the statement reserves a question mark ("?") as a placeholder for each IN parameter. The value of each question mark must be provided before the statement is executed using the appropriate setXXX method. When the PreparedStatement is enabled in the frontend, the SQL and its expressions are pre-computed and cached in an in-memory cache, allowing subsequent queries to directly use the cached object.

[0050] After receiving a data read / write request for the first database, the execution entity can determine whether the request template set contains a pre-compiled request template corresponding to the data read / write request. If so, the pre-compiled request template corresponding to the data read / write request can be used to process the data read / write request. That is, the specific parameters in the data read / write request can be added to the corresponding request template to obtain the parsed data read / write request.

[0051] This method allows for direct processing of requests by retrieving cached request templates from memory, thereby reducing the overhead of SQL statement parsing and expression calculation.

[0052] In some optional implementations, the aforementioned data read / write requests may include data update requests. The executing entity may receive data read / write requests for the first database in the following manner: if the executing entity determines that it has received at least two data update requests, it can determine the target data update request from among the at least two data update requests as the data update request for the first database. As an example, the data update request received earlier can be determined as the target data update request. In this case, a notification message can be sent to the user terminals from which the other data update requests besides the target data update request originate.

[0053] When inserting and updating data, eventual consistency must be ensured. If data conflicts are encountered, the conflicting data should be flagged.

[0054] As an example, such as Figure 2 As shown, Figure 2 This is a schematic diagram illustrating an application scenario of updating data in the data read / write method of this embodiment. Figure 2 In this scenario, if User 1 and User 2 simultaneously update the value of k1 in TiKV, only one user is allowed to update successfully; the other user will be prompted of a conflict. Here, since the earlier received request is from User 1, the request sent by User 1 will be used to update the value of k1, while a data conflict warning message will be sent to User 2's terminal.

[0055] Continue to refer to Figure 3 This illustrates a flow 300 of another embodiment of the data read / write method. Flow 300 of the data read / write method includes the following steps:

[0056] Step 301: Receive data read / write requests for the first database.

[0057] In this embodiment, step 301 can be performed in a similar manner to step 101, and will not be described again here.

[0058] Step 302: In the front-end node, determine the request type of the data read / write request.

[0059] In this embodiment, the execution entity of the data read / write method can determine the request type of the data read / write request in the aforementioned front-end node. The request type of the data read / write request can be determined based on the complexity of the operation indicated by the data read / write request.

[0060] As an example, if the above data read / write request indicates a relatively simple data operation such as data writing, point query, point update, or point delete, then the request type can be determined as the first request type. If the above data read / write request indicates a more complex data operation other than the above simple data operations (e.g., an operator with a shuffle (partition) operation), then the request type can be determined as the second request type.

[0061] Step 303: If the data read / write request is the first request type, determine that the front-end node and the back-end node perform data read / write on the second database using the first read / write path.

[0062] In this embodiment, if the data read / write request is determined to be the first request type in step 302, the execution entity can determine that the front-end node and the back-end node perform data read / write on the second database using the first read / write path.

[0063] Step 304: If the data read / write request is the second request type, determine that the front-end node and the back-end node use the second read / write path to perform data read / write on the second database.

[0064] In this embodiment, if the data read / write request is determined to be the second request type in step 302, the execution entity can determine that the front-end node and the back-end node perform data read / write on the second database using the second read / write path.

[0065] It should be noted that the first read / write path is usually shorter than the second read / write path. Since the data read / write operations indicated by the first request type are simpler than those indicated by the second request type, some data processing steps, such as data optimization steps, can be omitted. Therefore, the first read / write path is usually shorter than the second read / write path.

[0066] from Figure 3 It can be seen from this that, with Figure 1 Compared to the corresponding embodiments, the data read / write method flow 300 in this embodiment embodies the step of determining whether to use the first read / write path or the second read / write path to read / write data to the second database based on the request type of the data read / write request. Therefore, the solution described in this embodiment targets simpler data read / write requests and uses a shorter read / write path to perform data read / write, thereby improving the data read / write capability and efficiency of the row storage.

[0067] In some optional implementations, the first read / write path mentioned above may include: using a data interface via Remote Procedure Call (RPC) to read and write data to the second database. In RPC, the caller sends a request message to the remote host, which, upon receiving the request, executes the corresponding program and returns the result to the caller. Here, the first database is typically the caller, specifically the backend node of the first database, and the second database is typically the remote host. The first database can send a request message to the second database, and upon receiving the request, the second database executes the corresponding program, i.e., performs data read / write operations. Afterward, the second database can return the data read / write results to the first database.

[0068] In some optional implementations, the second read / write path mentioned above includes processing data read / write requests via a pipeline. A pipeline is a chain of operators; the SourceOperator acts as the starting operator of the pipeline, generating data for subsequent operators, while the SinkOperator acts as the ending operator, absorbing the pipeline's computation results and outputting the data. Here, the SourceOperator can be used to read data from the second database, and the SinkOperator can be used to write data to the second database.

[0069] Further reference Figure 4 This illustrates a flow 400 of an embodiment of a first read / write path in a data read / write method. The flow 400 of this first read / write path includes the following steps:

[0070] Step 401: In the front-end node, the data read / write request is parsed, a physical execution plan is generated, and the physical execution plan is sent to the back-end node.

[0071] In this embodiment, the first database can be StarRock. A StarRocks cluster typically consists of FrontEnd (FE) nodes and many BackEnd (BE) nodes. FE refers to the FrontEnd, the StarRocks front-end node, primarily responsible for cluster metadata management, client connection management, query parsing and planning, query execution plan generation, and query scheduling (distributing queries to BEs for execution). BE refers to the BackEnd, the StarRocks back-end node, primarily responsible for data storage, query execution plan execution, and replica management. The data in each table is distributed across multiple Tablets based on partitioning or bucketing mechanisms. To ensure fault tolerance, each Tablet is replicated multiple times, and these Tablets are ultimately distributed across different BEs.

[0072] In the front-end node of the first database mentioned above, the data read / write requests can be parsed to generate a physical execution plan. The physical execution plan is an execution tree composed of physical operators. The SQL statement undergoes processing stages such as parsing and analyzing to ultimately generate the physical execution plan. Here, the generated physical execution plan is used to perform remote procedure calls to the second database mentioned above. Specifically, the physical execution plan may include, but is not limited to: the region (slice) of data requested for reading / writing via RPC calls in TiKV, and encapsulated execution tasks recognizable by the BE end.

[0073] Then, the aforementioned front-end node can send the aforementioned physical execution plan to the aforementioned back-end node of the first database.

[0074] Step 402: In the backend node, data is read and written to the second database via remote procedure call using the data interface.

[0075] In this embodiment, in the backend node of the first database, the execution entity can perform data read and write operations on the second database via a data interface using a remote procedure call. Here, the backend node can send a request message to the second database. Upon receiving the request, the second database executes the corresponding program, i.e., performs data read and write operations. Afterward, the second database can return the data read and write results to the backend node.

[0076] The method provided by the above embodiments of this disclosure improves the QPS (Queries per second) of simple point operations (e.g., point queries, point updates, etc.) and write operations by sending RPC requests directly to a BE node without going through the PlanFragment and Pipeline execution chain.

[0077] Continue to refer to Figure 5 This illustrates a flow 500 of an embodiment of a second read / write path in a data read / write method. The flow 500 of this first read / write path includes the following steps:

[0078] Step 501: In the front-end node, the data read and write requests are parsed to generate a physical execution plan. The physical execution plan is then split into multiple plan fragments. Fragment instances are created based on the multiple plan fragments and sent to the back-end node.

[0079] In this embodiment, in the front-end node of the first database, the execution entity can parse the data read / write requests, generate a physical execution plan, split the physical execution plan into multiple plan fragments, create fragment instances based on the multiple plan fragments, and send the fragment instances to the back-end node.

[0080] Here, PlanFragment is a part of the physical execution plan. Only when the physical execution plan is split into several PlanFragments by the FE can it be executed in parallel on multiple machines. PlanFragment is also composed of physical operators, and also contains DataSink. The upstream PlanFragment sends data to the Exchange operator of the downstream PlanFragment through the DataSink.

[0081] A Fragment Instance is an execution instance of a PlanFragment. StarRocks tables are partitioned and bucketed into several tablets, each of which is stored as multiple replicas on compute nodes. PlanFragments can be instantiated into multiple Fragment Instances to process tablets distributed across different machines, thus achieving parallel data computation. The FE determines the number of Fragment Instances and the target BE (Executable Entity) to execute the Fragment Instances, and then the FE delivers the Fragment Instances to the BE.

[0082] In the Pipeline execution engine, the PipelineBuilder on the BE will further split the PlanFragment into several Pipelines. Each Pipeline will be instantiated into a set of PipelineDrivers according to the Pipeline parallelism parameter. The PipelineDriver is a Pipeline instance and also the basic task that the Pipeline execution engine can schedule.

[0083] Step 502: In the backend node, the fragment instance is processed using a pipeline link, and data is read from the second database using the start operator and written to the second database using the end operator.

[0084] In this embodiment, in the backend node of the first database, the execution entity can use a pipeline link to process the fragment instance, use the start operator to read data in the second database, and use the end operator to write data in the second database.

[0085] Here, the fragmented instances mentioned above are processed using a pipelined approach. A pipeline is a chain of operators. The SourceOperator acts as the starting operator of the pipeline, generating data for subsequent operators. The SinkOperator acts as the ending operator, absorbing the pipeline's computation results and outputting the data. Here, the SourceOperator can be used to read data from the second database, and the SinkOperator can be used to write data to the second database.

[0086] The method provided in the above embodiments of this disclosure performs data reading and writing through StarRocks' pipeline link, thereby enabling the processing of relatively complex SQL statements.

[0087] Please see Figure 6, Figure 6 This is a schematic diagram of the first read / write path and the second read / write path in the data read / write method according to this embodiment. Figure 6 In StarRocks, for more complex SQL statements, the SQL statements are processed through the following second read / write path: The front-end node of StarRocks performs the following operations on the SQL statement: parsing, analyzing, physical execution planner, optimizing, and generating fragmenter. After receiving the fragmenter, the back-end node of StarRocks performs the following operations: fragment execution FragmentExector, pipeline creation PipelineBuilder, pipeline execution PipelineExector, generating thread pool ThreadPool, and scanning connectors / data pools ConnectorScan / Datasink.

[0088] When StarRocks receives SQL statements submitted by clients, it goes through multiple processes, including syntax parsing, semantic parsing, and optimization rules, before converting the SQL statement into an executable physical execution plan. While this process itself isn't lengthy, it can be quite time-consuming for high QPS (queries, updates, and writes). In fact, for simpler SQL statements (e.g., point queries, updates, and writes), these complex processes are unnecessary.

[0089] For simpler SQL statements, the execution chain of PlanFragments and Pipelines can be bypassed, and the request can be sent directly to a BE node via RPC, thereby improving the QPS of simple queries. As an example, on the FE side, the scheduling of Optimizers and Fragments can be skipped, and the new RPC chain can be used directly. On the BE side, the construction and execution of the Pipeline can be skipped, and data can be read and written directly via ConnectorScan and returned.

[0090] SQL statements can be processed through the following first read / write path: The StarRocks front-end node will perform the following operations on the SQL statement: parser, analyze, and physical execution planner. After receiving the physical execution planner, the StarRocks back-end node will directly read and write data through ConnectorScan.

[0091] See further Figure 7 , Figure 7 This is a schematic diagram illustrating the data read / write operation performed according to the data read / write method of this embodiment. Figure 7In StarRocks, the front-end node can store pre-compiled SQL statements, reducing the overhead of SQL statement parsing and expression evaluation. The StarRocks front-end node provides transaction guarantees, ensuring eventual consistency during data insertion and updates, and flagging conflicting data. Furthermore, the StarRocks front-end node can manage metadata, enabling TiKV table management, such as storing column information and statistics for each row.

[0092] On the front-end nodes of StarRocks, SQL statements can be parsed, physical execution plans can be generated, and they can be split into PlanFragments. Fragment instances can be created based on PlanFragments and handed over to the back-end nodes of StarRocks for processing.

[0093] During the execution of the StarRocks front-end node, data reading and writing can be divided into two paths according to the type of SQL statement: one is the regular path, which uses Optimizer to optimize the SQL statement to obtain the optimal physical execution plan; then it is split into PlanFragment, and Fragment instances are created based on PlanFragment; DataSink is used to send data in the Fragment instance; after the Fragment instance is created, the front-end node hands it over to the back-end node for processing, as shown in the path of icon 701.

[0094] The other path is the short-circuit path. After receiving the SQL statement, the StarRocks front-end node checks whether it meets the short-circuit read / write pattern. If it does, it will skip the Optimizer optimization rules and directly take the RPC call, as shown in the path of icon 702.

[0095] During the execution of the StarRocks backend node, when a Fragment instance is received, it is also divided into two paths: the normal path and the short-circuit path.

[0096] The standard path uses a Pipeline to process Fragment instances. A Pipeline is a chain of operators, starting with a SourceOperator and ending with a SinkOperator. The SourceOperator is implemented in TikvScanner to retrieve data, and the SinkOperator is implemented in TikvWriter to write data, as shown in Figure 701.

[0097] The short-circuit path transforms the task into a new RpcService, which implements the Point interface functionality. This functionality directly operates the Tikv data source interface to read and write data, as shown in the path of icon 702.

[0098] These two paths allow you to read and write data in Tikv1, Tikv2, and Tikv3.

[0099] Further reference Figure 8 As an implementation of the methods shown in the above figures, this application provides an embodiment of a data read / write device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0100] like Figure 8 As shown, the data read / write device 800 of this embodiment includes a receiving unit 801, a sending unit 802, and a read / write unit 803. The receiving unit 801 receives data read / write requests for a first database, wherein the first database stores data in a columnar storage format, and the data read / write requests are used to read and write data in a row-based storage table. The first database includes a front-end node and a back-end node; the front-end node receives the data read / write requests, and the back-end node stores the data. The sending unit 802 parses the data read / write requests in the front-end node and sends the parsing results to the back-end node. The read / write unit 803 uses the parsing results in the back-end node to perform data read / write operations in a second database, wherein the first database retrieves data from the second database, and the second database stores data in a row-based storage format.

[0101] In this embodiment, the specific processing of the receiving unit 801, sending unit 802, and reading / writing unit 803 of the data reading / writing device 800 can be referred to Figure 1 The corresponding steps are 101, 102 and 103 in the embodiment.

[0102] In some optional implementations, the read / write unit 802 can be further used to parse the data read / write request in the following way: determine the request type of the data read / write request; and based on the request type of the data read / write request, determine the read / write path for the front-end node and the back-end node to read / write data to the second database.

[0103] In some optional implementations, the read / write unit 802 can be further configured to determine the read / write path for the front-end node and the back-end node to read / write data to the second database based on the request type of the data read / write request in the following manner: if the data read / write request is a first request type, determine that the front-end node and the back-end node perform data read / write on the second database using the first read / write path; if the data read / write request is a second request type, determine that the front-end node and the back-end node perform data read / write on the second database using the second read / write path, wherein the first read / write path is shorter than the second read / write path.

[0104] In some optional implementations, the first read / write path mentioned above may include: using a data interface to read and write data to the second database via remote procedure calls.

[0105] In some alternative implementations, the second read / write path may include: reading data from the second database using a start operator via a pipelined link, and writing data to the second database using an end operator.

[0106] In some alternative implementations, the first database may include a massively parallel processing database, and the second database may include a distributed transactional key-value database.

[0107] In some optional implementations, the first read / write path may include: in the front-end node, parsing the data read / write request, generating a physical execution plan, and sending the physical execution plan to the back-end node, wherein the physical execution plan is used to perform remote procedure calls on the second database; in the back-end node, data read / write is performed on the second database using a data interface via remote procedure calls.

[0108] In some optional implementations, the second read / write path may include: in the front-end node, parsing the data read / write request, generating a physical execution plan, splitting the physical execution plan into multiple plan fragments, creating fragment instances based on the multiple plan fragments, and sending the fragment instances to the back-end node; in the back-end node, using a pipeline link to process the fragment instances, using a start operator to read data from the second database, and using an end operator to write data to the second database.

[0109] In some optional implementations, the keys in the second database mentioned above include at least one of the following: a fixed prefix, a data shard identifier, an underscore, and data column data; and the values ​​in the second database mentioned above include at least one of the following: a version number, a number of columns, a number of null value columns, an index array of null value columns, and data column data.

[0110] In some optional implementations, the first database stores a set of pre-compiled request templates; and the data read / write device 800 may include a determining unit (not shown in the figure) and a processing unit (not shown in the figure). The determining unit is used to determine whether the set of request templates stores a pre-compiled request template corresponding to the data read / write request; the processing unit is used to process the data read / write request using the pre-compiled request template corresponding to the data read / write request if the data read / write request is found to be a pre-compiled request template.

[0111] In some alternative implementations, the aforementioned data read / write request includes a data update request; and the aforementioned receiving unit 801 may be further configured to receive a data read / write request for the first database in the following manner: in response to receiving at least two data update requests, determining a target data update request from the at least two data update requests as a data update request for the first database.

[0112] Figure 9 An exemplary system architecture 900 is shown, to which embodiments of the data read / write methods of this disclosure may be applied.

[0113] like Figure 9 As shown, system architecture 900 may include terminal devices 9011, 9012, and 9013, network 902, and server 903. Network 902 is used as a medium to provide communication links between terminal devices 9011, 9012, and 9013 and server 903. Network 902 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0114] Users can use terminal devices 9011, 9012, and 9013 to interact with server 903 via network 902 to send or receive messages, such as sending data read / write requests to server 903. Various communication client applications, such as short video software and search engines, can be installed on terminal devices 9011, 9012, and 9013.

[0115] Terminal devices 9011, 9012, and 9013 can be either hardware or software. When terminal devices 9011, 9012, and 9013 are hardware, they can be various electronic devices with displays and supporting information interaction, including but not limited to smartphones, tablets, and laptops. When terminal devices 9011, 9012, and 9013 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0116] Server 903 can be a server that provides various services. For example, it can be a backend server that processes data read and write requests. Server 903 can receive data read and write requests for a first database that stores data in a columnar storage format; then, in the frontend node of the first database, the data read and write requests are parsed, and the parsing results are sent to the backend node of the first database; then, in the backend node, the parsing results can be used to perform data read and write operations in a second database that stores data in a row-oriented storage format.

[0117] It should be noted that the server 903 can be either hardware or software. When the server 903 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server 903 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0118] It should also be noted that the data reading and writing method provided in this embodiment is usually executed by server 903, and the data reading and writing device is usually set in server 903.

[0119] It should be understood that Figure 9 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0120] The following is for reference. Figure 10 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 9 A structural diagram of the server (1000) in the middle. Figure 10 The server shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0121] like Figure 10 As shown, the electronic device 1000 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device 1000. The processing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0122] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 10 Each box shown can represent a device or multiple devices as needed.

[0123] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 1009, or installed from a storage device 1008, or installed from a ROM 1002. When the computer program is executed by a processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure. It should be noted that the computer-readable medium described in embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0124] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: receive data read / write requests for a first database, wherein the first database stores data in a columnar storage format, the data read / write requests are used to read and write data in a row-oriented storage table, the first database includes a front-end node and a back-end node, the front-end node is used to receive data read / write requests, and the back-end node is used to store data; in the front-end node, the data read / write requests are parsed, and the parsing results are sent to the back-end node; in the back-end node, data read / write is performed in a second database using the parsing results, wherein the first database retrieves data from the second database, and the second database stores data in a row-oriented storage format.

[0125] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0127] The units described in the embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a receiving unit and a read / write unit. The names of these units do not necessarily limit the specific unit; for example, a receiving unit may also be described as "a unit that receives data read / write requests for a first database."

[0128] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A data read / write method, characterized in that, include: Receive data read / write requests for a first database, wherein the first database stores data in a columnar storage format, the data read / write requests are used to read and write data in a row-based storage table, the first database includes a front-end node and a back-end node, the front-end node is used to receive the data read / write requests, and the back-end node is used to store data; In the front-end node, the data read / write request is parsed, and the parsing result is sent to the back-end node; In the backend node, the parsing results are used to read and write data in the second database, wherein the first database retrieves data from the second database, and the second database stores data in a row-oriented storage format; The parsing of the data read / write request includes: Determine the request type of the data read / write request; Based on the request type of the data read / write request, determine the read / write path for the front-end node and the back-end node to read / write data to the second database; The step of determining the read / write path for the front-end node and the back-end node to read and write data to the second database based on the request type of the data read / write request includes: If the data read / write request is of the first request type, determine that the front-end node and the back-end node perform data read / write on the second database using the first read / write path; If the data read / write request is of the second request type, the front-end node and the back-end node are determined to perform data read / write on the second database using the second read / write path, wherein the first read / write path is shorter than the second read / write path.

2. The method according to claim 1, characterized in that, The first read / write path includes: Data can be read and written to the second database using a remote procedure call method and a data interface.

3. The method according to claim 1, characterized in that, The second read / write path includes: Using a pipelined approach, data is read from the second database using a start operator and written to the second database using an end operator.

4. The method according to claim 1, characterized in that, The first database includes a massively parallel processing database, and the second database includes a distributed transactional key-value database.

5. The method according to claim 4, characterized in that, The first read / write path includes: In the front-end node, the data read / write request is parsed, a physical execution plan is generated, and the physical execution plan is sent to the back-end node. The physical execution plan is used to perform remote procedure calls on the second database. In the backend node, data is read and written to the second database via a remote procedure call using a data interface.

6. The method according to claim 4, characterized in that, The second read / write path includes: In the front-end node, the data read / write request is parsed, a physical execution plan is generated, the physical execution plan is split into multiple plan fragments, fragment instances are created based on the multiple plan fragments, and the fragment instances are sent to the back-end node; In the backend node, the fragment instance is processed using a pipeline link, and data is read from the second database using a start operator and written to the second database using an end operator.

7. The method according to claim 1, characterized in that, The keys in the second database include at least one of the following: a fixed prefix, a data shard identifier, an underscore, and data column data. The values ​​in the second database include at least one of the following: a version number, a number of columns, a number of null value columns, an index array of null value columns, and data column data.

8. The method according to claim 1, characterized in that, The first database stores a collection of pre-compiled request templates; as well as After receiving the data read / write request for the first database, the method further includes: Determine whether the request template set contains a pre-compiled request template corresponding to the data read / write request; If so, the data read / write request is processed using the pre-compiled request template corresponding to the data read / write request.

9. The method according to claim 1, characterized in that, The data read / write request includes a data update request; as well as Receiving data read / write requests for the first database includes: In response to receiving at least two data update requests, a target data update request is determined from the at least two data update requests as a data update request for the first database.

10. A data read / write device, characterized in that, include: A receiving unit is configured to receive data read / write requests for a first database, wherein the first database stores data in a columnar storage format, the data read / write requests are used to read and write data in a row-based storage table, the first database includes a front-end node and a back-end node, the front-end node is configured to receive the data read / write requests, and the back-end node is configured to store data. The sending unit is used in the front-end node to parse the data read / write request and send the parsing result to the back-end node; The read / write unit is used in the backend node to read and write data in the second database using the parsing result, wherein the first database retrieves data from the second database and the second database stores data in a row-oriented storage format. The sending unit is further configured to determine the request type of the data read / write request; based on the request type of the data read / write request, determine the read / write path for the front-end node and the back-end node to read / write data to the second database; if the data read / write request is a first request type, determine that the front-end node and the back-end node perform data read / write on the second database using the first read / write path; if the data read / write request is a second request type, determine that the front-end node and the back-end node perform data read / write on the second database using the second read / write path, wherein the first read / write path is shorter than the second read / write path.

11. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-9.

Images