A data processing method and device for columnar storage data warehouse

Abstract

The application discloses a data processing method for columnar storage data warehouse, comprising: receiving a data write request for requesting to write target data in a target write mode, the target data comprising at least one row of data. If the target write mode is a first write mode corresponding to a small data write amount, the target data is stored into a key-value storage module by using a key-value storage mode, so that other systems subsequently read the target data from the key-value storage module. In the key-value storage mode, each row of data in the at least one row of data corresponds to a key-value pair. If the target write mode is a second write mode corresponding to a large data write amount, the target data is stored into a columnar storage module by using a columnar storage technology. In the application, in the case that the data write amount corresponding to the target data is relatively small, the target data is timely stored into the key-value storage module, thereby meeting the requirement of real-time analysis on the target data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular, to a data processing method and device for a columnar storage data warehouse. BACKGROUND

[0002] With the development of computer technology, a large amount of data is generated, and accordingly, more and more data storage technologies have emerged. Columnar storage technology is a widely used data storage technology. The so-called columnar storage technology refers to storing data in columns as the storage granularity when storing tabular data. A data storage system that uses columnar storage technology to store data can also be referred to as a columnar storage data warehouse, where "data warehouse" is an abbreviation of "data warehouse".

[0003] In an example, after data is generated, the columnar storage technology can be used to store the data into a columnar storage data warehouse. In some scenarios, after the data is stored into the columnar storage data warehouse, other systems such as analysis systems can read the data from the columnar storage data warehouse in order to analyze the read data.

[0004] Currently, the time delay between the generation of data and the reading of the data from the columnar storage data warehouse by other systems is relatively long, and thus cannot meet the demand for real-time analysis of data. Therefore, there is an urgent need for a solution to solve the above problems. SUMMARY

[0005] To solve or at least partially solve the above technical problems, the embodiments of the present application provide a data processing method and device for a columnar storage data warehouse.

[0006] In a first aspect, the embodiments of the present application provide a data processing method for a columnar storage data warehouse, the method comprising:

[0007] receiving a data write request, the data write request being used to request writing target data in a target write mode, the target data comprising at least one row of data;

[0008] in response to the target write mode being a first write mode, storing the target data into a key-value storage module by using a key-value storage mode, so as to facilitate subsequent reading of the target data from the key-value storage module, each row of data in the at least one row of data corresponding to a key-value pair, the data write amount corresponding to the first write mode being less than a first threshold;

[0009] or,

[0010] In response to the target writing mode being a second writing mode, the target data is stored into a columnar storage module by using a columnar storage technology; the second writing mode corresponds to a data writing amount greater than a data writing amount corresponding to the first writing mode.

[0011] Optionally, the storing the target data into the key-value storage module by using the key-value storage mode comprises:

[0012] A first key-value object corresponding to a data lock in the key-value storage module is obtained, a key of the first key-value object indicating a data lock corresponding to the first key-value object, and a value of the first key-value object indicating a state of the data lock;

[0013] When the value of the first key-value object indicates that the data lock is in an un-locked state, the value of the first key-value object is modified to obtain a second key-value object, the value of the second key-value object indicating that the data lock is in a locked state;

[0014] The target data is stored into the key-value storage module by using the key-value storage mode.

[0015] Optionally, the key of the first key-value object indicates a data lock of a first data storage unit corresponding to the first key-value object, and the value of the first key-value object further indicates a first row number of data already stored in the first data storage unit;

[0016] The storing the target data into the key-value storage module by using the key-value storage mode comprises:

[0017] The target data is stored into the first data storage unit in the key-value storage module by using the key-value storage mode;

[0018] Correspondingly, the value of the second key-value object further indicates a second row number of data stored in the first data storage unit after the target data is stored into the first data storage unit.

[0019] Optionally, after the target data is stored into the first data storage unit in the key-value storage module by using the key-value storage mode, the method further comprises:

[0020] The value of the second key-value object is modified to obtain a third key-value object, wherein the value of the third key-value object indicates that the data lock is in an un-locked state and a second row number of data already stored in the first data storage unit.

[0021] Optionally, the method further comprises:

[0022] The data stored in the first data storage unit is stored into a columnar storage module by using columnar storage technology, the first data storage unit includes at least one batch of written data, and the at least one batch of written data includes the target data.

[0023] Optionally, the data stored in the first data storage unit is stored into a columnar storage module by using columnar storage technology, including:

[0024] The first version information corresponding to the first data storage unit is obtained, the first version information indicates a first batch of data that has been written into the first data storage unit and a second batch of data that is currently used for writing data;

[0025] The data corresponding to the first batch is read from the first data storage unit, and the data corresponding to the first batch is stored into the columnar storage module by using columnar storage technology.

[0026] Optionally, before the data corresponding to the first batch is read from the first data storage unit and the data corresponding to the first batch is stored into the columnar storage module by using columnar storage technology, the method further includes:

[0027] The value of the fourth key-value object carrying the first version information is modified to obtain a fifth key-value object, and the value of the fifth key-value object indicates that the data corresponding to the first batch is in a dumping state;

[0028] After the data corresponding to the first batch is stored into the columnar storage module by using columnar storage technology, the method further includes:

[0029] The value of the fifth key-value object is modified to obtain a sixth key-value object, and the value of the sixth key-value object indicates that the data corresponding to the first batch has been successfully stored into the columnar storage module.

[0030] Optionally, the method further includes:

[0031] According to the value of the sixth key-value object, the data corresponding to the first batch is deleted from the key-value storage module.

[0032] Optionally, the first version information is generated by the following method:

[0033] The second version information corresponding to the first data storage unit is obtained, and the second version information indicates a first batch of data that has been included in the first data storage unit;

[0034] On the basis of the first batch, a second batch for continuing to write data into the first data storage unit is generated.

[0035] According to the second version information and the second batch, the first version information is obtained.

[0036] Optionally, before generating the second batch for continuing to write data into the key-value storage module, the method further comprises:

[0037] obtaining a data lock of the first data storage unit and locking the data lock;

[0038] After obtaining the first version information, the method further comprises:

[0039] unlocking the data lock of the first data storage unit.

[0040] Optionally, the method further comprises:

[0041] receiving a data query request;

[0042] determining a storage range of data requested by the data query request in the key-value storage module;

[0043] processing data in the storage range to obtain a column block supported by the columnar storage module for processing;

[0044] processing the column block to obtain query data conforming to the data query request.

[0045] Optionally, the data query request includes a data storage unit identifier, the data storage unit identifier being used to identify a data storage unit to which data requested by the data query request belongs, and determining the storage range of data requested by the data query request in the key-value storage module comprises:

[0046] determining a target batch of data stored in the data storage unit identified by the data storage unit identifier, and determining a third row number of data stored in a largest batch in the target batch, the third row number corresponding to a last row of the largest batch;

[0047] determining the storage range according to the target batch and the third row number, the storage range being: a first row of a smallest batch in the target batch to the last row of the largest batch.

[0048] In a second aspect, an embodiment of the present application provides a data processing apparatus for a columnar storage data warehouse, the apparatus comprising:

[0049] a first receiving unit configured to receive a data write request, the data write request being used to request writing target data in a target write mode, the target data including at least one row of data;

[0050] a storage unit, configured to:

[0051] in response to the target write mode being a first write mode, store the target data into a key-value storage module by using a key-value storage mode, so as to subsequently read the target data from the key-value storage module, each row of data in the at least one row of data corresponding to a key-value pair respectively, the data write amount corresponding to the first write mode being less than a first threshold value;

[0052] or,

[0053] in response to the target write mode being a second write mode, store the target data into a columnar storage module by using a columnar storage technology; the data write amount corresponding to the second write mode being greater than the data write amount corresponding to the first write mode.

[0054] Optionally, the storage of the target data into the key-value storage module by using the key-value storage mode comprises:

[0055] obtaining a first key-value object corresponding to a data lock in the key-value storage module, the key of the first key-value object indicating a data lock corresponding to the first key-value object, and the value of the first key-value object indicating a state of the data lock;

[0056] when the value of the first key-value object indicates that the data lock is in an un-locked state, modifying the value of the first key-value object to obtain a second key-value object, the value of the second key-value object indicating that the data lock is in a locked state;

[0057] the storage of the target data into the key-value storage module by using the key-value storage mode.

[0058] Optionally, the key of the first key-value object indicates a data lock of a first data storage unit corresponding to the first key-value object, and the value of the first key-value object further indicates a first row number of data already stored in the first data storage unit;

[0059] the storage of the target data into the key-value storage module by using the key-value storage mode comprises:

[0060] the storage of the target data into the first data storage unit in the key-value storage module by using the key-value storage mode;

[0061] Correspondingly, the value of the second key-value object further indicates a second row number of data stored in the first data storage unit after the storage of the target data into the first data storage unit.

[0062] Optionally, the apparatus further comprises:

[0063] The first modification unit is configured to modify the value of the second key-value object to obtain a third key-value object after storing the target data into the first data storage unit in the key-value storage module by using the key-value storage mode, wherein the value of the third key-value object indicates that the data lock is in an unlocked state and a second row number of the data stored in the first data storage unit.

[0064] Optionally, the apparatus further comprises:

[0065] The dumping unit is configured to store the data stored in the first data storage unit into the columnar storage module by using the columnar storage technology, wherein the first data storage unit comprises at least one batch of written data, and the at least one batch of written data comprises the target data.

[0066] Optionally, the dumping unit is configured to:

[0067] The first version information corresponding to the first data storage unit is obtained, wherein the first version information indicates a first batch corresponding to the data written in the first data storage unit and a second batch currently used for writing data;

[0068] The data corresponding to the first batch is read from the first data storage unit, and the data corresponding to the first batch is stored into the columnar storage module by using the columnar storage technology.

[0069] Optionally, the apparatus further comprises a second modification unit and a third modification unit.

[0070] The second modification unit is configured to modify the value of a fourth key-value object carrying the first version information to obtain a fifth key-value object before reading the data corresponding to the first batch from the first data storage unit and storing the data corresponding to the first batch into the columnar storage module by using the columnar storage technology, wherein the value of the fifth key-value object indicates that the data corresponding to the first batch is in a dumping state.

[0071] The third modification unit is configured to modify the value of the fifth key-value object to obtain a sixth key-value object after storing the data corresponding to the first batch into the columnar storage module by using the columnar storage technology,

[0072] The value of the sixth key-value object indicates that the data corresponding to the first batch has been successfully stored into the columnar storage module.

[0073] Optionally, the apparatus further comprises:

[0074] A deleting unit is configured to delete the data corresponding to the first batch from the key-value storage module according to the value of the sixth key-value object.

[0075] Optionally, the first version information is generated in the following manner:

[0076] Obtain second version information corresponding to the first data storage unit, the second version information indicating a first batch to which data already included in the first data storage unit corresponds;

[0077] Generate a second batch for continuing to write data into the first data storage unit on the basis of the first batch;

[0078] Obtain the first version information according to the second version information and the second batch.

[0079] Optionally, the apparatus further comprises an obtaining unit and an unlocking unit.

[0080] The obtaining unit is configured to obtain a data lock of the first data storage unit and lock the data lock before generating a second batch for continuing to write data into the key-value storage module.

[0081] The unlocking unit is configured to unlock the data lock of the first data storage unit after obtaining the first version information.

[0082]

[0083] Optionally, the apparatus further comprises:

[0084] A second receiving unit is configured to receive a data query request.

[0085] A determining unit is configured to determine a storage range of data requested by the data query request in the key-value storage module.

[0086] A processing unit is configured to process the data in the storage range to obtain column blocks supported by the columnar storage module for processing.

[0087] A querying unit is configured to process the column blocks to obtain query data conforming to the data query request.

[0088] Optionally, the data query request comprises a data storage unit identifier, the data storage unit identifier being used to identify a data storage unit to which data requested by the data query request belongs, and the determining unit is configured to:

[0089] ​determining a target batch of data stored in the identified data storage unit identified by the data storage unit identifier, and determining a third row number of data stored in a largest batch in the target batch, the third row number corresponding to a last row of the largest batch;

[0090] determining the storage range according to the target batch and the third row number, the storage range being from a first row of a smallest batch in the target batch to the last row of the largest batch.

[0091] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising a processor and a memory;

[0092] The processor is configured to execute instructions stored in the memory, so that the device performs the method according to any one of the first aspect.

[0093] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions for instructing a device to perform the method according to any one of the first aspect.

[0094] In a fifth aspect, an embodiment of the present application provides a computer program product, when the computer program product is run on a computer, causing the computer to perform the method according to any one of the first aspect.

[0095] Compared with the prior art, the embodiments of the present application have the following advantages:

[0096] The embodiment of the present application provides a data processing method for columnar storage data warehouse, the method comprises the following steps: receiving a data write request, the data write request is used for requesting to write target data in a target write mode, and the target data can comprise at least one row of data. In the embodiment of the present application, it is considered that the data write mode can be associated with the data write amount. Or in other words, when writing data, a data write mode matched with the data write amount is generally used for writing. Therefore, how to store the target data can be determined according to the target write mode. In an example, if the target write mode is a first write mode corresponding to a relatively small data write amount (less than a first threshold value), the target data is directly stored into a key value (KV) storage module by using a KV storage mode, instead of being stored by using a columnar storage technology after the data amount corresponding to the data to be stored reaches a certain data amount, so as to facilitate other systems to read the target data from the KV storage module. When the target data is stored in the KV storage mode, each row of data in the at least one row of data corresponds to a key value pair respectively, in other words, when the target data is stored in the KV mode, the at least one row of data included in the target data can be stored row by row. If the target write mode is a second write mode corresponding to a relatively large data write amount, the target data is stored into a columnar storage module by using a columnar storage technology. The data write amount corresponding to the second write mode is greater than the data write amount corresponding to the first write mode. In the embodiment of the present application, in the case that the data write amount corresponding to the target data is relatively small, the target data is timely stored into the KV storage module in the KV storage mode, and correspondingly, other systems (for example, an analysis system) can also timely read the target data from the KV storage module, so as to meet the demand of real-time analysis on the target data. BRIEF DESCRIPTION OF DRAWINGS

[0097] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art without any creative effort.

[0098] Figure 1 A structural schematic diagram of a columnar storage data warehouse provided by the embodiment of the present application;

[0099] Figure 2 A data storage structure schematic diagram of a columnar storage data warehouse provided by the embodiment of the present application;

[0100] Figure 3 A flow schematic diagram of a data processing method for a columnar storage data warehouse provided by the embodiment of the present application;

[0101] Figure 4 A schematic diagram of a data storage method performed by a backend node according to an embodiment of the present application is shown in FIG. 2.

[0102] Figure 5 A flowchart of a method for storing target data into a key-value storage module according to an embodiment of the present application is shown in FIG. 3.

[0103] Figure 6 A flowchart of a data query method according to an embodiment of the present application is shown in FIG. 4.

[0104] Figure 7 A structural diagram of a data processing device of a columnar storage data warehouse according to an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0105] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor fall within the scope of protection of the present application.

[0106] The inventors of the present application have found that in the conventional technology, after data is generated, the columnar storage technology can be used to store the data into a columnar storage data warehouse.

[0107] Referring to FIG. 1, a structural diagram of a columnar storage data warehouse according to an embodiment of the present application is shown. Figure 1

[0108] Figure 1 The columnar storage data warehouse can be a batch storage-based columnar storage data warehouse, for example, a doris storage data warehouse.

[0109] As shown in FIG. 2, the columnar storage structure includes two frontend nodes (FE) and three backend nodes (BE). Here, only an example is shown, and in practice, the number of frontend nodes and backend nodes is not limited to that shown in FIG. 2. Figure 1 Figure 1

[0110] The frontend node is responsible for managing metadata, managing client connections, query planning, query scheduling, and the like. The backend node is responsible for data storage and structured query language (SQL) statement execution, and the like.

[0111] ​​​In terms of data storage, each backend node is completely equal, and the front-end node distributes data to the corresponding BE node according to a certain strategy. When data is imported, the data is directly written to the backend node, and the backend node is responsible for writing the imported data into the corresponding format and generating the related index.

[0112] When performing SQL calculation, a SQL statement is first planned into a logical execution unit according to specific semantics, and then split into a specific physical execution unit according to the distribution of data. The physical execution unit is executed on the specific backend node of the data storage, which can avoid data transmission and copying, thereby achieving the best query performance.

[0113] The columnar storage data warehouse exposes a My Structured Query Language (MySQL) protocol interface to the client, supporting standard SQL syntax. Users can conveniently query and analyze data in the columnar storage data warehouse through existing MySQL clients.

[0114] Figure 1 The columnar storage data warehouse shown has a storage structure similar to that of a Log-Structured Merge-tree (LSM). It can be understood in combination with Figure 2 Figure 2 A schematic diagram of a data storage structure of a columnar storage data warehouse provided by an embodiment of the present application. Figure 2 A data storage structure corresponding to a data storage unit in the columnar storage data warehouse is shown. The data storage unit mentioned here can be understood as the smallest data management unit corresponding to the columnar storage data warehouse, which can include several tables.

[0115] As Figure 2 shown, the data storage structure corresponding to a data storage unit can include multiple RowSets. RowSet has a number and version information. Whenever new data is written, a new RowSet is generated. Even if the corresponding data in the existing RowSet is modified, a new RowSet is generated instead of modifying the RowSet. However, the RowSet in the data storage unit is merged. Specifically, multiple small RowSets can be combined into a large RowSet according to the rules of the LSM tree. After the large RowSet is generated, the small RowSet of the large RowSet is deleted.

[0116] ​When data is written into a columnar storage data warehouse, if the amount of data written at a time is small, for example, only 1-10 rows of data are written at a time, a small RowSet is generated each time data is written, and when the small RowSets are merged subsequently, a large amount of performance of the columnar storage data warehouse is consumed.

[0117] To avoid the above problems, when the amount of data to be written is small, the data with the small amount can not be written into the columnar storage data warehouse immediately, but the data to be written is batched, and the data to be stored is stored by using the columnar storage technology after the amount of data corresponding to the data to be stored reaches a certain amount. However, in this way, for some data, there is a relatively large time delay (for example, the time delay is up to 10 seconds) from the generation of the data to the writing of the data into the columnar storage data warehouse. Correspondingly, the time delay from the generation of the data to the reading of the data by other systems from the columnar storage data warehouse is also relatively long, so that the real-time analysis of the data cannot be met. To solve the above problems, the embodiments of the present application provide a data processing method and device for a columnar storage data warehouse.

[0118] The various non-limiting embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0119] Exemplary method

[0120] Referring to Figure 3 , the figure is a flow diagram of a data processing method for a columnar storage data warehouse provided by an embodiment of the present application. In one example, the method provided by the embodiments of the present application can be applied to a columnar storage data warehouse. As a specific example, when the columnar storage data warehouse includes Figure 1 the structure shown, the method can be specifically applied to the backend node of the columnar storage data warehouse.

[0121] In the embodiment, the method may, for example, include the following steps: S101-S102.

[0122] S101: receiving a data write request, the data write request being used to request to write target data in a target writing mode, the target data including at least one row of data.

[0123] In one example, the data write request can be sent by a front-end node to the backend node. The data write request can be specifically a data write statement. The data write request can indicate a target writing mode, the target writing mode being used to indicate a mode of writing target data. In one example, the target writing mode can be determined by the aforementioned data write statement.

[0124] The target write mode is not specifically limited in the embodiments of the present application, and includes but is not limited to an Insert Values mode, a Stream Load mode, a Broker Load mode, or an Insert Select mode.

[0125] In response to the target write mode being a first write mode, the target data is stored into a key-value storage module by using a key-value storage mode, so as to be read from the key-value storage module later, each row of data in the at least one row of data corresponds to a key-value pair, and a data write amount corresponding to the first write mode is less than a first threshold; or, in response to the target write mode being a second write mode, the target data is stored into a columnar storage module by using a columnar storage technology, and a data write amount corresponding to the second write mode is greater than a data write amount corresponding to the first write mode.

[0126] In the embodiments of the present application, the data write mode can be associated with the data write amount. In other words, when data is written, a data write mode matching the data write amount is generally used for writing. Therefore, how to store the target data can be determined according to the target write mode.

[0127] In the embodiments of the present application, the data write amount corresponding to the first write mode is relatively small, specifically, less than a first threshold. For example, the data write amount corresponding to the first write mode can be less than M rows, M being a positive integer, for example, M is 100. In one example, considering that the data write amount corresponding to the Insert Values mode is relatively small, the first write mode can be the Insert Values mode. The data write amount corresponding to the second write mode is relatively large, greater than the data write amount corresponding to the first write mode. In one specific example, the data write amount corresponding to the second write mode is greater than a second threshold. The data write amount corresponding to the second write mode is relatively large, for example, the data write amount corresponding to the second write mode can be greater than N rows, N being a positive integer, for example, N is 100000. In one example, considering that the data write amount corresponding to the Stream Load mode, the Broker Load mode, and the Insert Select mode is relatively large, the second write mode can be the Stream Load mode, the Broker Load mode, or the Insert Select mode.

[0128] In the embodiments of the present application, in order to meet the requirement of real-time analysis on target data, after receiving the data writing request, if the target writing mode is a first writing mode corresponding to a smaller data writing amount, the data to be stored is not stored by using the columnar storage technology after the data amount corresponding to the data to be stored reaches a certain data amount as in the traditional technology, but is directly stored into the key-value storage module by using the key-value storage mode, so as to facilitate other systems (for example, an analysis system) to read the target data from the key-value storage module. Moreover, when the target data is stored by using the key-value storage mode, each row of data in the at least one row of data corresponds to a key-value pair.

[0129] In one example, for each row of data in the at least one row of data, the key of the key-value pair corresponding to the row of data is used to uniquely identify the row of data, and the keys in the key-value pairs corresponding to different rows of data are different. As an example, the target row in the at least one row of data is described with respect to the key-value pair corresponding to each row of data. The key of the key-value pair corresponding to the target row can include the identification of the data storage unit to which the target row belongs, the batch identification corresponding to the target row, and the row identification of the target row.

[0130] As an example, the key of the key-value pair corresponding to the target row can include the identification of the data storage unit to which the target row belongs, the batch identification corresponding to the target row, and the row identification of the target row. Among them:

[0131] The data storage unit is the smallest data management unit corresponding to the key-value storage module. In one example, the smallest data management unit corresponding to the key-value storage module and the smallest data management unit corresponding to the columnar storage warehouse mentioned above can be the same. For example, the smallest data management unit is a tablet.

[0132] The batch identification corresponding to the target row is used to identify the batch in which the target data is written into the key-value storage module. In some scenarios, the batch mentioned in the embodiments of the present application can also be referred to as a version.

[0133] In addition, the key of the key-value pair corresponding to the target row can include the cluster identification in addition to the above three elements, which is used to identify the cluster to which the backend node belongs. For this case, the key of the target row can be: {prefix}_{tablet_id}_{version_id}_{row_id}. Among them: prefix is used to identify the cluster to which the backend node belongs, tablet_id is the identification of the data storage unit to which the target row belongs, version_id is the batch identification corresponding to the target row, and row_id is the identification of the target row.

[0134] Correspondingly, if the target write mode is the second write mode, the target data can be directly stored into the columnar storage module by using columnar storage technology. The columnar storage technology mentioned here can be a traditional columnar storage technology (for example, the storage technology used by the doris storage data warehouse), and correspondingly, the columnar storage module mentioned here can be a module that stores data by using the traditional columnar storage technology. That is, the columnar storage module mentioned here can correspond to the traditional doris storage data warehouse. Details are not described here.

[0135] It can be understood in combination with Figure 4 that Figure 4 a schematic diagram of a data storage method performed by a backend node is provided in the embodiments of the present application. As Figure 4 shown, when the target write mode is the first write mode, the backend node writes the target data into the key-value storage module, and when the target write mode is the second write mode, the backend node writes the target data into the columnar storage module.

[0136] In one example, considering that the key-value storage module can store a limited amount of data as a "temporary" storage module, when the amount of data stored in the key-value storage module reaches a certain amount, the data stored in the key-value storage module can be transferred to the columnar storage module by using columnar storage technology. For specific implementation of transferring the data stored in the key-value storage module to the columnar storage module, reference can be made to the relevant description below. The data in the key-value storage module can also be referred to as "dumped" when transferred to the columnar storage module.

[0137] As can be known from the above description, by using the scheme of the embodiments of the present application, in the case where the data write amount corresponding to the target data is relatively small, the target data can be timely stored into the key-value storage module in the key-value storage mode, and correspondingly, other systems (for example, an analysis system) can also timely read the target data from the key-value storage module, thereby meeting the demand for real-time analysis of the target data.

[0138] Next, the specific implementation manner of storing the target data into the key-value storage module by using the key-value storage mode is introduced.

[0139] In one specific example, it is considered that at the same time, multiple backend nodes can have the need to write data into the key-value storage module. If multiple backend nodes write data into the key-value storage module at the same time, there will be a write conflict. In order to avoid this problem, a data lock is introduced in the embodiment of the present application. Before writing data into the key-value storage module, the data lock can be acquired and locked. For a certain backend node, the backend node locks the data lock, thereby obtaining the permission to write data into the key-value storage module, and other backend nodes cannot write data into the key-value storage module, thereby avoiding multiple backend nodes writing data into the key-value storage module at the same time.

[0140] Regarding the data lock, it needs to be explained that in one example, it can be a field, for example, it can correspond to a lock field, and the value of the field indicates whether the data lock is locked or not. For example, the value of the lock field is 0, indicating that the data lock is not locked, and the value of the lock field is 1, indicating that the data lock is locked.

[0141] In another example, the data lock can correspond to a key-value object. The key of the key-value object identifies the data lock corresponding to the key-value object, and the value of the key-value object can indicate the state of the data lock. The state of the data lock mentioned here can be a locked state or an unlocked state. For this case, "storing the target data into the key-value storage module by using the key-value storage method" in the specific implementation can include Figure 5 S201-S203. Figure 5 A flowchart of a method for storing target data into a key-value storage module provided by the embodiment of the present application.

[0142] S201: Acquire a first key-value object of the data lock in the key-value storage module. The key of the first key-value object indicates the data lock corresponding to the first key-value object, and the value of the first key-value object indicates the state of the data lock.

[0143] In one example, the key-value storage module can include multiple data storage units, and each data storage unit can correspond to a data lock. In one specific example, if it is needed to write target data into a first data storage unit in the key-value storage module, the data lock mentioned in S201 can be the data lock corresponding to the first data storage unit. The data lock corresponding to the first data storage unit is used to avoid multiple backend nodes writing data into the first data storage unit at the same time.

[0144] As to the first key-value object, it is to be noted that, in one example, the key of the first key-value object can be used to identify the data lock corresponding to the first key-value object. When the data lock is the data lock corresponding to the first data storage unit, the key of the first key-value object can indicate that the first key-value object corresponds to the data lock corresponding to the first data storage unit. In this case, the key of the first key-value object can include: the identification of the first data storage unit and the lock identification. Of course, the key of the first key-value object can also include: the cluster identification to which the backend node belongs. For example, the key of the first key-value object can be: {prefix}_{tablet_id}_lock. Wherein: prefix is used to identify the cluster to which the backend node belongs, tablet_id is the identification of the first data storage unit, and lock is the lock identification.

[0145] The value of the first key-value object is used to indicate the state of the data lock. In one example, in order to facilitate the backend node to write target data into the first data storage unit, the value of the first key-value object can be used to indicate the first row number of the data already stored in the first data storage unit in addition to indicating the state of the data lock. The first row number mentioned here means the maximum row number of the data already stored in the first data storage unit. For example, rows 1-9 in the first data storage unit have stored data, and the first row number can be 9. In one example, the key of the first key-value object can be, for example: Commit RowID, indicating that the data lock is in an unlocked state and the row number of the data currently stored is RowID.

[0146] S202: When the value of the first key-value object indicates that the data lock is in an unlocked state, the value of the first key-value object is modified to obtain a second key-value object, and the value of the second key-value object indicates that the data lock is in a locked state.

[0147] If the value of the first key-value object indicates that the data lock is in an unlocked state, it means that no other backend node is currently writing data into the first data storage unit, and for this case, the data lock can be locked. Specifically, the value of the first key-value object can be modified to obtain a second key-value object, and the value of the second key-value object indicates that the data lock is in a locked state. In a specific example, the modified second key-value object can embody, in addition to the first row number of the data already stored in the first data storage unit, a second row number of the data stored in the first data storage unit after the target data is stored in the first data storage unit. In this way, based on the second key-value object, the position of the currently written target data in the first data storage unit can be determined. Regarding the first row number and the second row number, it should be noted that if the target data includes m rows, the second row number can be equal to the first row number plus m.

[0148] Regarding the first key-value object and the second key-value object, Table 1 is now used for illustration.

[0149] Table 1

[0150]

[0151] S203: Store the target data into the key-value storage module using the key-value storage mode.

[0152] After the value of the first key-value object is modified to obtain the second key-value object, the data lock is locked, and further, after the data lock is locked, the target data can be stored into the key-value storage module using the key-value storage mode. In a specific example, the target data can be stored into the first data storage unit in the key-value storage module using the key-value storage mode.

[0153] In an example, after S203 is performed, the aforementioned data lock can also be unlocked to facilitate subsequent writing of data into the first data storage unit. Specifically, the value of the second key-value object can be modified to obtain a third key-value object, and the value of the third key-value object indicates that the data lock is in an unlocked state and a second row number of the data already stored in the first data storage unit. It is not difficult to understand that the difference between the third key-value object and the first key-value object is that the row number of the data already stored in the first data storage unit indicated by the value of the third key-value object is the second row number, while the row number of the data already stored in the first data storage unit indicated by the value of the first key-value object is the first row number.

[0154] Next, the key-value object corresponding to the data lock is illustrated in conjunction with Table 2.

[0155] Table 2

[0156]

[0157] As shown in Table 2, before the target data is written, the data lock corresponds to the first key-value object. Since the value of the first key-value object indicates that the data lock is in the unlocked state, the backend node can lock the data lock to obtain the second key-value object, and start writing the target data. That is, during the writing of the target data, the data lock corresponds to the second key-value object. After the writing of the target data is completed, the backend node can unlock the data lock to obtain the third key-value object.

[0158] In one example, if the data lock is in the locked state for too long, it indicates that the writing process of the target data may have an exception. In this case, in one example, the backend node can modify the second key-value object to the first key-value object. As an example, when the data lock is in the locked state, the backend node can periodically query the state of the data lock to determine whether the data lock is in the locked state for too long. The writing process of the target data having an exception can include two cases, one of which is that the data is not written, and the other is that the data is written but not committed.

[0159] As described above, the key-value storage module is a "temporary" storage module, and the amount of data it can store is limited. Therefore, when the amount of data stored in the key-value storage module reaches a certain amount, the columnar storage technology can be used to store the data stored in the key-value storage module to the columnar storage module. For example, the data stored in the key-value storage module can be stored to the columnar storage module at a regular time. As a specific example, the columnar storage technology can be used to store the data stored in the first data storage unit to the columnar storage module. The first data storage unit can include at least one batch of written data, and the aforementioned target data can be the data written by one of the at least one batch.

[0160] In one example, all data currently stored in the first data storage unit can be obtained, and the columnar storage technology can be used to store all the obtained data to the columnar storage module. During this process, writing new data to the first data storage unit can be prohibited.

[0161] In yet another example, it is considered that the key-value storage module is used to meet the requirement of real-time data analysis. Therefore, when storing the data in the first data storage unit into the columnar storage module, it should not be prohibited to write new data into the first data storage unit. In order to achieve this goal, "storing the data stored in the first data storage unit into the columnar storage module by using columnar storage technology" can include the following steps A1-A2 in specific implementation.

[0162] A1: obtaining the first version information corresponding to the first data storage unit, the first version information indicating the first batch corresponding to the data that has been written into the first data storage unit and the second batch currently used for writing data.

[0163] In one example, the first data unit can have corresponding version information, which is used to indicate the data write batch corresponding to the first data unit and the state of the data corresponding to each batch. In one example, the aforementioned version information can be obtained through the version corresponding key-value object.

[0164] In one example, the key of the version corresponding key-value object can be used to identify the version corresponding to the key-value object. In one example, the key of the version corresponding key-value object can include the identification of the first data storage unit and the version identification. Of course, the key of the version corresponding key-value object can also include the cluster identification to which the backend node belongs. For example, the key of the version corresponding key-value object can be: {prefix}_{tablet_id}_version. Wherein: prefix is used to identify the cluster to which the backend node belongs, tablet_id is the identification of the first data storage unit, and version is the version identification.

[0165] In one example, the value of the version corresponding key-value object can carry the aforementioned version information.

[0166] In one example, obtaining the first version information can include, for example, obtaining the fourth key-value object corresponding to the first version information, the value of the fourth key-value object including the first version information. The first version information is used to indicate the first batch corresponding to the data that has been written into the first data storage unit and the second batch currently used for writing data. Wherein, the first batch can include one or more batches. Regarding the value of the fourth key-value object, the following examples are given:

[0167] The value of the fourth key-value object can be, for example: {<version1, commit id1>, <version2, commit id2>, version3}, where <version1, commit id1> indicates that the data of batch 1 has been written, and commit id1 indicates the maximum row number in the data written by batch 1. <version2, commit id2> indicates that the data of batch 2 has been written, and commit id2 indicates the maximum row number in the data written by batch 2. version3 indicates that the current batch for writing data into the first data storage unit is batch 3. At this time, batch 1 and batch 2 are both first batches, and batch 3 is a second batch.

[0168] Alternatively, the value of the fourth key-value object can be, for example: {<version1, commit id1>, <version2, commit id2>, version2}, where <version1, commit id1> indicates that the data of batch 1 has been written, and commit id1 indicates the maximum row number in the data written by batch 1. version2 indicates that the current batch for writing data into the first data storage unit is batch 2. At this time, batch 1 is a first batch, and batch 2 is a second batch.

[0169] Regarding the first version information, it can be generated by the backend node before step A1 is performed. For the generation of the first version information, reference can be made to the description of steps B1-B3 below, which will not be described in detail here.

[0170] A2: Read the data corresponding to the first batch from the first data storage unit, and store the data corresponding to the first batch into the columnar storage module using columnar storage technology.

[0171] After obtaining the first version information, since the data corresponding to the first batch has been written, in other words, the first batch cannot be used to continue writing data, and the current batch for writing data is the second batch. Therefore, the data corresponding to the first batch can be read from the first data storage unit, and the data corresponding to the first batch can be stored into the columnar storage module using columnar storage technology. In this way, the data corresponding to the first batch can be dumped into the columnar storage module, and the writing of data into the key-value storage module based on the second batch can continue.

[0172] In one example, when the first batch includes multiple batches, the data corresponding to the multiple batches can be written into the columnar storage module in a parallel writing manner. For example, the data corresponding to each batch is written into a new RowSet.

[0173] In the embodiments of the present application, the data corresponding to the first batch can be stored into the columnar storage module by using the existing data storage mechanism of the columnar storage data warehouse. The existing data storage mechanism of the columnar storage data warehouse is not described in detail herein.

[0174] In one example, before step A2 is performed, the value of the fourth key-value object described above can be modified to obtain a fifth key-value object, and the value of the fifth key-value object indicates that the data corresponding to the first batch is in a dumping state. In this way, the state of the data corresponding to the first batch can be determined based on the value of the fifth key-value object, so as to manage the first key-value object based on the state. For example, when the state of the data corresponding to the first batch is the dumping state, the modification and deletion operations on the data corresponding to the first batch are prohibited.

[0175] Correspondingly, after the data corresponding to the first batch is successfully stored into the columnar storage module in step A2, the value of the fifth key-value object described above can be further modified to obtain a sixth key-value object, and the value of the sixth key-value object indicates that the data corresponding to the first batch has been successfully stored into the columnar storage module. In this way, the state of the data corresponding to the first batch can be determined based on the value of the sixth key-value object, so as to manage the first key-value object based on the state. In one specific example, the data corresponding to the first batch can be deleted from the key-value storage module according to the value of the sixth key-value object, so as to avoid that the data corresponding to the first batch is repeatedly stored in the key-value storage module and the columnar storage module. Deleting the data corresponding to the first batch from the key-value storage module can also release the storage space in the key-value storage module, so as to continue to store other data by using the storage space.

[0176] The fourth key-value object, the fifth key-value object and the sixth key-value object described above can be explained in combination with Table 3.

[0177]

[0178] In one example, the fifth key-value object can also be locked, and after a certain backend node locks the fifth key-value object, other backend nodes cannot store the data corresponding to the first batch into the columnar storage module according to the fifth key-value object, so as to avoid multiple backend nodes simultaneously performing the dumping operation on the data corresponding to the first batch. It should be noted that the fifth key-value object is locked, and the sixth key-value object obtained by modifying the value of the fifth key-value object is also in a locked state. Correspondingly, after the data corresponding to the first batch is deleted from the key-value storage module, the sixth key-value object can be unlocked.

[0179] In one example, if the writing of the data corresponding to the first batch into the columnar storage module fails, the fifth key-value object can be modified to the fourth key-value object, so as to continue to attempt to write the data corresponding to the first batch into the columnar storage module based on the fourth key-value object.

[0180] Next, the generation manner of the first version information will be introduced in combination with steps B1-B3.

[0181] B1: Obtain the second version information corresponding to the first data storage unit, and the second version information indicates the first batch corresponding to the data already included in the first data storage unit.

[0182] In one example, the seventh key-value object can be obtained, and the value of the key-value object includes the second version information. The first batch corresponding to the data already included in the first data storage unit can include a written batch and / or an uncompleted written batch. For example, the value of the seventh key-value object can be: {<version1, commitid1>, version2>}, at this time, version1 is the written batch 1, commit id1 represents the maximum row number in the data written in batch 1, version2 is the uncompleted written batch 2, and the first batch includes batch 1 and batch 2. For another example, the value of the seventh key-value object can be: {version1}, at this time, version1 is the uncompleted written batch 1, and the first batch can include batch 1. For another example, the value of the seventh key-value object can be:

[0183] {<version1, commit id1>}, at this time, version1 is the written batch 1, commit id1 represents the maximum row number in the data written in batch 1, and the first batch can include batch 1.

[0184] B2: On the basis of the first batch, a second batch for continuing to write data into the key-value storage module of the first data storage unit is generated.

[0185] In one example, a second batch whose batch number is greater than the maximum batch number in the first batch can be generated based on the first batch. For example, the first batch includes batch 1 and batch 2, and the generated second batch can be batch 3, i.e., the batch number of the second batch is 3.

[0186] B3: obtaining the first version information based on the second version information and the second batch.

[0187] As described above, the second version information includes the batch whose writing is completed and / or the batch whose writing is not completed. In one example, if the second version information includes the batch whose writing is completed and does not include the batch whose writing is not completed, in the implementation of step B3, the identifier of the second batch can be added to the second version information to obtain the first version information. For example:

[0188] The second version information is {<version1, commit id1>}, and the identifier of the generated second batch is version2. The first version information can be {<version1, commit id1>, version2}.

[0189] For example, the value of the seventh key-value object can be {<version1, commit id1>, version2>}. At this time, version1 is the batch 1 whose writing is completed, and commit id1 represents the maximum row number in the data written by batch 1, version2 is the batch 2 whose writing is not completed, and the first batch includes batch 1 and batch 2. For another example, the value of the seventh key-value object can be {version1}. At this time, version1 is the batch 1 whose writing is not completed, and the first batch can include batch 1. For another example, the value of the seventh key-value object can be {<version1, commit id1>}. At this time, version1 is the batch 1 whose writing is completed, and commit id1 represents the maximum row number in the data written by batch 1, and the first batch can include batch 1.

[0190] In one example, if the second version information includes the batch whose writing is not completed, in the implementation of step B3, the maximum row number of the data that has been written in the batch whose writing is not completed can be obtained, and the identifier of the second batch and the maximum row number can be added to the second version information to obtain the first version information. For example:

[0191] For example, the second version information is {<version1, commit id1>, version2}, the maximum row number of the data already written in the batch corresponding to version2 is commit id2, and the identifier of the second batch generated is version3. The first version information can be {<version1, commit id1>, <version2, commit id2>, version3}.

[0192] For example, the second version information is {<version1, commit id1>, version2}, the maximum row number of the data already written in the batch corresponding to version2 is commit id2, and the identifier of the second batch generated is version3. The first version information can be {<version1, commit id1>, <version2, commit id2>, version3}.

[0193] For example, the second version information is {<version1, commit id1>, version2}, the maximum row number of the data already written in the batch corresponding to version2 is commit id2, and the identifier of the second batch generated is version3. The first version information can be {<version1, commit id1>, <version2, commit id2>, version3}.

[0194] After obtaining the first version information, the value of the seventh key-value object can be modified to the first version information to obtain the fourth key-value object.

[0195] In one example, before step B2 is performed, or between step B1 and step B2, the data lock of the first data storage unit can also be acquired and the data can be locked. For example, the state indicated by the value of the key-value object corresponding to the data lock is modified to a locked state. In this way, other backend nodes cannot write data into the first data storage unit, and accordingly, the second version information obtained can be consistent with the data currently stored in the first data storage unit. Accordingly, after step B3 is performed, the data lock of the first data storage unit can be unlocked so that the backend node can continue to write data into the first data storage unit using the second batch. For the key-value object corresponding to the data lock, reference can be made to the foregoing description of the first key-value object, the second key-value object, and the third key-value object, which will not be described herein.

[0196] Next, the data query method provided by the embodiments of the present application will be described in combination with Figure 6 The data query method provided by the embodiments of the present application will be described in combination with Figure 6 The flowchart of a data query method provided by the embodiments of the present application is shown in FIG. 3. Figure 6 The method shown in FIG. 3 can include the following S301-S304.

[0197] S301: Receive a data query request.

[0198] In one example, the data query request can be an SQL statement.

[0199] In one example, S301, when implemented, can be receiving the data query request sent by the client, and the client and the columnar storage warehouse including the columnar storage module and the key-value storage module can communicate through, for example, a MySQL protocol. When the columnar storage warehouse includes the structure shown in FIG. 1, the data query request sent by the client can be received by the front-end node. Figure 1

[0200] In one example, the data query request can include a data storage unit identifier, which is used to indicate the data storage unit of the data requested by the data query request. For example, the data storage unit identifier can be the identifier of the first data storage unit.

[0201] In the embodiments of the present application, after receiving the data query request, the query data conforming to the data query request can be queried from the key-value storage module preferentially, or the query data conforming to the data query request can be queried from the key-value storage module and the columnar storage module simultaneously.

[0202] Next, how to query the query data from the key-value storage module is described in combination with S302-S304, and how to query the query data from the columnar storage module can follow the query logic of the traditional columnar storage warehouse, for example, the query logic of the traditional doris storage warehouse. Details are not described herein.

[0203] S302: determining the storage range of the data requested by the data query request in the key-value storage module.

[0204] In one example, the storage range of all data stored in the key-value storage module can be determined as the storage range of the data requested by the data query request.

[0205] In another example, when the data query request includes a data storage unit identifier, the storage range can be the range of the data stored in the data storage unit identified by the data storage unit identifier. The storage range can include a start position and an end position. In one example, the start position and the end position can each be represented by three elements, namely: a data storage unit identifier, batch information and a row number. For example, the determined storage range can be:

[0206] Start position: {tablet1}_{version1}_row1;

[0207] End position: {tablet1}_{version10}_row100.

[0208] ​That is, the determined storage location is from the first row of batch 1 of tablet 1 to the 100th row of batch 10 of tablet 1.

[0209] For this case, S302 may include steps C1-C2 in specific implementation.

[0210] Step C1: determining a target batch of data stored in the identified data storage unit according to the data storage unit, and determining a third row number of data stored in a maximum batch in the target batch, the third row number corresponding to a last row of the maximum batch.

[0211] Step C2: determining the storage range according to the target batch and the third row number, the storage range being from a first row of a minimum batch in the target batch to a last row of the maximum batch.

[0212] For convenience of description, the data storage unit to which the data requested by the data query request belongs is referred to as a second data storage unit, and the second data storage unit and the first data storage unit can be the same data storage unit or different data storage units, which are not specifically limited in the embodiments of the present application.

[0213] In the embodiments of the present application, the principles of the data storage and data reading processes performed by the backend node for each data storage unit are the same. As described above, the first data storage unit has a version corresponding key-value object, and therefore the second data storage unit also has a version corresponding key-value object. In the embodiments of the present application, the version corresponding key-value object of the second data storage unit can be obtained, so that the target batch of data stored in the second data storage unit is determined based on the key-value object. The target batch mentioned here includes at least a minimum batch and a maximum batch of data stored in the second data storage unit. For example, the target batch can include all batches (for example, batch 1 to batch 10) of data stored in the second data storage unit.

[0214] After determining the minimum batch, the aforementioned starting position is the first row of the minimum batch. The first row of the minimum batch refers to the first row of data written by the minimum batch, which can also be the first row in the second data storage unit.

[0215] Further, after determining the maximum batch, the last row of the maximum batch can be further determined, so as to determine the aforementioned ending position.

[0216] As described above, the first data storage unit has a data lock, and therefore, the second data storage unit also has a data lock. As described above for the key-value object corresponding to the data lock, the data lock can include the row number of the data that the data storage unit has written at present, and therefore, the key-value object corresponding to the data lock of the second data storage unit can be obtained to determine the third row number of the data stored in the maximum batch in the target batch, and the third row number is used to identify the last row of the data written by the maximum batch. After the third row number is obtained, the end position is determined as the last row of the maximum batch, and the last row of the maximum batch refers to the last row of the data written by the maximum batch.

[0217] S303: processing the data in the storage range to obtain a column block supported by the columnar storage module.

[0218] S304: processing the column block to obtain query data conforming to the data query request.

[0219] After the storage range is determined, the columnar storage module can be used to query the data to obtain query data conforming to the data query request. Specifically, the data in the storage range can be deserialized to obtain a column block supported by the columnar storage module. The column block mentioned here can be a memory structure supported by the columnar storage module. After the column block is obtained, the column block can be processed to obtain query data conforming to the data query request.

[0220] How to deserialize the data to obtain a column block and how to obtain query data based on the column block can follow the existing query processing logic of a traditional columnar storage warehouse (for example, the query logic of a doris storage warehouse), which will not be described in detail here.

[0221] Exemplary device

[0222] Based on the method provided in the above embodiments, the embodiments of the present application further provide a device, which will be described below with reference to the accompanying drawings.

[0223] Referring to Figure 7 The figure is a structural schematic diagram of a data processing device for a columnar storage warehouse provided by the embodiments of the present application. The device 700 can specifically include, for example: a first receiving unit 701 and a storage unit 702.

[0224] The first receiving unit 701 is configured to receive a data write request, and the data write request is used to request to write target data in a target write mode, and the target data includes at least one row of data.

[0225] The storage unit 702 is configured to:

[0226] in response to the target write mode being a first write mode, storing the target data into a key-value storage module by using a key-value storage mode, so as to subsequently read the target data from the key-value storage module, each of the at least one row of data corresponds to a key-value pair, and a data write amount corresponding to the first write mode is less than a first threshold value;

[0227] or,

[0228] in response to the target write mode being a second write mode, storing the target data into a columnar storage module by using a columnar storage technology; and a data write amount corresponding to the second write mode is greater than a data write amount corresponding to the first write mode.

[0229] Optionally, the storing the target data into the key-value storage module by using the key-value storage mode comprises:

[0230] obtaining a first key-value object corresponding to a data lock in the key-value storage module, a key of the first key-value object indicating that the first key-value object corresponds to the data lock, and a value of the first key-value object indicating a state of the data lock;

[0231] when the value of the first key-value object indicates that the data lock is in an un-locked state, modifying the value of the first key-value object to obtain a second key-value object, the value of the second key-value object indicating that the data lock is in a locked state;

[0232] storing the target data into the key-value storage module by using the key-value storage mode.

[0233] Optionally, the key of the first key-value object indicates that the first key-value object corresponds to a data lock of a first data storage unit, and the value of the first key-value object further indicates a first row number of data that has been stored in the first data storage unit.

[0234] The storing the target data into the key-value storage module by using the key-value storage mode comprises:

[0235] storing the target data into the first data storage unit in the key-value storage module by using the key-value storage mode.

[0236] Correspondingly, the value of the second key-value object further indicates a second row number of data stored in the first data storage unit after the target data is stored into the first data storage unit.

[0237] Optionally, the apparatus further comprises:

[0238] The first modification unit is configured to modify the value of the second key-value object to obtain a third key-value object after storing the target data into the first data storage unit in the key-value storage module by using the key-value storage mode, wherein the value of the third key-value object indicates that the data lock is in an unlocked state and a second row number of the data stored in the first data storage unit.

[0239] Optionally, the apparatus further comprises:

[0240] The dumping unit is configured to store the data stored in the first data storage unit into the columnar storage module by using the columnar storage technology, wherein the first data storage unit comprises at least one batch of written data, and the at least one batch of written data comprises the target data.

[0241] Optionally, the dumping unit is configured to:

[0242] The first version information corresponding to the first data storage unit is obtained, wherein the first version information indicates a first batch corresponding to the data written into the first data storage unit and a second batch currently used for writing data;

[0243] The data corresponding to the first batch is read from the first data storage unit, and the data corresponding to the first batch is stored into the columnar storage module by using the columnar storage technology.

[0244] Optionally, the apparatus further comprises a second modification unit and a third modification unit.

[0245] The second modification unit is configured to modify the value of a fourth key-value object carrying the first version information to obtain a fifth key-value object before reading the data corresponding to the first batch from the first data storage unit and storing the data corresponding to the first batch into the columnar storage module by using the columnar storage technology, wherein the value of the fifth key-value object indicates that the data corresponding to the first batch is in a dumping state.

[0246] The third modification unit is configured to modify the value of the fifth key-value object to obtain a sixth key-value object after storing the data corresponding to the first batch into the columnar storage module by using the columnar storage technology,

[0247] The value of the sixth key-value object indicates that the data corresponding to the first batch has been successfully stored into the columnar storage module.

[0248] Optionally, the apparatus further comprises:

[0249] A deleting unit is configured to delete the data corresponding to the first batch from the key-value storage module according to the value of the sixth key-value object.

[0250] Optionally, the first version information is generated in the following manner:

[0251] The second version information corresponding to the first data storage unit is obtained, and the second version information indicates the first batch corresponding to the data already included in the first data storage unit.

[0252] The second batch for continuing to write data into the first data storage unit is generated based on the first batch.

[0253] The first version information is obtained according to the second version information and the second batch.

[0254] Optionally, the apparatus further comprises an obtaining unit and an unlocking unit.

[0255] The obtaining unit is configured to obtain a data lock of the first data storage unit and lock the data lock before the second batch for continuing to write data into the key-value storage module is generated.

[0256] The unlocking unit is configured to unlock the data lock of the first data storage unit after the first version information is obtained.

[0257]

[0258] Optionally, the apparatus further comprises:

[0259] A second receiving unit is configured to receive a data query request.

[0260] A determining unit is configured to determine a storage range of data requested by the data query request in the key-value storage module.

[0261] A processing unit is configured to process the data in the storage range to obtain a column block supported by the columnar storage module for processing.

[0262] A querying unit is configured to process the column block to obtain query data conforming to the data query request.

[0263] Optionally, the data query request comprises a data storage unit identifier, and the data storage unit identifier is used to identify a data storage unit to which the data requested by the data query request belongs, and the determining unit is configured to:

[0264] ​determine a target batch of data stored in the identified data storage unit by the data storage unit identifier, and determine a third row number of data stored in a largest batch of the target batch, the third row number corresponding to a last row of the largest batch;

[0265] determine the storage range according to the target batch and the third row number, the storage range being from a first row of a smallest batch of the target batch to the last row of the largest batch.

[0266] Since the apparatus 700 is the apparatus corresponding to the data processing method for columnar storage data warehouse provided in the above method embodiments, the specific implementation of each unit of the apparatus 700 is the same as the same concept as the above method embodiments, therefore, as to the specific implementation of each unit of the apparatus 700, the related description part of the above method embodiments can be referred to, and will not be described here.

[0267] The present application embodiment further provides an electronic device, the electronic device comprising a processor and a memory;

[0268] The processor is configured to execute instructions stored in the memory, so that the device executes the data processing method for columnar storage data warehouse provided in the above method embodiments.

[0269] The present application embodiment provides a computer readable storage medium comprising instructions, the instructions instructing a device to execute the data processing method for columnar storage data warehouse provided in the above method embodiments.

[0270] The present application embodiment further provides a computer program product, when the computer program product runs on a computer, so that the computer executes the data processing method for columnar storage data warehouse provided in the above method embodiments.

[0271] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The specification and examples given are exemplary only and the true scope and spirit of the application is indicated by the following claims. The true scope and spirit of the application are indicated by the following claims.

[0272] It should be understood that the application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is indicated only by the appended claims.

[0273] The above only describes preferred embodiments of the present application and is not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method for columnar storage data warehouses, characterized in that, The method includes: Receive a data write request, the data write request being used to request the writing of target data in a target write mode, the target data including at least one row of data; In response to the target writing method being the first writing method, the target data is stored in the key-value storage module using a key-value storage method, so that the target data can be read from the key-value storage module later. Each row of data in the at least one row of data corresponds to a key-value pair, and the data writing amount corresponding to the first writing method is less than a first threshold. or, In response to the target write method being the second write method, the target data is stored in the columnar storage module using columnar storage technology; the data write volume corresponding to the second write method is greater than the data write volume corresponding to the first write method.

2. The method according to claim 1, characterized in that, The step of storing the target data in the key-value storage module using a key-value storage method includes: Obtain the first key-value object corresponding to the data lock in the key-value storage module. The key of the first key-value object indicates the data lock corresponding to the first key-value object, and the value of the first key-value object indicates the state of the data lock. When the value of the first key-value object indicates that the data lock is in an unlocked state, the value of the first key-value object is modified to obtain a second key-value object, and the value of the second key-value object indicates that the data lock is in a locked state. The target data is stored in the key-value storage module using a key-value storage method.

3. The method according to claim 2, characterized in that, The key of the first key-value object indicates the data lock of the first data storage unit corresponding to the first key-value object, and the value of the first key-value object also indicates the first row number of the data already stored in the first data storage unit; The step of storing the target data in the key-value storage module using a key-value storage method includes: The target data is stored in the first data storage unit of the key-value storage module using a key-value storage method; Accordingly, the value of the second key-value object also indicates the second row number of the data stored in the first data storage unit after the target data is stored in the first data storage unit.

4. The method according to claim 3, characterized in that, After storing the target data in the first data storage unit of the key-value storage module using a key-value storage method, the method further includes: The value of the second key-value object is modified to obtain a third key-value object, wherein the value of the third key-value object indicates that the data lock is in an unlocked state and the second row number of the data already stored in the first data storage unit.

5. The method according to claim 3, characterized in that, The method further includes: The data stored in the first data storage unit is transferred to the columnar storage module using columnar storage technology. The first data storage unit includes data written in at least one batch, and the data written in at least one batch includes the target data.

6. The method according to claim 5, characterized in that, The columnar storage technology is used to store the data stored in the first data storage unit into the columnar storage module, including: Obtain the first version information corresponding to the first data storage unit. The first version information indicates the first batch of data that has been written in the first data storage unit and the second batch of data currently being written. The data corresponding to the first batch is read from the first data storage unit, and the data corresponding to the first batch is stored in the columnar storage module using columnar storage technology.

7. The method according to claim 6, characterized in that, Before reading the data corresponding to the first batch from the first data storage unit and storing the data corresponding to the first batch in the columnar storage module using columnar storage technology, the method further includes: The value of the fourth key-value object carrying the first version information is modified to obtain a fifth key-value object. The value of the fifth key-value object indicates that the data corresponding to the first batch is in a dump state. After storing the data corresponding to the first batch in the columnar storage module using columnar storage technology, the method further includes: The value of the fifth key-value object is modified to obtain the sixth key-value object. The value of the sixth key-value object indicates that the data corresponding to the first batch has been successfully stored in the columnar storage module.

8. The method according to claim 7, characterized in that, The method further includes: Based on the value of the sixth key-value object, the data corresponding to the first batch is deleted from the key-value storage module.

9. The method according to claim 6, characterized in that, The first version information is generated in the following way: Obtain the second version information corresponding to the first data storage unit, where the second version information indicates the first batch of data already included in the first data storage unit. Based on the first batch, a second batch is generated for continuing to write data into the first data storage unit; The first version information is obtained based on the second version information and the second batch.

10. The method according to claim 9, characterized in that, Before generating a second batch for continuing to write data to the key-value storage module, the method further includes: Acquire the data lock of the first data storage unit and lock the data lock; After obtaining the first version information, the method further includes: Unlock the data lock on the first data storage unit.

11. The method according to claim 1, characterized in that, The method further includes: Receive data query requests; Determine the storage range of the data requested in the data query request within the key-value storage module; The data within the storage range is processed to obtain column blocks that the columnar storage module can process. The column blocks are processed to obtain query data that matches the data query request.

12. The method according to claim 11, characterized in that, The data query request includes a data storage unit identifier, which identifies the data storage unit to which the data requested in the data query request belongs, and determines the storage range of the data requested in the key-value storage module, including: Determine the target batch of data stored in the data storage unit identified by the data storage unit identifier, and determine the third row number of the data stored in the largest batch within the target batch, wherein the row corresponding to the third row number is the last row of the largest batch; The storage range is determined based on the target batch and the third row number, and the storage range is from the first row of the smallest batch in the target batch to the last row of the largest batch.

13. A data processing apparatus for a columnar storage data warehouse, characterized in that, The device includes: The first receiving unit is configured to receive a data write request, the data write request being used to request the writing of target data in a target write mode, the target data including at least one row of data; Storage unit, used for: In response to the target writing method being the first writing method, the target data is stored in the key-value storage module using a key-value storage method, so that the target data can be read from the key-value storage module later. Each row of data in the at least one row of data corresponds to a key-value pair, and the data writing amount corresponding to the first writing method is less than a first threshold. or, In response to the target write method being the second write method, the target data is stored in the columnar storage module using columnar storage technology; the data write volume corresponding to the second write method is greater than the data write volume corresponding to the first write method.

14. An electronic device, characterized in that, The device includes a processor and a memory; The processor is configured to execute instructions stored in the memory to cause the device to perform the method described in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, Includes instructions that instruct the device to perform the method described in any one of claims 1-12.

Images