A method, device, storage medium and electronic equipment for data query

Abstract

The specification discloses a data query method, determines a first data table stored in a row, establishes a column storage data table corresponding to the first data table as a second data table according to data stored in each column of the first data table, determines a row identifier of a row where data operated by a specified operation of the first data table and stores the row identifier in response to the specified operation, determines basic column data in the second data table according to a column identifier when a query operation carrying the column identifier is received, deletes data corresponding to the row identifier in the basic column data according to the stored row identifier, determines data corresponding to the row identifier in the first data table as supplementary data, and determines a data query result according to the basic column data after the data is deleted and the supplementary data.

Description

Technical Field

[0001] This specification relates to databases, and more particularly to a method, apparatus, storage medium, and electronic device for data retrieval. Background Technology

[0002] With the development of technology, database applications are becoming increasingly widespread. Currently, databases typically store data in a row-based manner. As the internet develops, the amount of data stored in databases is increasing. Because row-based indexes are stored on a row-by-row basis, the smallest unit for retrieving data is a single row.

[0003] When performing data queries, the database first needs to retrieve each row of data, and then, for each row, determine which column of data needs to be queried for the query operation. The data retrieved from the other columns of that row is useless for the query. This means that storing large amounts of data in a row-based manner can lead to a significant waste of resources when a query is performed on a specific column, potentially retrieving a large amount of data that is not relevant to the query.

[0004] To address the resource waste caused by performing column queries on row-stored data tables, databases have introduced column-store functionality. This allows queries to determine the result based on a specific column. However, this leads to the problem of frequently synchronizing row-stored and column-stored data.

[0005] Based on this, this specification provides a data query method that improves the speed of data query and reduces resource waste. Summary of the Invention

[0006] This specification provides a method, apparatus, storage medium, and electronic device for data retrieval, to at least partially solve the aforementioned problems existing in the prior art.

[0007] The following technical solution is adopted in this specification:

[0008] The data query methods provided in this manual include:

[0009] Determine the first data table for row storage;

[0010] Based on the data stored in each column of the first data table, a column storage data table corresponding to the first data table is created as the second data table;

[0011] In response to a specified operation on the first data table, the row identifier of the row containing the data operated on by the specified operation is determined and stored, wherein the type of the specified operation includes one of add, delete, and update;

[0012] When a query operation carrying a column identifier is received, the basic column data is determined in the second data table based on the column identifier;

[0013] Based on the stored row identifier, delete the data corresponding to the row identifier in the basic column data, and determine the data corresponding to the row identifier from the first data table as supplementary data;

[0014] The data query results are determined based on the base column data after data deletion and the supplementary data.

[0015] Optionally, determining and storing the row identifier of the row containing the data operated on by the specified operation specifically includes:

[0016] When the specified operation is of type delete or update, determine the row identifier of the row in the first data table where the data operated by the specified operation is located, and store the correspondence between the row identifier and the type of the specified operation;

[0017] When the specified operation is an add operation, a new row identifier is created, and the correspondence between the newly created row identifier and the specified operation type is stored. The newly created row identifier is different from the existing row identifiers in the first data table and the row identifiers of historical deletions.

[0018] Optionally, based on the stored row identifier, the data corresponding to the row identifier in the basic column data is deleted, and the data corresponding to the row identifier is determined from the first data table as supplementary data, specifically including:

[0019] For each row identifier stored, determine the row identifier and the type of the specified operation corresponding to it;

[0020] When the specified operation is a delete operation or an update operation, the data corresponding to the row identifier in the basic column data is deleted according to the row identifier, and the data corresponding to the row identifier is queried from the first data table according to the row identifier, and the query result is used as supplementary data.

[0021] When the specified operation is an add operation, the data corresponding to the row identifier is determined from the first data table based on the row identifier and used as supplementary data.

[0022] Optionally, determining and storing the row identifier of the row containing the data operated on by the specified operation specifically includes:

[0023] When the specified operation is received, the row identifier of the row containing the data operated on by the specified operation is determined and stored;

[0024] The time when the row identifier is stored is used as the first timestamp, and the correspondence between the second data table and the first timestamp is stored.

[0025] Optionally, based on the stored row identifier, the data corresponding to the row identifier in the basic column data is deleted, specifically including:

[0026] Determine the second timestamp carried by the query operation;

[0027] Determine whether the second timestamp is greater than the first timestamp;

[0028] If so, then based on the stored row identifier, determine the row identifier corresponding to the data to be deleted, and delete the corresponding data in the basic column data according to the determined row identifier.

[0029] Optionally, based on the stored row identifier, the row identifier corresponding to the data to be deleted is determined, specifically including:

[0030] Determine the transaction corresponding to the specified operation that triggered the recording of the first timestamp;

[0031] Determine whether the transaction to which the query operation belongs is the same as the identified transaction;

[0032] If so, then from the stored row identifiers, determine the row identifier whose storage time is no later than the first timestamp, and use it as the row identifier corresponding to the data to be deleted;

[0033] If not, then based on the transaction commit time corresponding to each specified operation in the record, determine the transactions committed before the second timestamp, and based on the row identifier corresponding to the transactions committed before the second timestamp, determine the row identifier corresponding to the data to be deleted.

[0034] Optionally, when the second timestamp is not greater than the first timestamp, the method further includes:

[0035] Identify other transactions that are different from the transaction to which the query request belongs, and determine the row identifier corresponding to the transaction committed before the second timestamp in the other transactions, as the row identifier corresponding to the data to be deleted;

[0036] When it is determined that a transaction to which the query operation belongs exists within an uncommitted transaction, the specified operation of the transaction to which the query operation belongs is determined;

[0037] Based on the row identifier of the row containing the data to be operated on, determine the row identifier of the data to be deleted.

[0038] Optionally, the second data table is updated based on the stored row identifiers and the first data table to obtain the third data table;

[0039] The system stores the second data table before the update, the third data table, and the third timestamp, where the third timestamp is the time when the third data table was created.

[0040] The basic column data is determined in the second data table based on the column identifier, specifically including:

[0041] Determine the second timestamp carried by the query operation;

[0042] Determine whether the second timestamp is greater than the third timestamp;

[0043] If so, determine the base column data in the third data table according to the column identifier, and delete the second data table;

[0044] If not, the base column data is determined in the second data table based on the column identifier.

[0045] Optionally, the second data table is updated based on the stored row identifiers and the first data table, specifically including:

[0046] For each row identifier in the storage, determine the type of the specified operation corresponding to that row identifier;

[0047] When the specified operation is an add operation, the data corresponding to the row identifier is determined from the first data table according to the row identifier, and the row identifier and the data corresponding to the row identifier are inserted into the second data table;

[0048] When the specified operation is a delete operation, the data corresponding to that row identifier in the second data table is deleted according to the row identifier.

[0049] When the specified operation is an update operation, the data corresponding to the row identifier in the second data table is deleted according to the row identifier. Then, the data corresponding to the row identifier is determined from the first data table according to the row identifier, and the row identifier and the data corresponding to the row identifier are inserted into the second data table.

[0050] Optionally, based on the data stored in each column of the first data table, multiple column storage data tables corresponding to the first data table are created according to a preset number of rows, serving as each second data table;

[0051] Determine and store the row identifier of the row containing the data operated on by the specified operation, specifically including:

[0052] Determine the row identifier of the row in the first data table where the data to be operated on by the specified operation is located;

[0053] Based on the row identifier, determine the second data table corresponding to the row identifier in each second data table, and use it as the storage data table;

[0054] Determine the correspondence between the row identifier and the stored data table, and store it.

[0055] Optionally, the basic column data is determined in the second data table based on the column identifier, specifically including:

[0056] Determine the row identifier carried by the query operation;

[0057] Based on the stored correspondence between the row identifier and the second data table, determine the second data table corresponding to the row identifier, and use it as the query data table;

[0058] Based on the column identifier, the base column data is determined in the query data table.

[0059] Optionally, determining the data corresponding to the row identifier from the first data table specifically includes:

[0060] Based on the row identifier, determine the data corresponding to the row identifier from the first data table;

[0061] Based on the query conditions carried in the query operation, data that meets the query conditions is filtered out from the data corresponding to the row identifier.

[0062] The data query apparatus provided in this specification includes:

[0063] The first determining module is used to determine the first data table of the row storage;

[0064] The second determining module is used to establish a column storage data table corresponding to the first data table based on the data stored in each column of the first data table, as the second data table;

[0065] The row identifier determination module is used to determine and store the row identifier of the row containing the data operated on by the specified operation in response to a specified operation on the first data table, wherein the type of the specified operation includes one of addition, deletion, and update;

[0066] The column determination module is used to determine the basic column data in the second data table based on the column identifier when a query operation carrying a column identifier is received;

[0067] The data processing module is used to delete the data corresponding to the row identifier in the basic column data according to the stored row identifier, and to determine the data corresponding to the row identifier from the first data table as supplementary data;

[0068] The query result determination module is used to determine the data query result based on the base column data after data deletion and the supplementary data.

[0069] Optionally, the row identifier determination module is specifically used to determine the row identifier of the row in the first data table where the data operated by the specified operation is located when the type of the specified operation is a deletion operation or an update operation, and to store the correspondence between the row identifier and the type of the specified operation. When the type of the specified operation is an add operation, a new row identifier is created, and the correspondence between the newly created row identifier and the type of the specified operation is stored. The newly created row identifier is different from the existing row identifiers in the first data table and the row identifiers of historical deletions.

[0070] Optionally, the data processing module is specifically used to determine the type of the specified operation for each stored row identifier; when the type of the specified operation is a delete operation or an update operation, delete the data corresponding to the row identifier in the basic column data according to the row identifier, and query the data corresponding to the row identifier from the first data table according to the row identifier, and use the query result as supplementary data; when the type of the specified operation is an add operation, determine the data corresponding to the row identifier from the first data table according to the row identifier, and use it as supplementary data.

[0071] Optionally, the row identifier determination module is specifically used to determine and store the row identifier of the row in which the specified operation operates when the specified operation is received, use the time of storing the row identifier as a first timestamp, and store the correspondence between the second data table and the first timestamp.

[0072] Optionally, the data processing module is specifically used to determine the second timestamp carried by the query operation, determine whether the second timestamp is greater than the first timestamp, and if so, determine the row identifier corresponding to the data to be deleted based on the stored row identifier, and delete the data corresponding to the row identifier in the basic column data based on the determined row identifier.

[0073] Optionally, the data processing module is specifically used to determine the transaction corresponding to the specified operation that triggered the recording of the first timestamp, and to determine whether the transaction to which the query operation belongs is the same transaction as the determined transaction. If so, the module determines the row identifier whose storage time is no later than the first timestamp from the stored row identifiers, and uses it as the row identifier corresponding to the data to be deleted. If not, the module determines the transaction that was submitted before the second timestamp based on the submission time of the transaction corresponding to each specified operation recorded, and determines the row identifier corresponding to the data to be deleted based on the row identifier corresponding to the transaction that was submitted before the second timestamp.

[0074] Optionally, when the second timestamp is not greater than the first timestamp, the data processing module is further configured to determine the transactions committed before the second timestamp based on the transaction commit time corresponding to each specified operation recorded, determine the row identifier corresponding to the data to be deleted based on the row identifier corresponding to the transactions committed before the second timestamp, and when it is determined that there is a transaction to which the query operation belongs among the transactions not committed before the second timestamp, determine the specified operation of the transaction to which the query operation belongs, and determine the row identifier corresponding to the data to be deleted based on the row identifier of the row where the data operated by the determined specified operation is located.

[0075] Optionally, the column determination module is further configured to update the second data table according to the stored row identifiers and the first data table to obtain a third data table, and store the second data table before the update, the third data table, and a third timestamp, wherein the third timestamp is the time when the third data table was created. When determining the basic column data in the second data table according to the column identifier, the module is specifically configured to determine the second timestamp carried by the query operation, and determine whether the second timestamp is greater than the third timestamp. If so, the module determines the basic column data in the third data table according to the column identifier and deletes the second data table. If not, the module determines the basic column data in the second data table according to the column identifier.

[0076] Optionally, the column determination module is specifically used to determine the type of a specified operation corresponding to each stored row identifier. When the type of the specified operation is an add operation, the module determines the data corresponding to the row identifier from the first data table based on the row identifier and inserts the row identifier and the data corresponding to the row identifier into the second data table. When the type of the specified operation is a delete operation, the module deletes the data corresponding to the row identifier from the second data table based on the row identifier. When the type of the specified operation is an update operation, the module deletes the data corresponding to the row identifier from the second data table based on the row identifier, then determines the data corresponding to the row identifier from the first data table based on the row identifier and inserts the row identifier and the data corresponding to the row identifier into the second data table.

[0077] Optionally, the second determining module is further configured to establish multiple column storage data tables corresponding to the first data table according to a preset number of rows based on the data stored in each column of the first data table, as each second data table;

[0078] The row identifier determination module is specifically used to determine the row identifier of the row in the first data table when the row identifier of the data operated by the specified operation is determined and stored, and based on the row identifier, determine the second data table corresponding to the row identifier in each second data table as a storage data table, determine the correspondence between the row identifier and the storage data table, and store it.

[0079] Optionally, the column determination module is specifically used to determine the row identifier carried by the query operation, determine the second data table corresponding to the row identifier based on the stored correspondence between the row identifier and the second data table, and use it as the query data table, and determine the basic column data in the query data table based on the column identifier.

[0080] Optionally, the data processing module is specifically used to determine the data corresponding to the row identifier from the first data table based on the row identifier, and to filter out the data that meets the query conditions from the data corresponding to the row identifier based on the query conditions carried in the query operation.

[0081] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data query method.

[0082] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described data query method.

[0083] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0084] In the data query method provided in this specification, a first data table for row storage is determined. Based on the data stored in each column of the first data table, a column-stored data table corresponding to the first data table is established as a second data table. In response to a specified operation on the first data table, the row identifier of the row containing the data operated on by the specified operation is determined and stored. The type of the specified operation includes one of addition, deletion, and update. When a query operation carrying a column identifier is received, basic column data is determined in the second data table based on the column identifier. Based on the stored row identifier, the data corresponding to the row identifier in the basic column data is deleted, and the data corresponding to the row identifier is determined from the first data table as supplementary data. Based on the basic column data after data deletion and the supplementary data, the data query result is determined.

[0085] As can be seen from the above method, this method reduces resource waste during querying and improves the efficiency of data querying by querying the newly created column storage data table. Attached Figure Description

[0086] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and their descriptions, serving to explain this specification and do not constitute an undue limitation thereof.

[0087] In the picture:

[0088] Figure 1 This is a flowchart illustrating a data query method provided in this specification.

[0089] Figure 2a This is a schematic diagram of the first data table;

[0090] Figure 2b This is a schematic diagram of the first data table after it has been split.

[0091] Figure 3 This is a schematic diagram of the internal structure of the second data table;

[0092] Figure 4 A diagram illustrating the execution of a query operation;

[0093] Figure 5 A schematic diagram of a data query device provided in this specification;

[0094] Figure 6 The corresponding information provided in this specification Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0095] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0096] Currently, databases generally store data using row-based storage. When the database contains a large amount of data, if only a specific column needs to be processed, the server needs to retrieve data from all rows related to that column and then select the required column. This means the server will expend more resources retrieving other data besides the required column, slowing down the retrieval process and impacting the efficiency of subsequent processing. For example, if a database contains 10 million rows representing the scores of 10 million students in nine subjects (Chinese, Math, English, etc.), and a user needs to calculate the average English score for these 10 million students, the server will first retrieve the scores for all nine subjects and then filter out the corresponding English score. The remaining 80 million rows are considered useless data. Clearly, retrieving this useless data takes more time and resources than retrieving the required data, and this time and resource consumption increases with the amount of data stored in the database.

[0097] Currently, databases also incorporate column storage and incremental data recording capabilities. When querying a specific column, the query result is determined based on both the column data and the incremental data. However, incremental data consumes significant memory, and frequent column updates can degrade database performance. To reduce resource waste, minimize time spent on useless data, and improve data query efficiency, this manual provides a data query method.

[0098] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0099] Figure 1 This document provides a flowchart illustrating a data query method, which includes the following steps:

[0100] S100: Determine the first data table for row storage.

[0101] In one or more embodiments of this specification, the data query process can be executed by a database management system. Of course, the data query process can also be executed by other devices, and this specification does not impose any limitations. For ease of description, the database server is used as an example to illustrate the execution of the data query process in the following description.

[0102] Specifically, the database can contain multiple row-stored data tables. Each row-stored data table can use the data query provided in this manual to realize column-stored data and column-query, thereby improving query efficiency while reducing memory usage.

[0103] Since the query execution process is performed by the database server, the incremental data is stored in the server's memory. If the server's memory space is limited, it may be difficult to support the storage of incremental data for a large number of column-oriented data tables. Therefore, the server can identify the data in the database that is frequently used in query operations and designate the row-oriented data table storing this data as the first data table. That is, the row-oriented data table containing the data that query requests frequently target is designated as the first data table. Subsequent steps establish the column-oriented data table corresponding to this first data table and the incremental data, achieving efficient querying. Simultaneously, it avoids allocating space for storing incremental data in all row-oriented data tables, reducing the server's memory usage.

[0104] For example, for each piece of data, the server can determine how frequently the data is queried based on a preset query count. Suppose the preset query count is 6, and the data is queried 7 times; then the data is considered frequently queried, and the row-based data table storing that data is selected as the first data table.

[0105] S102: Based on the data stored in each column of the first data table, establish a column storage data table corresponding to the first data table as the second data table.

[0106] In one or more embodiments of this specification, in order to reduce the resources and time wasted on useless data when querying data, the server may store the data of the row storage data table in additional memory in a columnar storage manner.

[0107] Specifically, the server can determine the column index of the first data table based on the column identifier corresponding to each column. Then, based on the data stored in each column of the first data table and the column index, it can create a column storage data table corresponding to the first data table, which serves as the second data table.

[0108] The second data table can maintain data inconsistency with the first data table. By recording incremental data, the second data table can return the correct query results when querying.

[0109] S104: In response to a specified operation on the first data table, determine and store the row identifier of the data operated on by the specified operation, wherein the type of the specified operation includes one of addition, deletion, and update.

[0110] In one or more embodiments of this specification, due to business needs, the server may update, delete, or add data to the first data table. The server needs to record the row identifiers corresponding to the modified data in the first data table. That is, if the data in the first data table changes, the server needs to record the row identifiers corresponding to the changed data in the first data table. In other words, in response to a specified operation on the first data table, the server determines and stores the row identifier of the data operated on by the specified operation, so that the server can subsequently query the first data table based on the corresponding row identifier.

[0111] Specifically, when the specified operation is a delete operation or an update operation, the server can determine the row identifier of the row in the first data table where the specified operation operates based on the data operated on by the specified operation, and store it.

[0112] When the specified operation is an add operation, the server can create and store the row identifier corresponding to the add operation. This row identifier is globally unique, meaning it is unique across all rows currently stored in the first data table, as well as row identifiers of deleted rows. In other words, a row identifier that has appeared historically cannot be used as a new row identifier; the row identifier determined each time a new row is added is a historically unused row identifier. This avoids situations where different rows correspond to the same row identifier. Therefore, the server needs to record historically used row identifiers to determine the possible values ​​for the new row identifier. Of course, the server can also reset the row identifiers for each row as needed. This means resetting historically occurring row identifiers (i.e., those for deleted records) and reallocating row identifiers for the currently stored rows.

[0113] If the newly created row identifier is the same as an existing row identifier in the first data table, after the specified operation, if the server performs a query operation based on the row identifier, the server will query the required data from the data of multiple rows corresponding to the row identifier, resulting in redundant data in the query results. Therefore, the newly created row identifier must be different from the existing row identifier in the first data table.

[0114] Since the second data table can be updated according to preset conditions, if the server deletes an entire row of data according to the specified operation before updating the second data table, then the row identifier of that row will also be deleted after the update, resulting in a previously deleted row identifier. If the second data table contains deleted rows of data, and the server performs an add operation on the second data table, assuming the newly created row identifier is the same as the row identifier corresponding to the deleted row, then when the server updates the second data table, it will delete the data in the row with the newly created row identifier, resulting in a missing data situation in the updated second data table. Therefore, the newly created row identifier must be different from the previously deleted row identifier.

[0115] In one or more embodiments of this specification, an update operation can be broken down into performing a delete operation first, followed by an add operation. The server can store the row identifier of the object to be updated based on the delete operation. Of course, since the update operation is broken down into delete and add operations, it is not necessary to create a new row identifier when performing the add operation. It is only necessary to determine the row identifier of the object to be updated and then store that row identifier.

[0116] Furthermore, in this embodiment of the specification, the purpose of storing row identifiers is to determine which data in the second data table has changed and requires retrieving the latest data from the first data table. Therefore, a row identifier only needs to be stored once. Before storing a row identifier, the server can also determine whether the row identifier has already been stored. If it has already been stored, it does not need to be stored again; otherwise, the row identifier is stored.

[0117] S106: When a query operation carrying a column identifier is received, the basic column data is determined in the second data table according to the column identifier.

[0118] In one or more embodiments of this specification, when the server receives a query operation carrying column identifiers, the server first determines the list as the corresponding column data in the second data table based on the column identifiers, and uses it as the base column data. After determining the base column data, the server can determine which data in the base column data has been subjected to the specified operation based on the row identifiers carried by the query operation, and update the data corresponding to the row identifiers in the queried base column data through subsequent steps.

[0119] S108: Based on the stored row identifier, delete the data corresponding to the row identifier in the basic column data, and determine the data corresponding to the row identifier from the first data table as supplementary data.

[0120] In one or more embodiments of this specification, to save memory space, the second data table of the server only stores the row identifier corresponding to the specified operation, without storing the data corresponding to the row identifier. However, the specified operation may change the data in the first data table, but the data in the second data table does not change simultaneously. Therefore, the data retrieved by the server from the second data table based on the row identifier may be inaccurate. Therefore, for each stored row identifier, the server iterates through the basic column data based on the row identifier, determines the row corresponding to the row identifier, and deletes the data corresponding to the row identifier from the basic column data. The remaining data in the basic column data are all data that has not undergone the specified operation, and the remaining data in the basic column data is consistent with the data in the row containing the row identifier in the first data table. Afterwards, the server determines the data corresponding to the row identifier from the first data table as supplementary data.

[0121] For supplementary data, the server determines the data corresponding to the row identifier from the first data table based on the row identifier, and then filters out the data that meets the query conditions carried in the query operation. These query conditions can be restrictions such as querying numbers greater than 100; this specification does not impose such restrictions, as long as the data filtered by the server from the first data table based on the query conditions meets the requirements of the user or other querying parties.

[0122] S110: Determine the data query result based on the base column data after data deletion and the supplementary data.

[0123] In one or more embodiments of this specification, the server determines the data query result based on the base column data after data deletion and the supplementary data.

[0124] Specifically, after the supplementary data is filtered according to the query conditions, the resulting data meets the requirements of the user or other querying party. The base column data after data deletion is determined based on the column identifier of the query operation, and since the data corresponding to the row identifier that underwent the specified operation has been deleted, the base column data after data deletion also meets the requirements of the user or other querying party. Therefore, the server can return the base column data after data deletion and the supplementary data as the data query result to the user or other querying party.

[0125] The server can also process the supplementary data and the base column data after data deletion, such as performing mathematical operations on them. The server then uses the processed supplementary data and the base column data to determine the query result and returns it to the user or other querying party.

[0126] based on Figure 1 The data query method shown involves determining a first data table for row storage, establishing a second data table for column storage based on the data stored in each column of the first data table, and determining and storing the row identifier of the data operated on by the specified operation in response to a specified operation on the first data table. The specified operation type includes one of add, delete, and update. When a query operation carrying a column identifier is received, basic column data is determined in the second data table based on the column identifier. Based on the stored row identifier, the data corresponding to the row identifier in the basic column data is deleted, and the data corresponding to the row identifier is determined from the first data table as supplementary data. The data query result is determined based on the basic column data after data deletion and the supplementary data.

[0127] As can be seen from the above method, this method uses additional memory to store the data of the row storage data table in column storage mode in that memory, and stores the row identifiers corresponding to the incremental data and the newly established column storage data table, thereby reducing the memory space used and improving the data query efficiency.

[0128] Furthermore, in one or more embodiments of this specification, the server can receive specified operations and query operations sent by different database service initiators. Since the server needs some time to perform the specified operation on the first data table, if a query operation is performed during this period, and the data retrieved by the query operation is related to the data operated on by the specified operation, the query result obtained by the server may be incorrect.

[0129] Therefore, to ensure data consistency and make the query results obtained by each database service initiator more accurate, in step S104, when the server receives the specified operation, it can determine and store the row identifier of the data operated on by the specified operation, and use the time of storing the specified operation as the first timestamp, storing the correspondence between the second data table and the first timestamp. Then, when determining column data from the second data table, the time when the specified operation was most recently executed in the corresponding first data table can be determined. This allows the server to determine, in subsequent step S106, which row identifiers in the column data of the second data table need to be deleted based on the second timestamp carried by the query operation. The second timestamp is the timestamp corresponding to the moment the query operation was sent by the sender.

[0130] In one or more embodiments of this specification, when the server deletes data corresponding to row identifiers in the basic column data according to the stored row identifiers in step S108, it can determine the second timestamp carried by the query operation received in step S106. It then determines whether the second timestamp is greater than the first timestamp corresponding to the second data table. If so, it indicates that the query operation is the most recently received operation, and all specified operations received in step S104 occurred before this query operation. Therefore, the server can determine which data in the acquired basic column data needs to be deleted based on the stored row identifiers corresponding to the second data table.

[0131] Specifically, the server can determine the row identifier corresponding to the data to be deleted in the basic column data based on the stored row identifier, and delete the corresponding data in the basic column data according to the determined row identifier.

[0132] Furthermore, in one or more embodiments of this specification, since database transactions also require some time between execution and commit, in order to further ensure the accuracy of the query results of the query request, the server can further determine the transaction corresponding to the specified operation that triggered the recording of the first timestamp when it is determined that the second timestamp is greater than the first timestamp. Since the server can determine the first timestamp and store the correspondence with the second household table as long as the specified operation is executed, the server can determine the most recently recorded first timestamp and the transaction to which the specified operation that triggered the recording of the first timestamp belongs. Then, the transaction described in the query operation received in step S106 is determined, and it is determined whether the transaction described in the query operation and the transaction to which the specified operation that triggered the recording of the first timestamp belongs are the same transaction. That is, it is determined whether the most recently executed specified operation and the received query request are different operations within the same database transaction.

[0133] If so, it means that the most recently executed database transaction is the one corresponding to the query request. Since statements within the same database transaction can only be executed sequentially, no other specified operations will be executed before the query request is completed. Therefore, the server can determine the row identifiers whose storage time is no later than the first timestamp from the stored row identifiers, and use them as the row identifiers corresponding to the data that needs to be deleted.

[0134] If the query operation and the most recently executed specified operation do not belong to the same transaction, the row identifiers stored in the second data table may include database transactions that were committed before the query request was received, or database transactions that were executed but not yet committed before the query request was received. Obviously, if other database transactions have been executed but not committed, the query request should not reveal the modifications made to the data by such transactions. Therefore, it is necessary to determine the row identifiers of the data to be deleted based on other committed database transactions.

[0135] The server needs to determine the modifications made by database transactions committed before the query request, specifically the row identifiers of the rows operated on by the specified operations corresponding to those database transactions committed before the query request. Therefore, the server can determine the transactions committed before the second timestamp based on the recorded commit times of each specified operation, and then determine the row identifiers of the data to be deleted based on the row identifiers corresponding to the transactions committed before the second timestamp.

[0136] The server can be a database with Multiversion Concurrency Control (MVCC) functionality. In this case, the server can record the commit time of each database transaction. Therefore, based on the commit time of each database transaction, it can determine the database transactions committed before the second timestamp, and determine which specified operations correspond to these database transactions. Based on the row identifiers of the data operated on by these specified operations, it can determine the row identifiers of the data that need to be deleted.

[0137] Furthermore, in the embodiments of this specification, when the server determines that the second timestamp is not greater than the first timestamp corresponding to the second data table, it can determine that the query request was sent before some of the specified operations. Therefore, it needs to determine the query result based on the data in the database at the time the query request was sent. Similarly, using the MVCC function, the server can determine the transactions committed before the second timestamp based on the transaction commit times corresponding to each specified operation, and determine the row identifier corresponding to the data to be deleted based on the row identifier corresponding to the transactions committed before the second timestamp. In other words, the server can identify other transactions different from the transaction to which the query request belongs, and determine the row identifier corresponding to the transactions committed before the second timestamp within those other transactions as the row identifier corresponding to the data to be deleted.

[0138] Furthermore, the query request may also be a query request within an uncommitted transaction. Therefore, the server can also determine whether the database transaction to which the query request belongs has not yet been committed. If so, it determines the specified operation of the transaction to which the query operation belongs, and determines the row identifier corresponding to the data to be deleted based on the row identifier of the data operated on by the determined specified operation. Then, based on the row identifiers of transactions committed before the second timestamp and the row identifiers of the data rows to which the specified operation of the transaction to which the query operation belongs, it determines the row identifier corresponding to the data to be deleted.

[0139] In addition, in one or more embodiments of this specification, the server can determine whether to continue the query operation by comparing the second timestamp with the first timestamp.

[0140] Specifically, step S104 mentions that the server needs a certain amount of time to complete the specified operation. The query operation can be performed before or after the server executes the specified operation to obtain the correct query result. Therefore, in step S106, the server can first determine the second timestamp carried by the query operation, and then determine whether the second timestamp is greater than the first timestamp. If so, the server determines the basic column data in the second data table based on the column identifier; otherwise, it returns an error message and terminates the current query operation.

[0141] Timestamps are characters that increase in chronological order. For example, 9:00 AM on November 20, 2022, can have a timestamp of 9. 9:00 PM on November 20, 2022, can have a timestamp of 21. Therefore, the timestamp 21 for 9:00 PM on November 20, 2022, is greater than the timestamp for 9:00 AM on November 20, 2022. In other words, 9:00 AM on November 20, 2022, is earlier than 9:00 PM on November 20, 2022. Error messages can be messages sent to the user who submitted the query or other parties indicating that the data is unavailable or that other operations are currently being performed on the data. This manual does not impose any restrictions on this.

[0142] Regarding step S102, in one or more embodiments of this specification, when the first data table stores a large amount of data, if all the data in the first data table is created in a second data table, during subsequent query operations, since the row identifiers in the second data table may be unordered, the server may need to traverse the second data table to determine the corresponding supplementary data, which takes a long time. Therefore, the server can create multiple column storage data tables corresponding to the first data table, according to a preset number of rows, based on the data stored in each column of the first data table, as each second data table.

[0143] Specifically, the server can divide the data stored in each column of the first data table into multiple data blocks according to a preset number of rows. For each data block, the server creates a column-based data table corresponding to that data block, which serves as the second data table, based on the data stored in each column of that data block.

[0144] Figure 2a This is a schematic diagram of the first data table, as shown below. Figure 2aAs shown, there are 10 different mobile phone models, designated A, B, C, D, etc. The first data table stores the model number, body color, screen size, and corresponding row identifiers 1-10 of these 10 phones in row-based storage. Assuming a preset number of rows of 2, the server divides the first data table into 5 data blocks, each containing two rows of data and their corresponding row identifiers. Each data block is then used to generate a column-oriented data table, resulting in 5 second data tables.

[0145] The server can also create multiple column-oriented storage tables corresponding to the first data table, based on a preset data volume, as various second data tables. This specification does not impose any limitations on this.

[0146] Regarding step S104, in one or more embodiments of this specification, after the server establishes multiple second data tables, it determines the row identifier of the row in the first data table where the data operated on by the specified operation is located. Based on the row identifier, it determines the second data table corresponding to the row identifier in each second data table as a storage data table, determines the correspondence between the row identifier and the storage data table, and stores it.

[0147] Continue Figure 2a , Figure 2b For example, for each second data table, each row of data in the second data table has a corresponding row identifier. When the row identifier of the row containing the data operated on by the specified operation in the first data table is 1, then the second data table containing row identifiers ranging from 1 to 2 is the storage data table, storing the correspondence between the row identifier and the storage data table (i.e., the second data table). As long as the unique corresponding second data table can be correctly found based on the row identifier of the row containing the data operated on by the specified operation in the first data table, this specification does not impose any restrictions. Since the server can determine the range of row identifiers corresponding to the data stored in each second data table, it facilitates the server's subsequent determination of the corresponding second data table based on the row identifier, saving query time and improving query efficiency.

[0148] Regarding step S106, in one or more embodiments of this specification, when multiple second data tables exist, the server determines the row identifier carried by the query operation, and determines the second data table corresponding to the row identifier as the query data table based on the stored row identifier and the correspondence between the row identifier and the second data table. The server then determines the basic column data in the query data table based on the column identifier carried by the query operation.

[0149] Continue Figure 2a , Figure 2b For example, when the row identifier of the data to be operated on in the first data table is 2, in Figure 2bThe first second data table contains row identifiers ranging from 1 to 2. This second data table, containing row identifiers ranging from 1 to 2, is a storage data table, storing the correspondence between the row identifiers and the stored data table. When the server determines that the row identifier carried by the query operation is 2, it determines, based on the stored correspondence between the row identifier and the second data table, that the second data table corresponding to that row identifier is the second data table containing row identifiers ranging from 1 to 2, and uses this second data table as the query data table. The server then determines the base column data in this query data table based on the column identifier.

[0150] Regarding step S104, in one or more embodiments of this specification, when the type of the specified operation is a delete operation or an update operation, the server can determine the row identifier of the row containing the data operated on by the specified operation in the first data table based on the data operated on by the specified operation. The server stores the correspondence between the row identifier and the type of the specified operation. This correspondence means that if the type of the specified operation is a delete operation or an update operation, then the row identifier has a corresponding flag value. This specification does not limit the value of the flag value, as long as the server can determine which specified operation was performed on the row based on the flag value.

[0151] For example, if there are 10 rows of data with row identifiers from 1 to 10, and the server performs a deletion or update operation on the data of rows with row identifiers 3, 7, and 9, then the tag value of the rows with row identifiers 3, 7, and 9 will all be 0. The server stores the row identifiers 3, 7, and 9 and their corresponding tag values ​​0.

[0152] When the specified operation is an insert operation, the server can create a new row identifier corresponding to the insert operation. This row identifier is globally unique, meaning it is unique across all rows currently stored in the first data table, as well as rows that have been deleted. In other words, any row identifier that has appeared historically cannot be used as the new row identifier. Therefore, the server needs to record all historically occurring row identifiers to determine the possible values ​​for the new row identifier. Of course, the server can also reset the row identifiers for each row as needed; in this case, it can reset historically occurring row identifiers, i.e., delete all stored historically occurring row identifiers.

[0153] If the newly created row identifier is the same as an existing row identifier in the first data table, after the specified operation, if the server performs a query operation based on the row identifier, the server will query the required data from the data of multiple rows corresponding to the row identifier, resulting in redundant data in the query results. Therefore, the newly created row identifier must be different from the existing row identifier in the first data table.

[0154] Since the second data table can be updated according to preset conditions, if the server deletes an entire row of data according to the specified operation before updating the second data table, then the row identifier of that row will also be deleted after the update, resulting in a previously deleted row identifier. If the second data table contains deleted rows of data, and the server performs an add operation on the second data table, assuming the newly created row identifier is the same as the row identifier corresponding to the deleted row, then when the server updates the second data table, it will delete the data in the row with the newly created row identifier, resulting in a missing data situation in the updated second data table. Therefore, the newly created row identifier must be different from the previously deleted row identifier.

[0155] After the server creates a new row identifier, it can also store the correspondence between the newly created row identifier and the type of the specified operation.

[0156] When the specified operation type is an add operation, the server can determine the flag value corresponding to the row identifier. This flag value only needs to be different from the flag value when the specified operation type is a delete or update operation. This flag value can be used to determine the type of the specified operation. Furthermore, since the flag value of the row identifier is determined when the specified operation type is a delete or update operation, the server can also set the flag value of the row identifier when the specified operation type is an add operation. Subsequently, when the server needs to determine the type of the specified operation, it can determine it based on whether the row identifier has a flag value.

[0157] Regarding step S108, in one or more embodiments of this specification, the server determines the row identifier and the type of the specified operation corresponding to it based on the stored row identifier. That is, the server determines the flag value of the row identifier based on the row identifier and determines the specified operation type corresponding to the row identifier based on the flag value of the row identifier.

[0158] Assuming the specified operation type is a delete or update operation, the flag value of this row identifier is 0; when the specified operation type is an add operation, the flag value of this row identifier is not set. The server determines whether the row identifier has a flag value to determine whether the specified operation type corresponding to this row identifier is an add, delete, or update operation. If the row identifier does not have a flag value, the specified operation type corresponding to this row identifier is an add operation; if the row identifier has a flag value, the specified operation type corresponding to this row identifier is a delete or update operation.

[0159] When the specified operation is a delete or update operation, the server deletes the data corresponding to the row identifier in the basic column data according to the row identifier, and queries the data corresponding to the row identifier from the first data table according to the row identifier, and uses the query result as supplementary data.

[0160] When the specified operation is an add operation, the data corresponding to the row identifier is determined from the first data table based on the row identifier and used as supplementary data.

[0161] Regarding step S106, in one or more embodiments of this specification, since the server handles a large number of transactions, it will perform a large number of specified operations on the database. The second data table stores a large number of correspondences, such as the correspondence between row identifiers and the type of specified operation, and the correspondence between newly created row identifiers and the type of specified operation, etc., which occupies a significant amount of memory space. If the data in the second data table is not updated for a long time, determining these correspondences during query operations on the second data table will also waste a lot of resources and time. Therefore, the server can update the second database according to preset conditions.

[0162] Specifically, assuming the preset condition is to update the second data table every 12 hours, the server determines the time corresponding to the timestamp of the second data table and the current time, and decides whether to update the second data table. If the time corresponding to the timestamp of the second data table is 12 hours away from the current time, the preset condition is met. Then, the server updates the second data table according to the stored row identifiers and the first data table to obtain the third data table. That is, the following steps are performed: For each stored row identifier, the server determines the type of the specified operation corresponding to the row identifier. When the type of the specified operation is an insert operation, the server determines the data corresponding to the row identifier from the first data table according to the row identifier, and inserts the row identifier and the data corresponding to the row identifier into the second data table to obtain the third data table.

[0163] When the specified operation type is a delete operation, the data corresponding to the row identifier in the second data table is deleted according to the row identifier, and a third data table is obtained.

[0164] When the specified operation is an update operation, the data corresponding to the row identifier in the second data table is deleted according to the row identifier. Then, the data corresponding to the row identifier is determined from the first data table according to the row identifier, and the row identifier and the data corresponding to the row identifier are inserted into the second data table to obtain the third data table.

[0165] After obtaining the third data table, the server stores the second data table before the update, the third data table, and the third timestamp, which is the time when the third data table was created.

[0166] If the preset conditions are not met, the second data table will not be updated. The preset conditions may be time, the amount of data in the second data table, etc., and this manual does not impose any restrictions on them.

[0167] Because there may be a time difference between when the server receives the query and when the query is submitted—for example, if a user or other party submits a query at 9:00 AM on November 20, 2022, but the server's threads are processing other tasks and the query is received at 10:00 PM on November 20, 2022—the server should be querying data from the second data table at 9:00 AM on November 20, 2022. Therefore, when executing the query, the server first determines the second timestamp carried by the query, then checks if the second timestamp is greater than the third timestamp. If it is, the server determines the base column data in the third data table based on the column identifier and deletes the second data table. If not, the server determines the base column data in the second data table based on the column identifier.

[0168] Figure 3 This is a schematic diagram of the internal structure of the second data table, as shown below. Figure 3 As shown, in one or more embodiments of this specification, assuming a preset number of rows is 4, the server creates a columnar storage data table for every 4 rows of data in the first data table, i.e., creates multiple second data tables. For each second data table, the second data table stores the data corresponding to the first data table, the row identifier of the data in the first data table, the flag value corresponding to the row identifier, and the correspondence between the row identifier and the second data table. Specifically, when the flag value corresponding to the row identifier is 1, it indicates that the server has performed a deletion operation on the row with that row identifier; when the flag value corresponding to the row identifier is 0, it indicates that the server has not performed an operation on the row with that row identifier. The server splits the update operation into a deletion operation and an insert operation; that is, when an update operation is performed on the first data table, the flag value of the row identifier for that row will also be 1. When the server performs an insert operation on the first data table, it will store the row identifier of that row in the second data table.

[0169] Figure 4 A diagram illustrating the execution of a query operation, such as... Figure 4 As shown, when the server receives a query operation, it first determines the second data table corresponding to the row identifier based on the stored row identifier and its correspondence with each second data table. Then, the server uses the column identifier carried in the query operation to determine the basic column data in that second data table. Figure 4 In the second data table, the shaded rectangle represents the data whose row identifier is marked as 1. The server then deletes data with a row identifier of 1 from this base column data and retrieves the corresponding data from the first data table based on the stored row identifier. In other words, it identifies the corresponding data from the shaded rectangle in the first data table based on the stored row identifier, using this as supplementary data. Figure 4In the first data table, the shaded rectangle overlaps with the dashed rectangle frame. Finally, the server returns the supplementary data and the deleted base column data as the query results to the user or other querying party. It can be seen that the server stores the row identifiers as tag values, which can be considered a tag vector. Tag values ​​0 and 1 indicate which rows' row identifiers are stored. The number of bits in this tag vector matches the number of rows in the second data table. Therefore, when deleting data corresponding to a row identifier, data in the base column data can be deleted based on this tag vector, improving query efficiency.

[0170] The above describes a data query method provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding data query device, such as... Figure 5 As shown.

[0171] Figure 5 This specification provides a schematic diagram of a data query device, the device comprising:

[0172] The first determining module 400 is used to determine the first data table of row storage;

[0173] The second determining module 402 is used to establish a column storage data table corresponding to the first data table based on the data stored in each column of the first data table, as the second data table;

[0174] The row identifier determination module 404 is used to determine and store the row identifier of the data operated on by the specified operation in response to a specified operation on the first data table, wherein the type of the specified operation includes one of addition, deletion, and update;

[0175] The column determination module 406 is used to determine the basic column data in the second data table according to the column identifier when a query operation carrying a column identifier is received;

[0176] Data processing module 408 is used to delete the data corresponding to the row identifier in the basic column data according to the stored row identifier, and to determine the data corresponding to the row identifier from the first data table as supplementary data;

[0177] The query result determination module 410 is used to determine the data query result based on the base column data after data deletion and the supplementary data.

[0178] Optionally, the row identifier determination module 404 is specifically used to determine the row identifier of the data operated on by the specified operation in the first data table when the type of the specified operation is a deletion operation or an update operation, and to store the correspondence between the row identifier and the type of the specified operation; when the type of the specified operation is an add operation, to create a new row identifier, and to store the correspondence between the new row identifier and the type of the specified operation, wherein the new row identifier is different from the existing row identifiers in the first data table and the row identifiers of historical deletions.

[0179] Optionally, the data processing module 408 is specifically configured to determine the type of the specified operation for each stored row identifier; when the type of the specified operation is a delete operation or an update operation, delete the data corresponding to the row identifier in the basic column data according to the row identifier, and query the data corresponding to the row identifier from the first data table according to the row identifier, and use the query result as supplementary data; when the type of the specified operation is an add operation, determine the data corresponding to the row identifier from the first data table according to the row identifier, and use it as supplementary data.

[0180] Optionally, the row identifier determination module 404 is specifically used to determine and store the row identifier of the data operated by the specified operation when the specified operation is received, use the time of storing the row identifier as the first timestamp, and store the correspondence between the second data table and the first timestamp.

[0181] Optionally, the data processing module 408 is specifically used to determine the second timestamp carried by the query operation, determine whether the second timestamp is greater than the first timestamp, and if so, determine the row identifier corresponding to the data to be deleted according to the stored row identifier, and delete the data corresponding to the row identifier in the basic column data according to the determined row identifier.

[0182] Optionally, the data processing module 408 is specifically used to determine the transaction corresponding to the specified operation that triggered the recording of the first timestamp, and to determine whether the transaction to which the query operation belongs is the same transaction as the determined transaction. If so, it determines the row identifier whose storage time is no later than the first timestamp from the stored row identifiers, and uses it as the row identifier corresponding to the data to be deleted. If not, it determines the transaction that was submitted before the second timestamp based on the submission time of the transaction corresponding to each specified operation recorded, and determines the row identifier corresponding to the data to be deleted based on the row identifier corresponding to the transaction that was submitted before the second timestamp.

[0183] Optionally, when the second timestamp is not greater than the first timestamp, the data processing module 408 is further configured to determine the transactions committed before the second timestamp based on the transaction commit time corresponding to each specified operation recorded, determine the row identifier corresponding to the data to be deleted based on the row identifier corresponding to the transactions committed before the second timestamp, and when it is determined that there is a transaction to which the query operation belongs among the transactions not committed before the second timestamp, determine the specified operation of the transaction to which the query operation belongs, and determine the row identifier corresponding to the data to be deleted based on the row identifier of the row where the data operated by the determined specified operation is located.

[0184] Optionally, the column determination module 406 is further configured to update the second data table according to the stored row identifiers and the first data table to obtain a third data table, and store the second data table before the update, the third data table, and a third timestamp, wherein the third timestamp is the time when the third data table was created. When determining the basic column data in the second data table according to the column identifier, it is specifically configured to determine the second timestamp carried by the query operation, and determine whether the second timestamp is greater than the third timestamp. If so, the basic column data is determined in the third data table according to the column identifier, and the second data table is deleted. If not, the basic column data is determined in the second data table according to the column identifier.

[0185] Optionally, the column determination module 406 is specifically used to determine the type of a specified operation corresponding to each stored row identifier. When the type of the specified operation is an add operation, the data corresponding to the row identifier is determined from the first data table according to the row identifier, and the row identifier and the data corresponding to the row identifier are inserted into the second data table. When the type of the specified operation is a delete operation, the data corresponding to the row identifier in the second data table is deleted according to the row identifier. When the type of the specified operation is an update operation, the data corresponding to the row identifier in the second data table is deleted according to the row identifier, and then the data corresponding to the row identifier is determined from the first data table according to the row identifier, and the row identifier and the data corresponding to the row identifier are inserted into the second data table.

[0186] Optionally, the second determining module 402 is further configured to establish multiple column storage data tables corresponding to the first data table according to the data stored in each column of the first data table and according to a preset number of rows, as each second data table;

[0187] The row identifier determination module 404 is specifically used to determine the row identifier of the row in the first data table when the row identifier of the data operated by the specified operation is determined and stored, and to determine the second data table corresponding to the row identifier in each second data table as a storage data table, and to determine and store the correspondence between the row identifier and the storage data table.

[0188] Optionally, the column determination module 406 is specifically used to determine the row identifier carried by the query operation, determine the second data table corresponding to the row identifier according to the stored correspondence between the row identifier and the second data table, and use it as the query data table, and determine the basic column data in the query data table according to the column identifier.

[0189] Optionally, the data processing module 408 is specifically used to determine the data corresponding to the row identifier from the first data table based on the row identifier, and to filter out the data that meets the query conditions from the data corresponding to the row identifier based on the query conditions carried in the query operation.

[0190] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided data query methods.

[0191] This instruction manual also provides Figure 6 The diagram shows the structure of the electronic device. Figure 6 At the hardware level, the autonomous driving device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for various operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The data query method described above. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0192] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0193] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0194] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0195] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0196] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0197] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0198] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0199] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0200] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0201] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0202] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0203] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0204] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0205] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0206] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0207] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this application.

Claims

1. A method for data querying, the method comprising: Determine the first data table in the row storage; Based on the data stored in each column of the first data table, a column storage data table corresponding to the first data table is created as the second data table; In response to a specified operation on the first data table, the row identifier of the row containing the data operated on by the specified operation is determined and stored, wherein the type of the specified operation includes one of addition, deletion, and update; When a query operation carrying a column identifier is received, the basic column data is determined in the second data table based on the column identifier; Based on the stored row identifier, delete the data corresponding to the row identifier in the basic column data, and determine the data corresponding to the row identifier from the first data table as supplementary data; The data query result is determined based on the base column data after data deletion and the supplementary data; Specifically, based on the stored row identifier, the data corresponding to the row identifier in the basic column data is deleted, and the data corresponding to the row identifier is determined from the first data table as supplementary data, including: For each row identifier stored, determine the row identifier and the type of the specified operation corresponding to it; When the specified operation is a delete operation or an update operation, the data corresponding to the row identifier in the basic column data is deleted according to the row identifier, and the data corresponding to the row identifier is queried from the first data table according to the row identifier, and the query result is used as supplementary data. When the specified operation is an add operation, the data corresponding to the row identifier is determined from the first data table based on the row identifier and used as supplementary data.

2. The method as described in claim 1, wherein determining and storing the row identifier of the row containing the data operated on by the specified operation, specifically includes: When the type of the specified operation is a delete operation or an update operation, determine the row identifier of the row in the first data table where the data operated by the specified operation is located, and store the correspondence between the row identifier and the type of the specified operation; When the specified operation is an add operation, a new row identifier is created, and the correspondence between the newly created row identifier and the specified operation type is stored. The newly created row identifier is different from the existing row identifiers in the first data table and the row identifiers of historical deletions.

3. The method as described in claim 1, wherein determining and storing the row identifier of the row containing the data operated on by the specified operation specifically includes: When the specified operation is received, the row identifier of the row containing the data operated on by the specified operation is determined and stored; The time when the row identifier is stored is used as the first timestamp, and the correspondence between the second data table and the first timestamp is stored.

4. The method as described in claim 3, wherein deleting the data corresponding to the row identifier in the basic column data according to the stored row identifier specifically includes: Determine the second timestamp carried by the query operation; Determine whether the second timestamp is greater than the first timestamp; If so, then based on the stored row identifier, determine the row identifier corresponding to the data to be deleted, and delete the corresponding data in the basic column data according to the determined row identifier.

5. The method as described in claim 4, wherein determining the row identifier corresponding to the data to be deleted based on the stored row identifier, specifically includes: Determine the transaction corresponding to the specified operation that triggered the recording of the first timestamp; Determine whether the transaction to which the query operation belongs is the same as the identified transaction; If so, then from the stored row identifiers, determine the row identifier whose storage time is no later than the first timestamp, and use it as the row identifier corresponding to the data to be deleted; If not, then based on the transaction commit time corresponding to each specified operation in the record, determine the transactions committed before the second timestamp, and based on the row identifier corresponding to the transactions committed before the second timestamp, determine the row identifier corresponding to the data to be deleted.

6. The method of claim 4, wherein when the second timestamp is not greater than the first timestamp, the method further comprises: Identify other transactions that are different from the transaction to which the query request belongs, and determine the row identifiers of the other transactions that were committed before the second timestamp, as the row identifiers of the data to be deleted; When it is determined that a transaction to which the query operation belongs exists within an uncommitted transaction, the specified operation of the transaction to which the query operation belongs is determined; Based on the row identifier of the row containing the data to be operated on, determine the row identifier of the data to be deleted.

7. The method of claim 1, further comprising: Based on the stored row identifiers and the first data table, update the second data table to obtain the third data table; The system stores the second data table before the update, the third data table, and the third timestamp, where the third timestamp is the time when the third data table was created. The basic column data is determined in the second data table based on the column identifier, specifically including: Determine the second timestamp carried by the query operation; Determine whether the second timestamp is greater than the third timestamp; If so, determine the base column data in the third data table according to the column identifier, and delete the second data table; If not, the base column data is determined in the second data table based on the column identifier.

8. The method as described in claim 7, wherein updating the second data table based on the stored row identifiers and the first data table specifically includes: For each row identifier in the storage, determine the type of the specified operation corresponding to that row identifier; When the specified operation is an add operation, the data corresponding to the row identifier is determined from the first data table according to the row identifier, and the row identifier and the data corresponding to the row identifier are inserted into the second data table; When the specified operation is a delete operation, the data corresponding to that row identifier in the second data table is deleted according to the row identifier. When the specified operation is an update operation, the data corresponding to the row identifier in the second data table is deleted according to the row identifier. Then, the data corresponding to the row identifier is determined from the first data table according to the row identifier, and the row identifier and the data corresponding to the row identifier are inserted into the second data table.

9. The method of claim 1, further comprising: Based on the data stored in each column of the first data table, multiple column storage data tables corresponding to the first data table are created according to a preset number of rows, serving as each second data table; Determine and store the row identifier of the row containing the data operated on by the specified operation, specifically including: Determine the row identifier of the row in the first data table where the data to be operated on by the specified operation is located; Based on the row identifier, determine the second data table corresponding to the row identifier in each second data table, and use it as the storage data table; Determine the correspondence between the row identifier and the stored data table, and store it.

10. The method of claim 9, wherein determining the basic column data in the second data table based on the column identifier specifically includes: Determine the row identifier carried by the query operation; Based on the stored correspondence between the row identifier and the second data table, determine the second data table corresponding to the row identifier, and use it as the query data table; Based on the column identifier, the base column data is determined in the query data table.

11. The method of claim 1, wherein determining the data corresponding to the row identifier from the first data table specifically includes: Based on the row identifier, determine the data corresponding to the row identifier from the first data table; Based on the query conditions carried in the query operation, data that meets the query conditions is filtered out from the data corresponding to the row identifier.

12. A data query apparatus, the apparatus comprising: The first determining module is used to determine the first data table of the row storage; The second determining module is used to establish a column storage data table corresponding to the first data table based on the data stored in each column of the first data table, as the second data table; The row identifier determination module is used to determine and store the row identifier of the row containing the data operated on by the specified operation in response to a specified operation on the first data table, wherein the type of the specified operation includes one of addition, deletion, and update; The column determination module is used to determine the basic column data in the second data table based on the column identifier when a query operation carrying a column identifier is received; The data processing module is used to delete the data corresponding to the row identifier in the basic column data according to the stored row identifier, and to determine the data corresponding to the row identifier from the first data table as supplementary data; The query result determination module is used to determine the data query result based on the base column data after data deletion and the supplementary data; Specifically, the data processing module is used to determine the type of the specified operation for each stored row identifier; when the type of the specified operation is a delete operation or an update operation, it deletes the data corresponding to the row identifier in the basic column data according to the row identifier, and queries the data corresponding to the row identifier from the first data table according to the row identifier, and uses the query result as supplementary data; when the type of the specified operation is an add operation, it determines the data corresponding to the row identifier from the first data table according to the row identifier, and uses it as supplementary data.

13. The apparatus of claim 12, wherein the row identifier determination module is specifically configured to, when the type of the specified operation is a deletion operation or an update operation, determine the row identifier of the row in the first data table where the data operated by the specified operation is located, and store the correspondence between the row identifier and the type of the specified operation; when the type of the specified operation is an add operation, create a new row identifier, and store the correspondence between the newly created row identifier and the type of the specified operation, wherein the newly created row identifier is different from the existing row identifiers in the first data table and the row identifiers of historical deletions.

14. The apparatus of claim 12, wherein the row identifier determination module is specifically configured to, when receiving the specified operation, determine and store the row identifier of the row in which the data operated by the specified operation is located, use the time of storing the row identifier as a first timestamp, and store the correspondence between the second data table and the first timestamp.

15. The apparatus of claim 14, wherein the data processing module is specifically configured to determine the second timestamp carried by the query operation, determine whether the second timestamp is greater than the first timestamp, and if so, determine the row identifier corresponding to the data to be deleted based on the stored row identifier, and delete the data corresponding to the row identifier in the basic column data based on the determined row identifier.

16. The apparatus of claim 15, wherein the data processing module is specifically configured to determine the transaction corresponding to the specified operation that triggers the recording of the first timestamp, determine whether the transaction to which the query operation belongs is the same transaction as the determined transaction, and if so, determine the row identifier whose storage time is no later than the first timestamp from the stored row identifiers as the row identifier corresponding to the data to be deleted; if not, determine the transaction submitted before the second timestamp according to the submission time of the transaction corresponding to each specified operation recorded, and determine the row identifier corresponding to the data to be deleted according to the row identifier corresponding to the transaction submitted before the second timestamp.

17. The apparatus of claim 15, wherein when the second timestamp is not greater than the first timestamp, the data processing module is further configured to: determine the transactions committed before the second timestamp based on the transaction commit times corresponding to each specified operation recorded; determine the row identifier corresponding to the data to be deleted based on the row identifier corresponding to the transactions committed before the second timestamp; when it is determined that there is a transaction to which the query operation belongs among the transactions not committed before the second timestamp; determine the specified operation of the transaction to which the query operation belongs; and determine the row identifier corresponding to the data to be deleted based on the row identifier of the row where the data operated by the determined specified operation is located.

18. The apparatus of claim 12, wherein the column determination module is further configured to update the second data table according to the stored row identifiers and the first data table to obtain a third data table, and to store the second data table before the update, the third data table, and a third timestamp, wherein the third timestamp is the time when the third data table is created. When determining the basic column data in the second data table according to the column identifier, the module is specifically configured to determine the second timestamp carried by the query operation, determine whether the second timestamp is greater than the third timestamp, and if so, determine the basic column data in the third data table according to the column identifier and delete the second data table; if not, determine the basic column data in the second data table according to the column identifier.

19. The apparatus of claim 18, wherein the column determination module is specifically configured to, for each stored row identifier, determine the type of a specified operation corresponding to that row identifier; when the type of the specified operation is an add operation, determine the data corresponding to that row identifier from the first data table according to the row identifier, and insert the row identifier and the data corresponding to that row identifier into the second data table; when the type of the specified operation is a delete operation, delete the data corresponding to that row identifier from the second data table according to the row identifier; when the type of the specified operation is an update operation, delete the data corresponding to that row identifier from the second data table according to the row identifier, then determine the data corresponding to that row identifier from the first data table according to the row identifier, and insert the row identifier and the data corresponding to that row identifier into the second data table.

20. The apparatus of claim 12, wherein the second determining module is further configured to establish, according to the data stored in each column of the first data table and according to a preset number of rows, a plurality of column storage data tables corresponding to the first data table, as each second data table; The row identifier determination module is specifically used to determine the row identifier of the row in the first data table when the row identifier of the data operated by the specified operation is determined and stored, and based on the row identifier, determine the second data table corresponding to the row identifier in each second data table as a storage data table, determine the correspondence between the row identifier and the storage data table, and store it.

21. The apparatus of claim 20, wherein the column determination module is specifically configured to determine the row identifier carried by the query operation, determine the second data table corresponding to the row identifier as a query data table based on the stored correspondence between the row identifier and the second data table, and determine the basic column data in the query data table based on the column identifier.

22. The apparatus of claim 12, wherein the data processing module is specifically configured to determine, based on the row identifier, the data corresponding to the row identifier from the first data table, and to filter out data that meets the query conditions from the data corresponding to the row identifier based on the query conditions carried in the query operation.

23. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 11.

24. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1 to 11.

Images