Data processing method and device, electronic equipment and computer readable storage medium

Abstract

The present disclosure provides a data processing method and device, electronic equipment and computer readable storage medium, which can be applied to the field of big data technology and also can be applied to the field of financial technology. The data processing method comprises: reading a target index tuple from at least one index tuple in an index table based on a target query value, wherein the index tuple comprises a first index field and a second index field, at least one first index field value of the at least one index tuple is different, so as to identify and distinguish different index tuples, and the second index field is used to locate a chain head tuple from a chain head table; reading the chain head tuple from the chain head table based on the second index field value of the target index tuple, wherein the chain head tuple comprises a chain head field; and reading a first target data tuple from a data table based on the chain head field.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of big data, and in particular, to a data processing method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] A hot record in a database refers to a record that is frequently updated in a unit of time. In daily business processing, performing update, query, deletion and the like on hot data in the database is a data operation behavior that is often needed to be performed.

[0003] In the process of implementing the present disclosure, the inventors have found that the existing database storage structure and data operation method cause some problems, for example: as the number of updates of the same record increases, the number of data access jumps will also increase, the query will become slower, and the update of the hot record will increase the concurrency, causing the same record to traverse multiple versions and perform multiple locking and releasing processes, which magnifies the hot spot problem from one point to multiple points, resulting in a decline in overall performance. SUMMARY

[0004] In view of the above problems, the present disclosure provides a data processing method, device, equipment, medium and program product.

[0005] In one aspect of the present disclosure, a data processing method is provided, comprising:

[0006] reading a target index tuple from at least one index tuple in an index table based on a target query value, wherein the index tuple comprises a first index field and a second index field, at least one first index field value of the at least one index tuple is different, so as to distinguish different index tuples, and the second index field is used to locate a chain head tuple from a chain head table;

[0007] reading a chain head tuple from the chain head table based on a second index field value of the target index tuple, wherein the chain head tuple comprises a chain head field, and the value of the chain head field is the storage location of the latest version of the data tuple in the data table in the plurality of versions of the data tuple in the data table;

[0008] reading a first target data tuple from the data table based on the chain head field, wherein the first target data tuple is the current latest version of the data tuple.

[0009] According to an embodiment of the present disclosure, reading a target index tuple from at least one index tuple in an index table based on a target query value comprises:

[0010] reading an index tuple with the same first index field value as the target query value from the at least one index tuple as the target index tuple.

[0011] According to an embodiment of the present disclosure, wherein:

[0012] The at least one second index field value of the at least one index tuple is a storage location of the chain head tuple in the chain head table, so that the chain head tuple can be located by the second index field of any index tuple.

[0013] According to an embodiment of the present disclosure, the method further comprises:

[0014] In the case of needing to update the first target data tuple, an exclusive lock of the chain head tuple is acquired;

[0015] A second target data tuple is added in the data table, wherein the second target data tuple is a data tuple after updating the first target data tuple;

[0016] The value of the chain head field is updated to a storage location of the second target data tuple in the data table.

[0017] According to an embodiment of the present disclosure, the method further comprises:

[0018] The length of a lock waiting queue of the exclusive lock of the chain head tuple is acquired;

[0019] In the case that the length of the lock waiting queue is greater than a preset threshold, the first target data tuple is determined as a hot data.

[0020] According to an embodiment of the present disclosure, wherein:

[0021] The data tuple comprises a first data field, a plurality of first data field values of the plurality of versions of the data tuple, respectively being the main body data of each version of the data tuple, and the plurality of versions of the main body data are different from each other but contain the same data primary key.

[0022] According to an embodiment of the present disclosure, wherein:

[0023] The plurality of data key values of the data primary key under the plurality of versions are the same or different;

[0024] The at least one first index field value of the at least one index tuple matches the plurality of data key values.

[0025] According to an embodiment of the present disclosure, the method further comprises:

[0026] In the case that the key value of the data primary key of the second target data tuple is different from the key value of the data primary key of the first target data tuple, an index tuple is added in the index table, wherein the value of the first index field of the added index tuple is the key value of the data primary key of the second target data tuple.

[0027] According to an embodiment of the present disclosure, wherein:

[0028] The data tuple further comprises a second data field, and a plurality of second data field values of the plurality of versions of the data tuple are respectively a storage location of a previous version of the data tuple of each version in the data table, so that the previous version of the data tuple can be linked to respectively by each second data field value.

[0029] According to an embodiment of the present disclosure, wherein:

[0030] The data tuple further comprises a third data field, and a plurality of third data field values of the plurality of versions of the data tuple are respectively used to represent whether the data tuple of each version is deleted.

[0031] According to an embodiment of the present disclosure, the method further comprises:

[0032] In a case where the third data field value of the second target data tuple is a first value and the second target data tuple needs to be deleted, an exclusive lock of the head tuple is acquired, wherein the first value is used to represent that the second target data tuple is not deleted before the deletion operation is performed;

[0033] A third target data tuple is added in the data table, wherein the third data field value of the third target data tuple is a second value, and the first data field value of the third target data tuple is empty, and the second value is used to represent that the second target data tuple is deleted after the deletion operation is performed;

[0034] The value of the head field is updated to a storage location of the third target data tuple in the data table.

[0035] According to an embodiment of the present disclosure, wherein:

[0036] The data tuple further comprises a fourth data field, and a plurality of fourth data field values of the plurality of versions of the data tuple are respectively an operation transaction number of the data tuple of each version.

[0037] Another aspect of the present disclosure provides a data processing apparatus, comprising a first reading module, a second reading module and a third reading module.

[0038] The first reading module is configured to read a target index tuple from at least one index tuple in an index table based on a target query value, wherein the index tuple comprises a first index field and a second index field, at least one first index field value of the at least one index tuple is different from each other, so as to identify and distinguish different index tuples, and the second index field is used to locate a head tuple from a head table.

[0039] The second reading module is configured to read the head tuple from the head table based on a second index field value of the target index tuple, wherein the head tuple comprises a head field, and a value of the head field is a storage location of a latest version of a data tuple in the plurality of versions of the data tuple in the data table.

[0040] The third reading module is configured to read, based on the chain head field, a first target data tuple from the data table, wherein the first target data tuple is a current latest version of data tuple.

[0041] According to an embodiment of the present disclosure, the first reading module comprises a reading unit configured to read, from the at least one index tuple, an index tuple having a same first index field value as the target query value as the target index tuple.

[0042] According to an embodiment of the present disclosure, at least one second index field value of the at least one index tuple is a storage location of the chain head tuple in the chain head table, so that the chain head tuple can be located by the second index field of any index tuple.

[0043] According to an embodiment of the present disclosure, the apparatus further comprises a first obtaining module, a first adding module and a first updating module.

[0044] The first obtaining module is configured to obtain an exclusive lock of the chain head tuple in a case where the first target data tuple needs to be updated.

[0045] The first adding module is configured to add a second target data tuple in the data table, wherein the second target data tuple is a data tuple after updating the first target data tuple.

[0046] The first updating module is configured to update a value of the chain head field to a storage location of the second target data tuple in the data table.

[0047] According to an embodiment of the present disclosure, the apparatus further comprises a second obtaining module and a determining module.

[0048] The second obtaining module is configured to obtain a length of a lock waiting queue of the exclusive lock of the chain head tuple.

[0049] The determining module is configured to determine the first target data tuple as a hotspot data in a case where the length of the lock waiting queue is greater than a preset threshold.

[0050] According to an embodiment of the present disclosure, the data tuple comprises a first data field, a plurality of first data field values of the plurality of versions of data tuples are respectively main data of the respective versions of data tuples, and the plurality of versions of main data are different from each other but contain a same data primary key.

[0051] According to an embodiment of the present disclosure, the data primary key has a same or different plurality of data key values in the plurality of versions.

[0052] The at least one first index field value of the at least one index tuple matches the plurality of data key values.

[0053] According to an embodiment of the present disclosure, the apparatus further includes a second adding module configured to add an index tuple in the index table when the key value of the data primary key of the second target data tuple is different from the key value of the data primary key of the first target data tuple, wherein the value of the first index field of the added index tuple is the key value of the data primary key of the second target data tuple.

[0054] According to an embodiment of the present disclosure, the data tuple further includes a second data field, and the plurality of second data field values of the plurality of versions of the data tuple are respectively the storage locations of the previous versions of the data tuple in the data table, so that each second data field value can be linked to the previous version of the data tuple.

[0055] According to an embodiment of the present disclosure, the data tuple further includes a third data field, and the plurality of third data field values of the plurality of versions of the data tuple are respectively used to represent whether the data tuple of each version is deleted.

[0056] According to an embodiment of the present disclosure, the apparatus further includes a third obtaining module, a third adding module, and a second updating module.

[0057] The third obtaining module is configured to obtain an exclusive lock of the head tuple when the third data field value of the second target data tuple is a first value and the second target data tuple needs to be deleted, wherein the first value is used to represent that the second target data tuple is not deleted before the deletion operation is performed.

[0058] The third adding module is configured to add a third target data tuple in the data table, wherein the third data field value of the third target data tuple is a second value, and the first data field value of the third target data tuple is empty, and the second value is used to represent that the second target data tuple is deleted after the deletion operation is performed.

[0059] The second updating module is configured to update the value of the head field to the storage location of the third target data tuple in the data table.

[0060] According to an embodiment of the present disclosure, the data tuple further includes a fourth data field, and the plurality of fourth data field values of the plurality of versions of the data tuple are respectively the operation transaction numbers of the data tuple of each version.

[0061] Another aspect of the present disclosure provides an electronic device, including one or more processors, a memory configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the above data processing method.

[0062] Another aspect of the present disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the above data processing method.

[0063] Another aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the above data processing method.

[0064] According to embodiments of the present disclosure, the database storage structure of the embodiments of the present disclosure is optimized and improved, because the second index field in the index tuple can be used to locate the head tuple from the head table, and because the value of the head field is the storage location of the latest version of the data tuple in the data table. Therefore, based on the above data storage structure of the embodiments of the present disclosure, by executing the above data processing method of the embodiments of the present disclosure, the latest version of the data tuple can be quickly queried. The above method of the embodiments of the present disclosure, on the basis of retaining the original database to store new and old versions, by adjusting the arrangement mode of new and old versions of data and the index storage mode, avoids the problem that the query becomes slower with the increase of versions, solves the problem that the query becomes slower with the increase of updated versions under the database multi-version concurrency control mechanism, and greatly improves the processing efficiency of hot data query. BRIEF DESCRIPTION OF DRAWINGS

[0065] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure taken in conjunction with the accompanying drawings, in which:

[0066] Figure 1 A schematic diagram illustrating the principle of performing data operation according to the database storage structure in the related art is shown;

[0067] Figure 2 A trend chart showing the change of lock time with the heat of hot record in the process of hot record data operation is shown;

[0068] Figure 3 A schematic diagram illustrating the application scenario of the data processing method, device, equipment, medium and program product according to embodiments of the present disclosure is shown;

[0069] Figure 4 A flowchart illustrating the data processing method according to an embodiment of the present disclosure is shown;

[0070] Figure 5 A schematic diagram illustrating the database storage structure according to an embodiment of the present disclosure is shown;

[0071] Figure 6 A flowchart illustrating the data processing method according to another embodiment of the present disclosure is shown;

[0072] Figure 7A flowchart of a data processing method according to yet another embodiment of the present disclosure is schematically shown;

[0073] Figure 8 A schematic diagram of a principle of performing data operations by a database storage structure according to an embodiment of the present disclosure is shown;

[0074] Figure 9 A block diagram of a structure of a data processing apparatus according to an embodiment of the present disclosure is shown; and

[0075] Figure 10 A block diagram of an electronic device suitable for implementing the data processing method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0076] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely exemplary of the present disclosure, and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present disclosure.

[0077] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so forth, mean the term "comprises," as long as the above terms do not themselves appear herein.

[0078] All terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure pertains, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of the specification, and should not be interpreted in an idealized or overly formal manner.

[0079] In the case of using expressions similar to "at least one of A, B, and C, etc.", it is generally to be interpreted as including one or more of the same. For example, "a system having at least one of A, B, and C" should be interpreted as including a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C together, etc.

[0080] There are two methods for storing new and old versions of data in the current database implementing the multi-version concurrency control mechanism. One is to separately store the new and old versions of data, and store the new and old versions of data in different areas. When updating data, the original position is updated to new data, and the old data is moved to a centralized area. When rolling back, the old data needs to be updated back to the original position. Oracle and Mysql use this method to implement it. The second is to centrally store the new and old versions of data: the new and old versions of data are stored in the same area. When writing new data, the old data is not deleted, and new data is inserted. The transaction number state is used to determine transaction submission or rollback. Some databases, such as PostgreSQL, use the centralized storage method for multi-version data.

[0081] Figure 1 A schematic diagram illustrating the principle of performing data operations according to the database storage structure in the related art is shown.

[0082] As shown in Figure 1 , in the related art, for the centralized storage method, the database storage structure includes two parts of data tuples and data indexes. The data tuple structure for storing a row of data includes two parts of a tuple header and tuple content. The tuple header mainly includes:

[0083] xmin: transaction number for inserting the tuple, which can be understood as the effective transaction number of the tuple.

[0084] xmax: transaction number for deleting the tuple, which can be understood as the invalid transaction number of the tuple.

[0085] ctid: current position of the data tuple, storing page information and page tuple pointer subscript information. If the tuple is updated, the ctid will be updated to the position of the new version.

[0086] The data index adopts a tree structure, and each leaf node stores an index value, that is, a key value (index key value shown in the figure) of a specific field data in the data table. The ctid in the index stores the storage position of the data tuple pointed to by the ctid, storing page information and page tuple pointer subscript information.

[0087] As shown in Figure 1 , when the data tuple is updated, an exclusive lock of the tuple to be updated needs to be acquired first, and then it is judged whether the tuple is updated by other transactions. If yes, the exclusive lock is released, and a new version of exclusive lock is acquired repeatedly until the latest tuple that is not updated by other transactions. Then, the xmax of the last old tuple is updated to the current transaction number, and a new tuple is inserted. The ctid of the old tuple is updated to the position of the new version. In this way, the data pointing method is: from the old version to the latest version.

[0088] For the index, there are two processing methods when the data tuple is updated:

[0089] HOT (Heap Only Tuple) scenario: no new index tuple is inserted, if the index value does not change before and after the update, and the new and old tuples before and after the update are in the same page.

[0090] Non-HOT scenario: a new index tuple is inserted, if the index value changes before and after the update, or the index value does not change before and after the update, but the new and old tuples are not in the same page.

[0091] The hot record in the database refers to a record that is frequently operated in a unit of time, such as update, deletion, etc. In daily business processing, it is often necessary to perform data operation behaviors such as updating, querying, deleting, etc. on the hot data in the database.

[0092] The hotness of the hot record can represent the possibility of being locked by the update operation in a unit of time. The calculation method of the lock waiting time caused by updating the hot record can refer to the following calculation formula: hot record update lock waiting time = (hot record update hotness / (100%-hot record update hotness)) * lock holding time after obtaining the hot record update lock.

[0093] Figure 2 A trend chart showing the change of the lock waiting time with the hot record hotness in the process of the data operation of the hot record is schematically shown.

[0094] For example: the average lock holding time of a certain transaction after obtaining the update lock of a certain hot record is 1 millisecond, the hot record update hotness is x, and the hot record update lock waiting time is y. The relationship between the lock waiting time and the hotness is as shown in Figure 2 According to Figure 2 It can be seen that when the hotness is relatively low, the lock waiting time is not obvious, and when the hotness exceeds 80%, the lock waiting time will sharply rise.

[0095] In the process of implementing the present disclosure, based on the data storage structure in the related art and the above theoretical basis, the inventors found that at least the following problems exist in the related art:

[0096] For example, the existing database storage structure and data operation method causes some problems, for example: because the data pointing mode is from the old version to the latest version, with the increase of the number of updates of the same record, the number of data access jumps will also increase, the query will slow down, and the update of the hot record will increase with the concurrency, causing the same record to traverse multiple versions to perform multiple locking and releasing processes, which enlarges the original hot spot problem of one point to multiple points, causing the overall performance to decline. There is still no solution to the problem of poor query and processing efficiency of the update hot record of the PostgreSQL and other databases.

[0097] In addition, for the discovery of hot record, currently, it depends on the production transaction performance monitoring. There are two monitoring methods, but there are certain shortcomings.

[0098] One of the methods is that the transaction response time becomes slow as the hot spot identification standard. However, according to the hot record update lock time calculation formula, according to Figure 2 It can be known that, before the hotness is 80%, the transaction lock time of updating the hot record has limited influence on the response time, and is easy to be submerged in a large number of ordinary transactions; after exceeding 80%, the lock time will increase rapidly, and the SQL response time may increase rapidly from 2-3 milliseconds to seconds or even tens of seconds, which will cause the transaction response time of updating the hot record to become long. Once the hot record to be processed accumulates, a large amount of precious resources such as database connection number and memory will be consumed, and finally the database connection number and memory may be exhausted, which cannot accept external services. This production transaction performance monitoring belongs to post-monitoring, and cannot play an active role.

[0099] The other is to monitor the change details and other running information, and to take the records existing at two time points close to each other as the hot spot identification standard. However, this scheme depends on the registration of change details for each operation on the record, and the application transformation cost is large. Usually, only table-level hot spot discovery can be achieved; and it is difficult to accurately distinguish whether the two time points close to each other are caused by normal serial batch processing or the real hot spot scenario caused by parallel processing, so as to be not conducive to accurately locating the hot data.

[0100] In summary, compared with the databases (such as Oracle and Mysql) realized by the separate storage mode, for the multi-version concurrency control of the PostgreSQL database, in addition to the problem that the lock time increases with the increase of the update hotness, the query time also increases with the increase of the update old version, the traversal multi-version lock increases, and the hot record lock processing flow becomes long, so that the transaction response time of updating the record hot spot becomes slower. In addition, there is no method for early discovery and solution of the hot spot problem. Generally, the problem can be discovered only after the production problem occurs and causes business impact. The problem cannot be discovered in advance when the hotness of the updated record is low. For the case that the update hotness of individual records in the table is high, the problem may be covered due to a large number of normal serial batch processing records.

[0101] Therefore, an embodiment of the present disclosure provides a data processing method to at least partially solve the above technical problems. The method comprises the following steps.

[0102] read, based on the target query value, a target index tuple from at least one index tuple in the index table, wherein the index tuple comprises a first index field and a second index field, at least one first index field value of the at least one index tuple is different from each other, such that different index tuples are distinguished, and the second index field is used to locate a chain head tuple from the chain head table;

[0103] read, based on the second index field value of the target index tuple, the chain head tuple from the chain head table, wherein the chain head tuple comprises a chain head field, and a value of the chain head field is a storage location of a latest version of data tuples in the plurality of versions of data tuples in the data table in the data table;

[0104] read, based on the chain head field, the first target data tuple from the data table, wherein the first target data tuple is the currently latest version of data tuple.

[0105] Figure 3 An application scenario diagram of the data processing method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0106] As shown in Figure 3 application scenario 300 according to the embodiments can comprise a terminal device 301, a server 302 and a database 303. The terminal device 301, the server 302 and the database 303 are connected through a network for communication, and the network can comprise various connection types, such as wired, wireless communication link or optical cable, etc.

[0107] The terminal device 301 can be various electronic devices with display screen and supporting web browsing, including but not limited to smart phone, tablet computer, laptop computer and desktop computer, etc.

[0108] The server 302 can be a server providing various services, such as a background management server supporting the website browsed by the user using the terminal device 301 (only as an example). The background management server can analyze and process the received user request and other data, and feed back the processing result (such as the webpage, information or data generated or obtained according to the user request) to the terminal device.

[0109] The database 303 can be any type of database, including but not limited to various relational databases and non-relational databases, etc. The database 303 can store various types of business data according to a preset data structure, such as various transaction data, consumption data, product sales data, customer data, etc.

[0110] The user can use the terminal device 301 to interact with the server 302 through a network to receive or send messages, etc. Various communication client applications can be installed on the terminal device 301, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0111] In the execution of the service process of the server 302, the related data operations such as query, update, deletion, etc. can be performed by accessing the database 303. For example, the data query operation can be performed according to the method described in the embodiments of the present disclosure: first, based on the query keyword, the target index tuple is read from at least one index tuple in the index table, then based on the index field of the target index tuple, the chain head tuple is read from the chain head table, and finally based on the chain head tuple, the current latest version of the data tuple is read from the data table.

[0112] It should be noted that the data processing method provided by the embodiments of the present disclosure can generally be executed by the server 302. Correspondingly, the data processing apparatus provided by the embodiments of the present disclosure can generally be arranged in the server 302. The data processing method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 302 and capable of communicating with the terminal device 301 and / or the server 302. Correspondingly, the data processing apparatus provided by the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 302 and capable of communicating with the terminal device 301 and / or the server 302.

[0113] It should be understood that, Figure 1 The number of terminal devices, servers, and databases in the above-mentioned system is only illustrative. According to the needs of implementation, there can be any number of terminal devices, servers, and databases.

[0114] It should be noted that the data processing method and apparatus of the present disclosure can be applied to the field of big data technology, and can also be applied to the field of financial technology, and can also be applied to any field other than the field of big data technology and the field of financial technology. The embodiments of the present disclosure do not limit the application field of the above-mentioned data processing method and apparatus.

[0115] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with relevant laws and regulations, necessary security measures are taken, and do not violate public order and good customs.

[0116] In the technical solution of the present disclosure, the authorization or consent of the user is obtained before the user's personal information is acquired or collected.

[0117] The following will be based on Figure 3 the scenario described above, through Figures 4-10The data processing method of the disclosed embodiment is described in detail.

[0118] Figure 4 A flowchart of a data processing method according to an embodiment of the present disclosure is schematically shown. Figure 5 A database storage structure diagram according to an embodiment of the present disclosure is schematically shown. The following describes the method of the present embodiment in combination with the database storage structure diagram. Figure 4 、 Figure 5 The method of the present embodiment is described.

[0119] As shown in Figure 4 , the data processing method of the present embodiment includes operation S401 to operation S403.

[0120] In operation S401, a target index tuple is read from at least one index tuple in an index table based on a target query value, where the index tuple includes a first index field and a second index field, at least one first index field value of the at least one index tuple is different from each other, so as to identify and distinguish different index tuples, and the second index field is used to locate a chain head tuple from a chain head table.

[0121] In operation S402, the chain head tuple is read from the chain head table based on the second index field value of the target index tuple, where the chain head tuple includes a chain head field, and the value of the chain head field is the storage location of the latest version of the data tuple in the data table in the plurality of versions of the data tuple in the data table.

[0122] In operation S403, the first target data tuple is read from the data table based on the chain head field, where the first target data tuple is the currently latest version of the data tuple.

[0123] According to the embodiments of the present disclosure, the above-mentioned data processing method can be applied to a database using a centralized storage mode, for example, a PostgreSQL database.

[0124] For the centralized storage mode, the present embodiment optimizes and improves the database storage structure in the related art, and the storage structure used is as shown in Figure 5 Compared with the data storage structure in the related art (for example, Figure 1 ), the data storage structure of the present embodiment adds a data chain page storage space, including an index page, a chain page, and a data page, and also adjusts the storage structure of the index page and the data page.

[0125] Specifically, information pointing to the position of the latest version of the data tuple in the same record new and old version chain structure is extracted separately and stored in an independent page space for management. The space is the chain header page space, and the tuple in the chain header page space is called the chain header tuple. The chain header tuple only stores information pointing to the latest version of the data tuple, and such page information can be placed in the data cache area to reduce data access overhead and will not cause excessive memory space occupation pressure.

[0126] In the chain header tuple, the value of the chain header field (ctid) is the storage position of the latest version of the data tuple in the data table, the page information of the latest data tuple, and the tuple pointer subscript information in the page. As shown in the example, Figure 5 The chain header field (ctid) of the chain header tuple in the example is (2, 1), which represents that the storage position of the current latest version of the data tuple in the data table is the second group of the second page.

[0127] The data page is used to store a plurality of different versions of data tuples, and the data tuples are mainly used to store different versions of entity data and version information. The data tuple structure for storing a row of data can include one or more fields, for example, as shown in the example, Figure 5 The example includes four fields, which are a first data field (data), a second data field (ctid), a third data field (delFlag), and a fourth data field (xmin). The second data field, the third data field, and the fourth data field are tuple headers, and the first data field is a tuple content part. The value (value) of the first data field (data) is the main data of each version of the data tuple. As shown in the example, Figure 5 The data value value1 of the first version of the data tuple (insertion transaction number 10) can be: account-xxx1, name-xx, and amount-100 yuan. The value of the second data field (ctid) is the storage position (storage page information and tuple pointer subscript information in the page) of the previous version of the data tuple of each version in the data table. The value of the third data field (delFlag) is a mark value, which is used to represent whether each version of the data tuple is deleted. The value of the fourth data field (xmin) represents the operation transaction number of each version of the data tuple.

[0128] According to an embodiment of the present disclosure, the data index of the database adopts a tree structure, and each leaf node stores an index tuple. The index tuple includes a first index field (key) and a second index field (ctid). The values of the first index fields of a plurality of index tuples are different, so as to identify and distinguish different index tuples, as shown in the example, Figure 5In the shown example, the first index fields of the two indexes are different, being key=v1 and key=v2 respectively. The value of the second index field is the storage location of the head tuple in the head table, which is used to locate the head tuple from the head table. The ctid in the index stores the page information of the head tuple, and the tuple pointer index information in the page. For example, the second index field (ctid) is (0, 1), which represents that the storage location of the head tuple in the head table is the first group of the 0th page.

[0129] According to an embodiment of the present disclosure, based on the above data storage structure of the embodiment of the present disclosure, by executing the above data processing method of the embodiment of the present disclosure, the latest version of the data tuple can be quickly queried.

[0130] Specifically, first, based on the target query value, the target index tuple corresponding to the target query value (the first index field value of the target index tuple is the same as the target query value) is read from the index tuple in the index table by traversing the first index field of the index tuple through the above operation S401. Since the second index field (ctid) in the index tuple can be used to locate the head tuple from the head table, further, the head tuple can be located based on the second index field (ctid) through operation S402. Again, since the value of the head field is the storage location of the latest version of the data tuple in the data table, the current latest version of the data tuple can be located from the data table based on the head field through operation S403.

[0131] For example, a user wants to query the latest version of the data of an account xxx1, and can match the target index tuple corresponding to the account xxx1 (the first index field value v1 of the target index tuple is xxx1) from the multiple index tuples. Then, based on the second index field ctid=(0, 1), the head tuple is located (the storage location of the head tuple in the head table is the first group of the 0th page). Finally, according to the head field ctid=(2, 2), the current latest version of the data tuple is located from the data table (its storage location in the data table is the second group of the 2nd page).

[0132] According to an embodiment of the present disclosure, compared with the related art, the database storage structure of the embodiment of the present disclosure is optimized and improved. Since the second index field in the index tuple can be used to locate the head tuple from the head table, and the value of the head field is the storage location of the latest version of the data tuple in the data table. Therefore, based on the above data storage structure of the embodiment of the present disclosure, by executing the above data processing method of the embodiment of the present disclosure, the latest version of the data tuple can be quickly queried.

[0133] Specifically, in the related art, because the index points to an older data version, when performing a data query, multiple versions of data need to be traversed in sequence to locate the latest data. With an increase in the number of updates to the same record, the number of data access jumps also increases, and the query becomes slower. In comparison, the above data query method of the embodiments of the present disclosure can locate the latest data through only one index plus one data access, regardless of the number of data versions, and greatly improves the processing efficiency of hot data query.

[0134] The above method of the embodiments of the present disclosure, on the basis of retaining the original database to store new and old versions in a centralized manner, avoids the problem of slower query with an increase in the number of versions by adjusting the arrangement of new and old version data and the storage mode of the index, solves the problem of slower query with an increase in the number of updated versions under the database multi-version concurrency control mechanism, and greatly improves the processing efficiency of hot data query. At the same time, compared with a database (such as Oracle or Mysql) implemented in a separate storage mode, the advantages of less IO overhead for update operations, fast rollback, no problem of old UNDO snapshot, and the like are still retained. In addition, the head-of-chain tuple is an independent page space, which facilitates placing such page information in a data cache area to reduce data access overhead and does not cause excessive memory space occupation pressure.

[0135] According to an embodiment of the present disclosure, in the index tuple, the first index field values of the plurality of index tuples are different, and are used as a query index. Based on a target query value, reading a target index tuple from at least one index tuple in the index table can specifically include: traversing the index table, reading an index tuple with the same first index field value as the target query value from the at least one index tuple, as the target index tuple. For example, a user wants to query the latest version of data of an account xxx1, and can match the index tuple with the same first index field value v1 as the target query value xxx1 to the target index tuple by using the account to be queried (i.e., using the target query value) xxx1.

[0136] According to an embodiment of the present disclosure, in the index tuple, the second index field values of the plurality of index tuples are the same, and are the storage positions of the head-of-chain tuples in the head-of-chain table, so that the head-of-chain tuple can be located through the second index field of any index tuple.

[0137] According to an embodiment of the present disclosure, by pointing the plurality of indexes to the head-of-chain tuple, in the process of performing data operation, the head-of-chain tuple can be located through any index, and the data can be further located based on the head-of-chain tuple, further improving the efficiency of data processing.

[0138] According to an embodiment of the present disclosure, among the plurality of fields included in the data tuple structure of a row of data, the value of the first data field is the main data of each version data tuple. For example, Figure 5In the shown example, the values of the first data field (data) of the same record of multiple versions are respectively the subject data of the data tuples of the respective versions, and the subject data of the multiple versions are different. For example, user Wang XX opens an account in a certain system and deposits 100 yuan, the value of the data field (data) of the first version of the data tuple (insert transaction number 10) value1 can be a set of data associated with account xxx1, which can be: account-xxx1, name-Wang XX, amount-100 yuan; the value of the data field (data) of the second version of the data tuple (insert transaction number 12) value2 can be data updated from the first version, which is another set of data associated with account xxx1, which can be: after Wang XX transfers 80 yuan, the data is updated to account-xxx1, name-Wang XX, amount-20 yuan; the value of the data field (data) of the fourth version of the data tuple (insert transaction number 15) value4 can be data updated from the third version, for example, the associated data after the original account xxx1 is updated to account xxx2 due to account change, which can be: account-xxx2, name-Wang XX, amount-120 yuan.

[0139] According to an embodiment of the present disclosure, the subject data of the multiple versions are different, respectively value1, value2, value3, …, but can contain the same data primary key, so as to establish an index according to the primary key. For example, Figure 5 In the shown example, the data tuples of the multiple versions all contain the same data primary key: key (for example, the key is the account number). However, the primary key values can be the same or different, for example Figure 5 In the shown example, the primary key values of the data tuples of the first version, the second version, and the third version are the same, all key=v1 (for example, account number=xxx1); the primary key value of the data tuple of the fourth version is key=v2 (for example, account number=xxx2).

[0140] According to an embodiment of the present disclosure, since the data index is established according to the subject data of the data tuple, which is used for indexing and positioning data, the first index field of the index tuple matches the subject data (the first data field) of the data tuple. In the index page of the database associated with the same set of multiple versions of data, one or more index tuples can be stored, such as Figure 5 In the shown example, in the index page of the database associated with the same set of multiple versions of data, two index tuples are stored, the values of the two first index fields of the index tuples (key=v1, key=v2) match the multiple data key values (key=v1, key=v2) of the first data field (data) of the data tuple. And after updating the data, if the data key value key changes, an index tuple corresponding thereto needs to be added.

[0141] According to an embodiment of the present disclosure, among the multiple fields included in the data tuple structure, the value of the second data field (ctid) is the storage location (storage page information and tuple pointer subscript information within the page) of the data tuple of the previous version in each version in the data table, so that the data tuple of the previous version can be linked to respectively through the value of each second data field. For example, as shown in the example of Figure 5 , the ctid of the data tuple of the first version is empty, indicating that it is the oldest version tuple of the data. The ctid of the data tuple of the second version is (0, 1), indicating that the storage location of the data tuple of the first version in the data table is the first group of the 0th page. The ctid of the data tuple of the fourth version is (1, 1), indicating that the storage location of the data tuple of the third version in the data table is the first group of the 1st page. In this way, when performing data query, any version of data can be located according to the data linking relationship between the new and old versions of data.

[0142] By setting the value of the second data field as the storage location of the data tuple of the previous version in each version in the data table, the pointing manner of the data can be changed from the old version pointing to the new version in the related art (see Figure 1 ) to the new version pointing to the old version. In this way, when performing data query, the data version that is queried first is the newer data version, and if the older version of data needs to be queried, the new version can be used to query the older version one by one until the data version that needs to be queried is found. Since when performing data query, the newer data version is more often queried, by changing the storage manner as described above, the newer data version can be quickly located, which is suitable for most data query scenarios and can speed up the data query.

[0143] According to an embodiment of the present disclosure, among the multiple fields included in the data tuple structure, the value of the third data field (delFlag) is a mark value, which is respectively used to represent whether the data tuple of each version is deleted, for example, delFlag is 0, indicating that the data of this version is not deleted, and delFlag is 1, indicating that the data of this version is deleted.

[0144] According to an embodiment of the present disclosure, among the multiple fields included in the data tuple structure, the value of the fourth data field (xmin) represents the operation transaction number of the data tuple of each version. The xmin can be the transaction number of inserting the tuple, which can be understood as the effective transaction number of the tuple. The operation transaction numbers of the data tuples of multiple versions can be different or partially the same. For example, as shown in the example of Figure 5 , the operation transaction numbers xmin of the data tuples of the third version and the fourth version are the same, indicating that the same transaction has performed two update operations on the second version of data successively.

[0145] According to embodiments of this disclosure, the index tuple may further include a third index field (xmin) and a fourth index field (xmax). The index tuple introduces xmin and xmax to record the insertion and deletion version numbers of the index, respectively, to improve the efficiency of index scanning and access.

[0146] Figure 6 A flowchart illustrating a data processing method according to another embodiment of the present disclosure is shown schematically; Figure 7 A flowchart illustrating a data processing method according to another embodiment of the present disclosure is shown schematically; Figure 8 A schematic diagram illustrating the principle of performing data operations according to an embodiment of the present disclosure is provided. Hereinafter, in conjunction with... Figure 6 , Figure 7 , Figure 8 Various data operations, such as insertion, update, deletion, and query, are performed on the data processing method based on the embodiments of this disclosure.

[0147] The following combination Figure 8 The present disclosure describes the method for performing data insertion in the data processing method of the present invention.

[0148] like Figure 8 As shown, a new data record with a data value of value1 is inserted through transaction operation with transaction number 10. This is the first version of the data. The current transaction number of the first version of the data tuple, i.e., the fourth data field xmin, is 10. The first data field data has a value of value1 (where the primary key key = v1). The second data field ctid is empty, indicating that it is the oldest version of the data tuple. The third data field delFlag has a value of 0, indicating that this version of the data has not been deleted.

[0149] At the same time, insert a new chain head tuple. The value of the chain head field (ctid) is the storage location of the first version data tuple in the data table. ctid = (0, 1), which means that the current latest version data tuple is stored in the first group on page 0 of the data table.

[0150] Simultaneously, a new index tuple is inserted. The first index field (key = v1) of the index tuple is the same as the primary key of the current data tuple. The second index field, ctid, points to the position of the first tuple in the chain (ctid), where ctid = (0, 1) indicates that the first tuple is stored in the first group of page 0 of the first table. The third index field, xmin, is the transaction number 10 for the current index insertion, and the fourth index field, xmax, is 0, indicating that there is currently no transaction deleting the index.

[0151] The following combination Figure 6 , Figure 8The method for performing data updating in the data processing method of the embodiments of the present disclosure is introduced.

[0152] According to the embodiments of the present disclosure, as shown in Figure 6 The method for performing data updating includes operations S601-S603.

[0153] In operation S601, if the first target data tuple needs to be updated, an exclusive lock of the head tuple is acquired.

[0154] In operation S602, a second target data tuple is added in the data table, where the second target data tuple is the data tuple after updating the first target data tuple.

[0155] In operation S603, the value of the head field is updated to the storage location of the second target data tuple in the data table.

[0156] The specific execution method of the above method can refer to the example shown in Figure 8 As shown in Figure 8 In the case of updating the data for the first time, the exclusive lock of the head tuple needs to be acquired first. In the case of multiple transactions operating the data at the same time, there is a lock waiting queue.

[0157] After acquiring the exclusive lock of the head tuple, the second version of the data tuple, i.e., the updated data tuple, is inserted. Specifically, a new data with a data value of value2 is inserted by the transaction with a transaction number of 12, i.e., the second version of the data. The current transaction number of the second version of the data tuple, i.e., the fourth data field xmin, is 12; the data value of the first data field data is value2 (where the data primary key key=v1); the second data field ctid=(0, 1) represents that the storage location of the previous version of the data tuple in the data table is the 0th page and the 1st group; and the value of the third data field delFlag is 0, indicating that the version of the data is not deleted.

[0158] As shown in Figure 8As shown, in the case of updating the data for the second time, the head tuple exclusive lock also needs to be acquired first, and the new version data is inserted after the head tuple exclusive lock is acquired. The current update transaction is used to continuously perform two updates on the previous version of the data, and therefore, two new data, the third version of the data tuple and the fourth version of the data tuple, are continuously inserted by the same transaction. The current transaction numbers of the third and fourth versions of the data tuple are the same, that is, the fourth data field xmin is 15; the values of the first data field data of the third and fourth versions of the data tuple are value3 and value4, respectively. In the fourth version of the data tuple, the key value of the data primary key is changed from key=v1 to key=v2. The second data field ctid of the third and fourth versions of the data tuple respectively points to the storage location of the previous version of the data tuple. The value of the third data field delFlag of the third and fourth versions of the data tuple is 0, indicating that the version data is not deleted.

[0159] According to an embodiment of the present disclosure, based on the data storage structure of the present disclosure, the latest version of the data tuple is pointed to by the head tuple, and therefore, when a data operation such as data update is performed, the exclusive lock of the head tuple is only needed to be acquired to perform the data operation. Since the data update is usually performed on the latest version of the data, by using the above data update method, the latest data can be located to perform the data update by only one index, one locking, and one data access in most cases. Compared with the related art in which the latest data is located by traversing multiple versions to lock, release the lock, and traverse and query the data of multiple versions, the above method of the present disclosure solves the problem that the update hotspot record processing is further slowed down due to the traversal of multiple versions to lock in the database update hotspot record, and greatly improves the processing efficiency of the hotspot data update.

[0160] According to an embodiment of the present disclosure, since the data index is established according to the primary data of the data tuple, the key value of the data primary key in the first data field data of the index tuple needs to match the key value of the data primary key of the data tuple, and therefore, if the key value of the data primary key is updated after the data version is updated, the index tuple corresponding to the key value needs to be newly added.

[0161] Specifically, in the process of performing the data update, the method of changing the index is as follows:

[0162] In the case where the key value of the data primary key of the second target data tuple (the data after the update) is the same as the key value of the data primary key of the first target data tuple (the data before the update), the index tuple does not need to be updated;

[0163] If the primary key value of the second target data tuple (updated data) is different from the primary key value of the first target data tuple (previous data), a new index tuple is added to the index table. The value of the first index field of the new index tuple is the primary key value of the second target data tuple.

[0164] like Figure 8 As shown, after the second data update, the primary key value in the fourth version of the data tuple has changed from key = v1 to key = v2. Therefore, a new index needs to be added. In the new index, the value of the first index field is the primary key value of the updated data tuple, i.e., the first index field key = v2, which is the same as the primary key value v2 of the fourth version of the data tuple. Simultaneously, the old index tuple xmax is updated to the current transaction number 15. In the newly inserted index tuple, xmin is the current transaction number 15, xmax is 0, and the second index field ctid = (0, 1) points to the position of the first tuple in the chain.

[0165] In the above-described index update method and related technologies of this disclosure (see...), the following are examples of the index update method and related technologies (see...). Figure 1 The difference is:

[0166] In related technologies, there are two situations that trigger data index updates: 1. The primary key values ​​of the old and new versions of the data tuple are different; 2. The primary key values ​​of the old and new versions of the data tuple are the same, but the old and new versions of the data tuple are not on the same page. Both of these situations trigger the insertion of a new index tuple, and the newly added index ctid points to the newly inserted data tuple. Based on this, when there are many data versions and frequent changes in the number of data pages, the number of indexes will increase significantly, and different indexes will point to different versions of data. When performing data operations, since it is impossible to know which index points to the latest version of the data based solely on the index, multiple indexes must be traversed sequentially for data operations, resulting in low data processing efficiency.

[0167] The index update method described in this embodiment only triggers index update when the key values ​​of the primary keys of the data tuples in the new and old versions are different. However, index update will not be triggered if the data tuples in the new and old versions are not on the same page. In addition, regardless of whether it is a new version index or an old version index, its second index field ctid points to the position of the first tuple in the chain.

[0168] Therefore, compared with the case that the number of indexes is large and different indexes point to different versions of data in the related art, the number of indexes can be greatly reduced, the number of data indexing can be greatly reduced, and the efficiency of data indexing can be improved by using the index updating method provided in the embodiments of the present disclosure. When performing data operation, the latest data can be located by using the chain head tuple regardless of which index is used, so that the latest data can be quickly located to perform data operation, and the efficiency of hot data processing can be further improved.

[0169] According to the embodiments of the present disclosure, the value of the third data field (delFlag) in the plurality of fields included in the data tuple structure is used to represent whether the data tuple of each version is deleted.

[0170] The following describes the data processing method of the embodiments of the present disclosure in combination with Figure 7 , Figure 8 the above data structure.

[0171] As shown in Figure 7 , the method for performing data deletion includes operation S701 to operation S703.

[0172] In operation S701, if the value of the third data field of the second target data tuple is a first value and the second target data tuple needs to be deleted, an exclusive lock of the chain head tuple is acquired, wherein the first value is used to represent that the second target data tuple is not deleted before the deletion operation is performed.

[0173] In operation S702, a third target data tuple is added in the data table, wherein the value of the third data field of the third target data tuple is a second value, and the value of the first data field of the third target data tuple is empty, and the second value is used to represent that the second target data tuple is deleted after the deletion operation is performed.

[0174] In operation S703, the value of the chain head field is updated to the storage location of the third target data tuple in the data table.

[0175] The specific execution method of the above method can refer to the example shown in Figure 8 , as shown in Figure 8 , in the case of performing data deletion on the latest fourth version of data (transaction number 15), the chain head tuple exclusive lock needs to be acquired first, and in the case of multiple transactions operating the data at the same time, there is a lock waiting queue.

[0176] After obtaining the exclusive lock of the chain head tuple, a set of empty data is added in the data table, and the operation transaction number of the data tuple is the current transaction number 18, that is, the fourth data field xmin is 18; the first data field data is empty, representing that the current version data is empty data; the second data field ctid=(2, 1) represents that the storage location of the previous version data tuple in the data table is the first set of the second page; the value of the third data field delFlag is 1, indicating that the version data has been deleted. Finally, the value of the chain head field is updated to the storage location of the current data tuple in the data table, ctid=(2, 2), and the xmax of the index tuple is updated to the current transaction number 18.

[0177] Based on the data storage structure of the embodiments of the present disclosure, the following introduces the method for executing data query in the data processing method of the embodiments of the present disclosure. Figure 8 The method for executing data query in the data processing method of the embodiments of the present disclosure is introduced.

[0178] As shown in the figure, when executing data query, first, based on the target query value, the target index tuple corresponding to the target query value is read from the index tuple in the index table by traversing the first index field key of the index tuple (the value of the first index field of the target index tuple is the same as the target query value). Further, based on the second index field ctid=(0, 1), the chain head tuple is located. Then, based on the chain head field ctid=(2, 2), the current latest version data tuple is located from the data table, and its storage location in the data table is the second set of the second page. Figure 8 The latest version data tuple (transaction number 18) is read, and it is judged that the delFlag is 1, representing that the current record has been deleted, and the record is invisible, and the query of this record is ended.

[0179] If the current latest version data is not queried, the data link relationship between the new and old versions (pointed by the new version to the old version) can be used to query from the new version to the old version in sequence until the data version to be queried is found.

[0180] According to the embodiments of the present disclosure, in order to solve the technical problem that the related art center cannot accurately locate the hot data in the aspect of hot data discovery, the embodiments of the present disclosure determine the hot record by the following method.

[0181] The method includes: obtaining the length of the lock waiting queue of the exclusive lock of the chain head tuple; in the case that the length of the lock waiting queue is greater than a preset threshold, determining the first target data tuple as the hot data.

[0182]

[0183] ​For example, the preset threshold is set to 5, at a certain time, 10 transactions concurrently acquire the lock of the chain head tuple, the length of the lock waiting queue is 10, which is greater than the preset threshold, and the group of data to which the chain head tuple is located is considered as hot data. The chain head tuple ctid, specific data, waiting queue depth greater than the threshold time point, total queue length, maximum depth, initial / maximum / final / average lock time, and the like can be registered in the "update hot record table", and the update hot record can be accurately and effectively found through the table, and specific hot execution information can be obtained.

[0184] According to an embodiment of the present disclosure, under the data operation mechanism in the related art (refer to Figure 1 ), since the number of indexes is large, and different indexes point to different versions of data, when performing data operations, multiple different data need to be locked in a scattered manner. For the case where the update frequency of individual records in the table is high, it can be masked by a large number of normal serial batch records, and the hot data cannot be accurately tracked.

[0185] Compared with the related art, the data storage structure based on the embodiment of the present disclosure points to the data tuple through the chain head tuple, so that when performing data operations such as data update and deletion, only the exclusive lock of the chain head tuple needs to be acquired, and the exclusive lock of the data tuple does not need to be acquired. Compared with the related art, the data processing method of the embodiment of the present disclosure is modified from the original need to traverse multiple versions of locks / locking to unified locking of the chain head tuple. Since the chain head tuple is directly associated with the data, the heat of the chain head tuple can directly reflect the heat of the data itself. At this time, the depth of the lock waiting queue is a direct manifestation of the update frequency of the data. The optimized database update locking method can be used to realize the discovery of update hot records by monitoring the depth of the lock waiting queue, accurately and effectively find the update hot records, and obtain specific hot execution information. Compared with other application layer update hot record discovery schemes, it has the advantages of accurate identification, no need for user modification, wide applicability, and strong universality. It can effectively avoid the occurrence of production problems and improve the reliability of business processing.

[0186] Based on the above data processing method, the present disclosure further provides a data processing apparatus. The apparatus will be described in detail below. Figure 9 .

[0187] Figure 9 The structure block diagram of the data processing apparatus according to the embodiment of the present disclosure is schematically shown. As shown in Figure 9 , the data processing apparatus 900 of the embodiment includes a first reading module 901, a second reading module 902, and a third reading module 903.

[0188] The first reading module 901 is configured to read a target index tuple from at least one index tuple in the index table based on a target query value, wherein the index tuple comprises a first index field and a second index field, and at least one first index field value of the at least one index tuple is different from each other, so as to identify and distinguish different index tuples, and the second index field is used to locate a chain head tuple from a chain head table.

[0189] The second reading module 902 is configured to read the chain head tuple from the chain head table based on the second index field value of the target index tuple, wherein the chain head tuple comprises a chain head field, and the value of the chain head field is the storage location of the latest version of the data tuple in the data table in the plurality of versions of the data tuple in the data table.

[0190] The third reading module 903 is configured to read the first target data tuple from the data table based on the chain head field, wherein the first target data tuple is the currently latest version of the data tuple.

[0191] According to the embodiments of the present disclosure, the database storage structure of the embodiments of the present disclosure is optimized and improved, because the second index field in the index tuple can be used to locate the chain head tuple from the chain head table, and because the value of the chain head field is the storage location of the latest version of the data tuple in the data table. Therefore, based on the above data storage structure of the embodiments of the present disclosure, the first reading module 901 of the embodiments of the present disclosure can quickly query the latest version of the data tuple by reading the target index tuple from at least one index tuple in the index table, reading the chain head tuple from the chain head table by the second reading module 902, and reading the first target data tuple from the data table by the third reading module 903. The above data processing device 900 of the embodiments of the present disclosure, on the basis of retaining the original database to store new and old versions, by adjusting the arrangement mode of new and old versions of data and the index storage mode, avoids the problem that the query becomes slower with the increase of versions, solves the problem that the query becomes slower with the increase of updated versions under the database multi-version concurrency control mechanism, and greatly improves the processing efficiency of hot data query.

[0192] According to the embodiments of the present disclosure, the first reading module 901 comprises a reading unit configured to read an index tuple with the same first index field value as the target query value from the at least one index tuple as the target index tuple.

[0193] According to the embodiments of the present disclosure, at least one second index field value of the at least one index tuple is the storage location of the chain head tuple in the chain head table, so that the chain head tuple can be located by the second index field of any index tuple.

[0194] According to the embodiments of the present disclosure, the above device further comprises a first obtaining module, a first adding module and a first updating module.

[0195] The first obtaining module is configured to obtain an exclusive lock of the chain head tuple when the first target data tuple needs to be updated.

[0196] According to an embodiment of the present disclosure, the apparatus further includes a second obtaining module and a determining module.

[0197] The second obtaining module is configured to obtain a length of a lock waiting queue of the exclusive lock of the chain head tuple.

[0198] According to an embodiment of the present disclosure, the data tuple includes a first data field, a plurality of first data field values of the plurality of versions of the data tuple, and a plurality of subject data of the plurality of versions of the data tuple, respectively.

[0199] According to an embodiment of the present disclosure, the data main key has a same or different data key value in the plurality of versions.

[0200] According to an embodiment of the present disclosure, the apparatus further includes a second adding module configured to add an index tuple in the index table when the data main key of the second target data tuple has a different key value from the data main key of the first target data tuple.

[0201] According to an embodiment of the present disclosure, the data tuple further includes a second data field, and a plurality of second data field values of the plurality of versions of the data tuple are respectively storage locations of previous versions of the data tuple in the data table.

[0202] According to an embodiment of the present disclosure, the data tuple further includes a third data field, and a plurality of third data field values of the plurality of versions of the data tuple are respectively used to represent whether the data tuple of each version is deleted.

[0203] According to an embodiment of the present disclosure, the apparatus further includes a third obtaining module, a third adding module, and a second updating module.

[0204] The third obtaining module is configured to obtain an exclusive lock of the head tuple in a case where the third data field value of the second target data tuple is a first value and the second target data tuple needs to be deleted, the first value being used to represent that the second target data tuple is not deleted before the deletion operation is performed.

[0205] According to an embodiment of the present disclosure, the data tuple further comprises a fourth data field, and the plurality of fourth data field values of the plurality of versions of the data tuple are respectively an operation transaction number of each version of the data tuple.

[0206] According to an embodiment of the present disclosure, any of the first reading module 901, the second reading module 902 and the third reading module 903 can be combined in one module, or any of the modules can be split into multiple modules. Alternatively, at least part of the function of one or more of the modules can be combined with at least part of the function of the other modules, and implemented in one module. According to an embodiment of the present disclosure, at least one of the first reading module 901, the second reading module 902 and the third reading module 903 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or any one of software, hardware and firmware or any appropriate combination of any of them. Alternatively, at least one of the first reading module 901, the second reading module 902 and the third reading module 903 can be at least partially implemented as a computer program module which can perform corresponding functions when the computer program module is run.

[0207] Figure 10 A block diagram of an electronic device suitable for implementing the data processing method according to an embodiment of the present disclosure is schematically shown.

[0208] As Figure 10As shown, the electronic device 1000 according to embodiments of the present disclosure includes a processor 1001 that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1002 or loaded from a storage section 1008 into a random access memory (RAM) 1003. The processor 1001 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chip set, and / or a special purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 1001 can also include an on-board memory for cache use. The processor 1001 can include a single processing unit or multiple processing units to perform the various actions of the method processes according to embodiments of the present disclosure.

[0209] In the RAM 1003, various programs and data required for the operation of the electronic device 1000 are stored. The processor 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. The processor 1001 performs various operations of the method processes according to embodiments of the present disclosure by executing the programs in the ROM 1002 and / or the RAM 1003. Note that the programs can also be stored in one or more memories other than the ROM 1002 and the RAM 1003. The processor 1001 can also perform various operations of the method processes according to embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0210] According to embodiments of the present disclosure, the electronic device 1000 can also include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 can also include one or more of the following components connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as necessary. A removable medium 1011 such as a magnetic disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1010 as necessary, so that a computer program read out therefrom is installed into the storage section 1008 as necessary.

[0211] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present disclosure.

[0212] According to the embodiments of the present disclosure, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to the embodiments of the present disclosure, the computer readable storage medium can include one or more memories of the ROM 1002 and / or the RAM 1003 described above and / or one or more memories other than the ROM 1002 and the RAM 1003.

[0213] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the data processing method provided by the embodiments of the present disclosure.

[0214] The above functions defined in the system / apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1001. According to the embodiments of the present disclosure, the system, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0215] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal via a network medium, and be downloaded and installed through the communication part 1009 and / or installed from the detachable medium 1011. The program codes contained in the computer program can be transmitted via any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0216] In such embodiments, the computer program can be downloaded and installed from the network through the communication section 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-described functions defined in the system of the embodiments of the present disclosure are executed. According to the embodiments of the present disclosure, the system, device, apparatus, module, unit, and the like described above can be implemented by computer program modules.

[0217] According to the embodiments of the present disclosure, the program code for executing the computer program provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language or similar programming language. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected through the Internet by using an Internet service provider).

[0218] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0219] Those skilled in the art can understand that the features described in various embodiments of the present disclosure and / or claims can be combined or / and integrated, even if such combinations or integrations are not explicitly described in the present disclosure. In particular, the features described in various embodiments of the present disclosure and / or claims can be combined and / or integrated in various combinations, without departing from the spirit and teachings of the present disclosure. All these combinations and / or integrations fall within the scope of the present disclosure.

[0220] The above described embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and all such substitutions and modifications shall fall within the scope of the present disclosure.

Claims

1. A data processing method, comprising: reading a target index tuple from at least one index tuple in an index table based on a target query value, wherein the index tuple comprises a first index field and a second index field, at least one first index field value of the at least one index tuple is different from each other, so as to identify different index tuples, and the second index field is used to locate a chain head tuple from a chain head table; reading the chain head tuple from the chain head table based on a second index field value of the target index tuple, wherein the chain head tuple comprises a chain head field, and a value of the chain head field is a storage location of a latest version of data tuples in a plurality of versions of data tuples in a data table in the data table; reading a first target data tuple from the data table based on the chain head field, wherein the first target data tuple is a current latest version of data tuple; the data tuple comprises a second data field, and a plurality of second data field values of a plurality of versions of data tuples are respectively storage locations of a previous version of data tuple of each version in the data table, so that each of the second data field values can be linked to the previous version of data tuple.

2. The method of claim 1, wherein, the reading of the target index tuple from the at least one index tuple in the index table based on the target query value comprises: reading an index tuple with the same first index field value as the target query value from the at least one index tuple as the target index tuple. 3.The method of claim 1, wherein: at least one second index field value of the at least one index tuple is a storage location of the chain head tuple in the chain head table, so that the chain head tuple can be located by the second index field of any of the index tuples. 4.The method of claim 1, further comprising: acquiring an exclusive lock of the chain head tuple in a case of needing to update the first target data tuple; adding a second target data tuple in the data table, wherein the second target data tuple is a data tuple after updating the first target data tuple; updating a value of the chain head field to a storage location of the second target data tuple in the data table. 5.The method of claim 4, further comprising: acquiring a length of a lock waiting queue of the exclusive lock of the chain head tuple; determining the first target data tuple as a hot data in a case that the length of the lock waiting queue is greater than a preset threshold. 6.The method of claim 4, wherein: the data tuple further comprises a first data field, and a plurality of first data field values of a plurality of versions of data tuples are respectively main data of each version of data tuple, and the plurality of versions of the main data are different from each other but contain the same data primary key. 7.The method of claim 6, wherein: the data primary key has the same or different data key values in a plurality of versions; at least one first index field value of the at least one index tuple matches the plurality of data key values. 8.The method of claim 7, further comprising: In a case where a key value of a data primary key of the second target data tuple is different from a key value of a data primary key of the first target data tuple, a new index tuple is added in the index table, wherein a value of a first index field of the new index tuple is the key value of the data primary key of the second target data tuple. 9.The method of claim 4, wherein: the data tuple further comprises a third data field, wherein a plurality of third data field values of the plurality of versions of the data tuple respectively represent whether a corresponding version of the data tuple is deleted. 10.The method of claim 9, further comprising: in a case where the third data field value of the second target data tuple is a first value and the second target data tuple needs to be deleted, acquiring an exclusive lock of the head tuple, wherein the first value represents that the second target data tuple is not deleted before a deletion operation is performed; adding a third target data tuple in the data table, wherein a third data field value of the third target data tuple is a second value and a first data field value of the third target data tuple is empty, and the second value represents that the second target data tuple is deleted after a deletion operation is performed; updating a value of the head field to a storage location of the third target data tuple in the data table. 11.The method of claim 1, wherein: the data tuple further comprises a fourth data field, wherein a plurality of fourth data field values of the plurality of versions of the data tuple respectively represent an operation transaction number of a corresponding version of the data tuple. 12.A data processing apparatus, comprising: a first reading module configured to read a target index tuple from at least one index tuple in an index table based on a target query value, wherein the index tuple comprises a first index field and a second index field, at least one first index field value of the at least one index tuple is different from each other, such that different index tuples are distinguished, and the second index field is used to locate a head tuple from a head table; a second reading module configured to read the head tuple from the head table based on a second index field value of the target index tuple, wherein the head tuple comprises a head field, and a value of the head field is a storage location of a latest version of a data tuple in a plurality of versions of the data tuple in a data table; a third reading module configured to read a first target data tuple from the data table based on the head field, wherein the first target data tuple is the latest version of the data tuple; the data tuple comprises a second data field, and a plurality of second data field values of the plurality of versions of the data tuple respectively represent a storage location of a previous version of a corresponding version of the data tuple in the data table, such that each of the second data field values can be linked to the previous version of the data tuple. 13.An electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to carry out the method according to any one of claims 1-11.

14. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images