Database data reordering method and device, equipment, medium and product

Abstract

The invention provides a data reordering method and device of a database, equipment, a medium and a product. After a source end database obtains a database log and determines to-be-stored first data, when a first cache storage space or a second cache storage space is not enough to store the first data, the first data is reordered; and storing the first data in a first disk storage space corresponding to the first transaction in the disk. And after the source end database subsequently receives a reordering instruction of the first transaction, the data stored in the disk and the cache at the same time are merged and processed, and finally the processed data are sent to the destination end database. Therefore, in the reordering process of the source end database, the storage space of the cache is not occupied greatly, reasonable control over the storage space of the cache is guaranteed, and the influence on execution of other operations on the source end database can be reduced.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a method, device, equipment, medium and product for reordering data in a database. Background Art

[0002] According to business needs, data synchronization operations are sometimes required between databases to ensure the consistency of data stored in the two databases. During data synchronization, the source database can send the log file of the data to be synchronized to the destination database. After receiving the log file, the destination database parses the log file to obtain the statement, and then executes the statement to store the data operated by the statement in the database.

[0003] Since adjacent data in the log file may correspond to different transactions, the source database traverses each data to be synchronized before sending the log file, and stores the data in the cache according to the corresponding transaction. Subsequently, the data corresponding to the transaction in the cache is translated and packaged, and then the processed data is sent to the destination database through the log file for storage. This processing performed by the source database can also be called data reordering.

[0004] However, when the source database stores transaction data through cache, it will occupy a large amount of cache storage space, affecting the source database from executing other operations. Summary of the invention

[0005] The present application provides a method, device, equipment, medium and product for reordering data in a database to solve the technical problem in the prior art that when a source database stores transaction data through a cache, the cache storage space occupies a large amount of space, so as to avoid affecting other operations performed on the source database.

[0006] A first aspect of the present application provides a method for reordering data in a database, which is applied to a source database and is characterized in that it includes: obtaining a database log; determining, based on the database log, first data to be stored and a first transaction corresponding to the first data; when a first cache storage space in a cache for storing the first transaction is insufficient to store the first data, or a second cache storage space in the cache for storing all transactions is insufficient to store the first data, storing the first data in a first disk storage space corresponding to the first transaction in a disk; when a reordering instruction for the first transaction is received, translating and encapsulating the data of the first transaction stored in the first cache storage space and / or the first disk storage space, and sending the processed data of the first transaction to a destination server.

[0007] In an embodiment of the first aspect of the present application, when a first cache storage space in a cache for storing the first transaction is insufficient to store the first data, or a second cache storage space in the cache for storing all transactions is insufficient to store the first data, storing the first data in a first disk storage space corresponding to the first transaction in a disk includes: obtaining a maximum value of a single transaction cache and a maximum value of all transaction caches in the cache; when the sum of the amount of the first data and the amount of data cached in the first cache storage space is greater than the maximum value of the single transaction cache, determining that the first cache storage space is insufficient to store the first data, and storing the first data in a first disk storage space corresponding to the first transaction in a disk; when the sum of the amount of the first data and the amount of data stored in the second cache storage space is greater than the maximum value of all transaction caches, determining that the second cache storage space is insufficient to store the first data, and storing the first data in a first disk storage space corresponding to the first transaction in a disk.

[0008] In an embodiment of the first aspect of the present application, after determining the first data to be stored and the first transaction corresponding to the first data according to the database log, it also includes: when the sum of the data volume of the first data and the data volume cached in the first cache storage space is less than or equal to the maximum value of the single transaction cache, and the sum of the data volume of the first data and the data volume stored in the second cache storage space is less than or equal to the maximum value of all transaction caches, the first data is stored in the first cache storage space.

[0009] In an embodiment of the first aspect of the present application, after determining the first data to be stored and the first transaction corresponding to the first data according to the database log, it also includes: searching for the first object allocator corresponding to the first transaction from a preset object allocator according to the identification information of the first transaction; when the first transaction does not include the corresponding first cache storage space, creating the first object allocator corresponding to the first transaction, and creating the first cache storage space in the cache and the first disk storage space in the disk for the first object allocator, and creating cache data volume information and disk data volume information of the transaction in the first object allocator.

[0010] In an embodiment of the first aspect of the present application, after translating and encapsulating the data of the first transaction stored in the first cache storage space and / or the first disk storage space, the processed data of the first transaction is sent to the destination server, including: when a reordering instruction for the first transaction is received and the disk data volume information in the first object allocator is less than 0, after translating and encapsulating the data of the first transaction stored in the first cache storage space, the processed data of the first transaction is sent to the destination server; when a reordering instruction for the first transaction is received and the disk data volume information in the first object allocator is not less than 0, the data of the first transaction stored in the first disk storage space and the data of the first transaction stored in the first cache storage space are merged, translated and encapsulated, and the merged data of the first transaction is sent to the destination database.

[0011] In an embodiment of the first aspect of the present application, when a reordering instruction of the first transaction is received, the data of the first transaction stored in the first cache storage space and / or the first disk storage space is translated and encapsulated, and the processed data of the first transaction is sent to the destination server, and it also includes: deleting the data of the first transaction stored in the first cache storage space and / or the first disk storage space, and deleting the first object allocator corresponding to the first transaction.

[0012] According to a second aspect of the present application, there is provided a data reordering device, comprising: an acquisition module for acquiring a database log; a determination module for determining, based on the database log, a first statement to be stored and a first transaction corresponding to the first statement; a storage module for storing the first statement in a first disk storage space corresponding to the first transaction in a disk when a first cache storage space in a cache for storing the first transaction is insufficient to store the first statement or a second cache storage space in the cache for storing all transactions is insufficient to store the first statement; and a reordering module for translating and encapsulating the data of the first transaction stored in the first cache storage space and / or the first disk storage space upon receiving a reordering instruction for the first transaction, and then sending the processed data of the first transaction to a destination server.

[0013] The third aspect of the present application provides an electronic device, comprising: a memory and a processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor executes the method described in any one of the first aspect of the present application.

[0014] A fourth aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, which, when executed, implement the method described in any one of the first aspects of the present application.

[0015] A fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, implements the method described in any one of the first aspects of the present application.

[0016] The data reordering method, device, equipment, medium and product of the database provided by the present application, after the source database obtains the database log and determines the first data to be stored, it also checks whether the first cache storage space used to store the first transaction in the cache is sufficient to store the first data, and whether the second cache storage space used to store the second transaction is sufficient to store the first data, and when the first cache storage space or the second cache storage space is insufficient to store the first data, the first data is stored in the first disk storage space corresponding to the first transaction in the disk. At the same time, the source database does not overflow the data already stored in the cache to the disk, but retains it in the cache, so that the source database can store part of the data of the same transaction through the cache and the disk at the same time. After the source database subsequently receives the reordering instruction of the first transaction, it merges the data stored in the disk and the cache at the same time, processes it, and finally sends it to the destination database. Therefore, in the reordering process of the source database, even if the data to be stored is large, the data is stored in the disk by judging the storage space of the cache, so that the storage space of the cache will not be occupied by a large amount, which ensures the reasonable control of the storage space of the cache and can reduce the impact of other operations on the source database. At the same time, the source database will not overflow the data already stored in the memory to the disk for storage, but will store the data based on the object allocator, which will not trigger the serialization and deserialization of the data, thus reducing unnecessary computing overhead when reordering data and improving the execution efficiency of data reordering. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 A schematic diagram of a scenario for data synchronization of a database used in this application;

[0019] Figure 2 A schematic diagram of data processing for the source database;

[0020] Figure 3 A schematic diagram of reordering data in a source database in the prior art;

[0021] Figure 4 A schematic diagram of a flow chart of an embodiment of a method for reordering data in a database provided by the present application;

[0022] Figure 5 A schematic diagram of the source database storing data provided for this application;

[0023] Figure 6 A schematic diagram of a flow chart of an embodiment of a method for reordering data in a database provided by the present application;

[0024] Figure 7 A schematic diagram of a flow chart of an embodiment of a method for reordering data in a database provided by the present application;

[0025] Figure 8 A schematic diagram of the structure of a data reordering device for a database provided by the present application;

[0026] Fig. 9 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Figure 1 A schematic diagram of a scenario for data synchronization of a database used in this application, such as Figure 1As shown in the figure, according to the needs of some database services, data synchronization operations are sometimes required between databases to ensure the consistency of data stored in the two databases. Figure 1 In the example shown, the database providing data is recorded as the source database 10, and the database receiving data is recorded as the destination database 20. In some synchronization technologies, source synchronization software 101 runs on the source database 10, and destination synchronization software 201 runs on the destination database 20, so that data synchronization is performed between the source database 10 and the destination database 20 through the source synchronization software 101 and the destination synchronization software 201.

[0030] In a specific synchronization method, data synchronization between the source database 10 and the target database 20 can be divided into the following three stages.

[0031] The first stage: the stock data between the source database 10 and the target database 20 is initialized to obtain the basic point of data synchronization between the source database 10 and the target database 20.

[0032] The second stage: incremental data synchronization is performed between the source database 10 and the target database 20 based on the synchronization base point established by initializing data loading.

[0033] The third stage: the source database 10 and the target database 20 periodically compare and verify the source data and the target data of the data synchronization to confirm that no data is lost during the data synchronization process.

[0034] Among them, the second stage and the third stage will be in a long-term parallel state. In the second stage, when the source database 10 and the destination database 20 are synchronized, the source database 10 first parses the source database 10 online log, archive log and other log files through the source synchronization software 101 to obtain the addition, deletion and change of data, and then converts these changes into a specific message format inside the synchronization software in units of transactions, and then sends them to the destination synchronization software 201 of the destination database 20 through the private transmission protocol of the data synchronization software. Subsequently, the destination database 20 restores the obtained log files into statements supported by the destination database 20 through the destination synchronization software 201, and executes them on the destination database 20, finally realizing the storage of the data of the source database 10 in the destination database 20.

[0035] A database transaction may contain multiple data changes. Due to concurrent operations on data connections, multiple changes in a transaction may not be continuous in the database transaction log. Therefore, in the second phase of incremental synchronization, the source database 10 traverses each data to be synchronized before sending the log file, and stores the data in the cache according to the corresponding transaction. Specifically, Figure 2The schematic diagram of the source database processing data, wherein the source database 10 reads the database log, reorders the data in the database log, and finally translates and encapsulates the reordered data and sends it to the destination database 20. Subsequently, when the source database 10 reads a transaction commit event, it translates and encapsulates the data corresponding to the transaction in the cache, and then sends the processed data to the destination database 20 through the log file for storage. This processing performed by the source database 10 can also be called reordering the data. Finally, when a transaction rollback is read, all changes of this transaction are directly deleted from the cache.

[0036] In some prior arts, in order to solve the problem of large cache storage space occupied by reordering when processing long transactions, the source database 10 only retains a preset number of changes for each transaction in the cache. When the number of changes for each transaction exceeds the preset number, the subsequent changes of the transaction are appended to the disk. For example, Figure 3 Schematic diagram of data reordering by a source database in the prior art. The source database 10 stores data 1, data 2, data 3, ... corresponding to transaction A in the cache. When new data N is obtained, the data of transaction A stored in the cache exceeds the preset amount. The source database 10 stores all the data already stored in the cache in the disk, and also stores the new data N in the disk. The data of transaction A obtained subsequently will also be stored in the disk, and the source database 10 will also read the data corresponding to transaction A from the disk for translation and packaging.

[0037] However, the source database 10 is Figure 3 In the reordering process shown, if the data contained in each change is large, such as the table contains Lob data, or the table contains a large number of fields, if this transaction contains multiple sub-transactions, when the source database 10 stores the transaction data through the cache, it will occupy a large amount of cache storage space, affecting the source database 10 to perform other operations. At the same time, if a long transaction contains many small changes, but the cache memory consumed by the entire transaction is very small, it will still overflow to the disk, causing additional IO overhead. In addition, when the transaction is committed to the reordering cache and the data overflows to the disk, the serialization of the changed data (converting the structure object to binary) will be triggered, and the deserialization of the changed data (converting the binary to the structure object) will be triggered when the transaction is read from the disk and read from the cache. When the transaction change does not overflow to the disk, the serialization and deserialization process of accessing the transaction change from the memory cache is an unnecessary overhead.

[0038] Based on the problems existing in the reordering process of the above-mentioned source database 10, the present application provides a method, device, equipment, medium and product for reordering data of a database to solve the technical problem that when the source database stores transaction data through cache in the prior art, the cache storage space occupies a large amount of cache, so as to avoid affecting other operations performed on the source database. The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0039] Figure 4 This is a flow chart of an embodiment of a method for reordering data in a database provided by the present application. Figure 4 The method shown can be applied to Figure 1 In the scenario shown, it is executed by the source database 10. Specifically, Figure 4 The data reordering methods shown include:

[0040] S101: The source database 10 obtains database logs.

[0041] S102: The source database 10 determines the first data to be stored and the first transaction corresponding to the first data according to the database log obtained in S101.

[0042] Specifically, the source database 10 can obtain the database log during data synchronization and parse the database log to obtain multiple data. The source database 10 then processes each data in turn, records the data currently processed by the source database 10 as the first data, and records the transaction corresponding to the first data as the first transaction.

[0043] S103: When a first cache storage space in the cache for storing the first transaction is insufficient to store the first data, or a second cache storage space in the cache for storing all transactions is insufficient to store the first data, the first data is stored in a first disk storage space corresponding to the first transaction in the disk.

[0044] Specifically, a plurality of cache storage spaces are provided in the cache of the source database 10, each of which is used to store the data of a transaction. At the same time, a plurality of disk storage spaces are also provided in the disk of the source database 10, each of which is used to store the data of a transaction. In other words, the data of the same transaction can be stored in both the cache and the disk. Taking the first data currently being processed as an example, the source database 10 can store the first data in the first cache storage space in the cache, or can store the first data in the first disk storage space in the disk.

[0045] The source-end database 10 determines whether the first cache storage space in the cache is sufficient to store the first data, and determines whether the second cache storage space in the cache is sufficient to store the first data, specifically based on the storage space required for the first data, and determines whether to store the first data in the first disk storage space in the disk or the first cache storage space in the cache based on the judgment result.

[0046] In one embodiment, the source database 10 may provide external configuration parameters and obtain the maximum value maxObjectBytes of a single transaction cache and the maximum memory maxCacheBytes of all transaction caches input by the user.

[0047] Subsequently, the source-side database 10 determines whether the sum of the amount of the first data and the amount of data cached in the first cache storage space is greater than the maximum value of the single transaction cache. If so, it is determined that the first cache storage space is insufficient to store the first data, and the source-side database 10 stores the first data in the first disk storage space corresponding to the first transaction in the disk.

[0048] In addition, the source database 10 also determines whether the sum of the data volume of the first data and the data volume stored in the second cache storage space is greater than the maximum value of all transaction caches. If so, it is determined that the second cache storage space is insufficient to store the first data, and the source database 10 stores the first data in the first disk storage space corresponding to the first transaction in the disk.

[0049] Correspondingly, when the sum of the data volume of the first data and the data volume cached in the first cache storage space is less than or equal to the maximum value of a single transaction cache, and the sum of the data volume of the first data and the data volume stored in the second cache storage space is less than or equal to the maximum value of all transaction caches, it means that the first data can be stored in the cache, and the source database 10 stores the first data in the first cache storage space corresponding to the first transaction in the cache.

[0050] In one embodiment, when the source database 10 obtains the storage size storeBytes and memory size memoryBytes of the object allocator objectAllocator, if this transaction does not use storage, and the transaction memory size does not exceed the maximum memory of a single transaction cache after adding the current changed memory size, and does not exceed the maximum memory of all transaction caches after adding the memory size occupied by all current cached transactions, that is, storeLength == 0 && newObjectBytes < maxObjectBytes && newCacheBytes < maxCacheBytes, it means that this change can be directly added to the memory cache, that is, added to the buffers of this transaction object allocator objectAllocator, and update the memoryBytes occupied by a single transaction buffer and the memory size currentMemoryBytes occupied by all cached transactions. If the disk storage storeBytes occupied by the object allocator objectAllocator is 0, then create the corresponding overflow file cacheFile first. Then serialize the changed data into binary data and write it into cacheFile. Finally, update the storage size storeBytes of the overflow file for a single transaction.

[0051] S104: When receiving the reordering instruction of the first transaction, after translating and encapsulating the data of the first transaction stored in the first cache storage space and / or the first disk storage space, send the processed data of the first transaction to the destination server.

[0052] When the source database 10 reads a commit event of a certain transaction, that is, when receiving the reordering instruction, since the data of the first transaction may be stored in both the first cache storage space and the first disk storage space at this time, therefore, the source database 10 needs to translate and encapsulate all the data of the first transaction stored in the first cache storage space and / or the first disk storage space at this time, and then send all the processed data of the first transaction to the destination server.

[0053] In one embodiment, the source database 10 can specifically manage the storage spaces in the cache and on the disk through the object allocator objectAllocator. Among them, the object allocator can be used to store information such as transaction id (key), single transaction cache list (buffers), cache occupied memory (memoryBytes), overflow file (cacheFile), overflow file storage size (storeBytes), etc.

[0054] In one embodiment, after determining the first data and the corresponding first transaction according to the database log, the source database 10 also searches for the first object allocator corresponding to the first transaction from the created preset object allocators according to the identification information (transaction ID) of the first transaction.

[0055] If the first transaction does not have a corresponding object allocator, the source database 10 creates a first object allocator corresponding to the first transaction, creates a first cache storage space in the cache and a first disk storage space in the disk for the first object allocator, and creates cache data volume information and disk data volume information of the first transaction in the first object allocator. The cache data volume information is used to indicate the size of the first cache storage space occupied by the data of the first transaction, and the disk data volume information is used to indicate the size of the first disk storage space occupied by the data of the first transaction.

[0056] In one embodiment, the source database 10 first searches the object allocator cache objectAllocatorCache for the transaction id of the first transaction, and if found, obtains the object allocator corresponding to the transaction. Otherwise, creates an object allocator objectAllocator for the transaction and puts it into the object allocator cache objectAllocatorCache.

[0057] In one embodiment, after creating the first object allocator, the source database 10 specifically defines an object cache (ObjectCache), records a hash list (objectAllocatorCache) of all cache transaction IDs and corresponding object allocators, the memory size (currentMemoryBytes) currently occupied by all cache transactions, and the like.

[0058] Specifically, in S104, after receiving the reordering instruction of the first transaction, the source database 10 determines whether the disk data volume information storeBytes in the first object allocator is less than 0. If so, it means that the data of the first transaction is not stored in the disk. After the source database 10 translates and encapsulates the data of the first transaction stored in the first cache storage space, it can send the processed data of the first transaction to the destination server 20. When the disk data volume information storeBytes in the first object allocator is not less than 0, it means that the data of the first transaction is stored in the disk. At this time, the source database 10 needs to merge the data of the first transaction stored in the first disk storage space and the data of the first transaction stored in the first cache storage space, and then translate and encapsulate the merged data, and finally send the merged data of the first transaction to the destination server 20.

[0059] In one embodiment, when the source database 10 obtains all changes of a transaction from the reordering cache, it first searches the object allocator objectAllocator corresponding to the transaction id from the object allocator cache objectAllocatorCache. If the storage size storeBytes in the transaction object allocator is less than 0, it means that all changes of the transaction are in the memory cache buffers, and the stream of the memory cache buffers is directly returned. Otherwise, it means that part of the transaction changes are in the memory cache buffers and part are in the overflow file cacheFile. The file stream of cacheFile and the stream of memory cache buffers are obtained in sequence, and the two streams are merged and returned.

[0060] For example, Figure 5 A schematic diagram of the source database storing data provided for this application, such as Figure 5 As shown, the source database 10 stores the data of transaction A through the first cache storage space in the cache, which is recorded as data 1, data 2, data 3, etc. When the source database 10 processes the latest data N, the first cache storage space is not enough to store data N, or the second cache storage space is not enough to store data N, the source database 10 stores data N through the first disk storage space in the disk. At this time, the cache and disk of the source database 10 store data belonging to the same transaction A at the same time, so the source database 10 needs to merge and reflow the data stored in the cache and disk later.

[0061] In summary, in the data reordering method of the database provided by the present embodiment, after the source database 10 obtains the database log and determines the first data to be stored, it also checks whether the first cache storage space for storing the first transaction in the cache is sufficient to store the first data, and whether the second cache storage space for storing the second transaction is sufficient to store the first data, and when the first cache storage space or the second cache storage space is insufficient to store the first data, the first data is stored in the first disk storage space corresponding to the first transaction in the disk. At the same time, the source database 10 does not overflow the data already stored in the cache to the disk, but retains it in the cache, so that the source database 10 can store part of the data of the same transaction through the cache and the disk at the same time. After the source database 10 subsequently receives the reordering instruction of the first transaction, it merges the data stored simultaneously in the disk and the cache, processes them, and finally sends them to the destination database 20. Therefore, during the reordering process, even if the data to be stored is large, the source database 10 stores the data in the disk by judging the storage space of the cache, so that the storage space of the cache will not be occupied by a large amount, and the reasonable control of the storage space of the cache is ensured, which can reduce the impact of other operations performed on the source database 10. At the same time, the source database 10 will not overflow the data already stored in the memory to the disk for storage on the basis of storing the data to the disk. It stores the data based on the object allocator, thereby not triggering the serialization and deserialization of the data, reducing unnecessary computing overhead when reordering the data and improving the execution efficiency of the data reordering.

[0062] In one embodiment, after the source database 10 translates and encapsulates the data of the first transaction and sends the processed data of the first transaction to the destination database 20, it also deletes the data of the first transaction stored in the first cache storage space and / or the first disk storage space, and deletes the first object allocator corresponding to the first transaction.

[0063] For example, after completing a transaction, when the source database 10 receives a commit / rollback process, it deletes the data in the cache storage space, deletes the overflow file if there is a file cacheFile that overflows into the disk storage space, and deletes the object allocator objectAllocator of this transaction from the object cache ObjectCache.

[0064] Figure 6 A schematic diagram of a flow chart of an embodiment of a method for reordering data in a database provided by the present application shows Figure 4A specific implementation of storing the first data in the method shown. Among them, after the source-end database 10 determines the first data to be stored and the corresponding first transaction, it searches for the transaction id of the first transaction from the object allocator cache, and obtains the first object allocator for the first transaction if it exists, and creates the first object allocator if it does not exist. Subsequently, the source-end database 10 obtains the storage size and memory size of the first object allocator, and calculates the current transaction memory size and the total size of all transactions after the first data is stored in the cache. Subsequently, the source-end database 10 stores the first data in the memory and updates the memory occupied by a single transaction and the memory occupied by all transactions when the disk storage size is 0 and the memory of a single transaction and all transactions in the cache do not exceed the threshold. When the disk storage size is not 0 or the memory of a single transaction and all transactions in the cache exceeds the threshold, the first data is stored in the disk, and the storage size of the overflow file of a single transaction is updated.

[0065] Figure 7 A schematic diagram of a flow chart of an embodiment of a method for reordering data in a database provided by the present application shows Figure 4 A specific implementation of merging and outputting the data of the first transaction in the method shown. The source database 10 searches the transaction ID of the first transaction from the object allocator cache, and when it is determined that the disk storage size is 0, returns the data of the first transaction in the cache. Otherwise, when the disk storage size is greater than 0, the data of the first transaction overflowed to the disk is obtained, and the data of the first transaction in the cache is obtained, and the data of the first transaction in the disk and the cache are merged and then returned.

[0066] In the above embodiments of the present application, the data reordering method of the database provided by the embodiments of the present application is introduced. In order to realize the functions of the data reordering method of the database provided by the embodiments of the present application, the source database 10 as the execution subject may include a hardware structure and / or a software module, and realize the above functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether one of the above functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.

[0067] For example, Figure 8 A schematic diagram of the structure of a data reordering device for a database provided in this application, such as Figure 8 The device shown can be used to execute the data reordering method of the database as described in any of the above-mentioned embodiments of the present application. Figure 8The data query device 1000 shown includes: an acquisition module 1001, a determination module 1002, a storage module 1003 and a reordering module 1004. The acquisition module 1001 is used to acquire a database log; the determination module 1002 is used to determine the first statement to be stored and the first transaction corresponding to the first statement according to the database log; the storage module 1003 is used to store the first statement to the first disk storage space corresponding to the first transaction in the disk when the first cache storage space used to store the first transaction in the cache is insufficient to store the first statement, or the second cache storage space used to store all transactions in the cache is insufficient to store the first statement; the reordering module 1004 is used to translate and encapsulate the data of the first transaction stored in the first cache storage space and / or the first disk storage space when receiving the reordering instruction of the first transaction, and then send the processed data of the first transaction to the destination server.

[0068] The specific implementation method and principle of the data reordering device of the above database refer to the description of the data reordering method of the above database, which will not be repeated here.

[0069] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the processing module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above-mentioned module is determined. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0070] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0071] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0072] For example, Fig. 9 A schematic diagram of the structure of an electronic device provided in this application, such as Fig. 9 The electronic device shown can be used to execute the method for reordering data in a database as shown in any of the above embodiments of the present application. In one embodiment, Fig. 9The electronic device 2000 shown includes one or more processors 2001 and a memory 2002; wherein the memory 2002 is used to store computer executable instructions, and the processor 2001 can execute the computer executable instructions stored in the memory 2002. When the computer executable instructions are executed by the processor 2001, the processor 2001 implements the data reordering method of the database as shown in any of the aforementioned embodiments of the present application.

[0073] In one embodiment, if Fig. 9 The electronic device 2000 shown further includes a communication interface 2003 , wherein the processor 2001 can communicate with other devices through the communication interface 2003 , for example, the processor 2001 obtains a database log through the communication interface 2003 .

[0074] The present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed, they can be used to implement a method for reordering data in any database in the aforementioned embodiments of the present application.

[0075] An embodiment of the present application also provides a chip for executing instructions, wherein the chip is used to execute the data reordering method of any database as described above in the present application.

[0076] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements the data reordering method of any database as described above in the present application.

[0077] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for reordering data in a database, applied to a source database, characterized in that: include: Get database logs; Determining, according to the database log, first data to be stored and a first transaction corresponding to the first data; When a first cache storage space in the cache for storing the first transaction is insufficient to store the first data, or a second cache storage space in the cache for storing all transactions is insufficient to store the first data, storing the first data in a first disk storage space corresponding to the first transaction in the disk; When the reordering instruction of the first transaction is received, the data of the first transaction stored in the first cache storage space and / or the first disk storage space is translated and encapsulated, and the processed data of the first transaction is sent to the destination server.

2. The method according to claim 1, characterized in that When a first cache storage space in the cache for storing the first transaction is insufficient to store the first data, or a second cache storage space in the cache for storing all transactions is insufficient to store the first data, storing the first data in a first disk storage space corresponding to the first transaction in the disk includes: Obtaining a maximum value of a single transaction cache and a maximum value of all transaction caches in the cache; When the sum of the amount of the first data and the amount of data cached in the first cache storage space is greater than the maximum value of the single transaction cache, it is determined that the first cache storage space is insufficient to store the first data, and the first data is stored in a first disk storage space corresponding to the first transaction in the disk; When the sum of the amount of the first data and the amount of data stored in the second cache storage space is greater than the maximum value of all transaction caches, it is determined that the second cache storage space is insufficient to store the first data, and the first data is stored in the first disk storage space corresponding to the first transaction in the disk.

3. The method according to claim 2, characterized in that After determining the first data to be stored and the first transaction corresponding to the first data according to the database log, the method further includes: When the sum of the data volume of the first data and the data volume cached in the first cache storage space is less than or equal to the maximum value of the single transaction cache, and the sum of the data volume of the first data and the data volume stored in the second cache storage space is less than or equal to the maximum value of all transaction caches, the first data is stored in the first cache storage space.

4. The method according to claim 3, characterized in that After determining the first data to be stored and the first transaction corresponding to the first data according to the database log, the method further includes: According to the identification information of the first transaction, searching for a first object allocator corresponding to the first transaction from preset object allocators; When there is no corresponding object allocator for the first transaction, create a first object allocator corresponding to the first transaction, create the first cache storage space in the cache and the first disk storage space in the disk for the first object allocator, and create cache data volume information and disk data volume information of the first transaction in the first object allocator.

5. The method according to claim 4, characterized in that After translating and encapsulating the data of the first transaction stored in the first cache storage space and / or the first disk storage space, sending the processed data of the first transaction to the destination server includes: When receiving the reordering instruction of the first transaction and the disk data volume information in the first object allocator is less than 0, translating and encapsulating the data of the first transaction stored in the first cache storage space, and sending the processed data of the first transaction to the destination server; When a reordering instruction of the first transaction is received and the disk data volume information in the first object allocator is not less than 0, the data of the first transaction stored in the first disk storage space and the data of the first transaction stored in the first cache storage space are merged and translated and packaged, and the merged data of the first transaction is sent to the destination database.

6. The method according to any one of claims 1 to 5, characterized in that: After receiving the reordering instruction of the first transaction, translating and encapsulating the data of the first transaction stored in the first cache storage space and / or the first disk storage space, and sending the processed data of the first transaction to the destination server, the method further includes: The data of the first transaction stored in the first cache storage space and / or the first disk storage space is deleted, and the first object allocator corresponding to the first transaction is deleted.

7. A data reordering device for a database, characterized in that: include: Acquisition module, used to obtain database logs; A determination module, configured to determine, according to the database log, a first statement to be stored and a first transaction corresponding to the first statement; a storage module, configured to store the first statement in a first disk storage space corresponding to the first transaction in a disk when a first cache storage space in a cache for storing the first transaction is insufficient to store the first statement or a second cache storage space in the cache for storing all transactions is insufficient to store the first statement; The reordering module is used to translate and encapsulate the data of the first transaction stored in the first cache storage space and / or the first disk storage space upon receiving the reordering instruction of the first transaction, and then send the processed data of the first transaction to the destination server.

8. An electronic device, characterized in that: include: Memory and processor; The memory stores computer executable instructions; The processor executes the computer executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and when the computer executable instructions are executed, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed.

Images