A data processing method, device, storage medium and electronic device

Abstract

The specification provides a data processing method, device, storage medium and electronic equipment. In the data processing method provided in the specification, the identity of the data to be written in the data write request is determined; then it is judged whether the target linked list contains data with the same identity as the data to be written; if yes, the data with the same identity as the data to be written in the target linked list is deleted, and the data to be written is written into the first queue; if not, the data to be written is directly written into the first queue; when the second queue meets the compression condition, the data of the second queue is moved to the external storage. When the data processing method provided in the specification is used to process data in a system using a log-structured merge tree structure database, the amount of data to be processed can be effectively reduced by deleting data with the same identity, thereby avoiding the reduction of system throughput and the impact on performance caused by repeatedly writing data with the same identity.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a data processing method, apparatus, storage medium and electronic device. Background Technology

[0002] Currently, users are paying increasing attention to the privacy of their data, and the Log-Structured Merge-Tree (LSM) data structure is widely used in non-relational databases. When writing data to a database using an LSM tree structure, the data is first stored in a memory table (MemTable) located in memory, and then subsequently persisted to external storage.

[0003] Ensuring efficient data processing has become one of the problems that need to be solved in the database field. Among them, improving write efficiency is a problem that needs to be addressed, and this manual provides a data processing method to address this issue. Summary of the Invention

[0004] This specification provides a data processing method and a data processing apparatus to partially solve the aforementioned problems existing in the prior art.

[0005] The following technical solution is adopted in this specification:

[0006] This specification provides a data processing method applied to a database system employing a log-structured merge tree. The system's memory contains a target linked list, which is divided into a first queue and a second queue by target pointers. The method includes:

[0007] Receive a data write request and determine the identifier of the data to be written;

[0008] Determine whether the target linked list contains historical data with the same identifier as the data to be written;

[0009] If so, delete the historical data and insert the data to be written into the first queue. When the data in the first queue meets the transfer conditions, adjust the position of the target pointer in the target linked list and transfer the data at the end of the first queue to the second queue of the target linked list.

[0010] If not, insert the data to be written into the first queue. When the data in the first queue meets the transfer condition, adjust the position of the target pointer in the target linked list and transfer the data at the end of the first queue to the second queue of the target linked list.

[0011] When the second queue meets the compression conditions, the data in the second queue is moved to external storage.

[0012] Optionally, the memory table may further include: a discarded linked list;

[0013] Deleting the historical data specifically includes:

[0014] Remove the historical data from the target linked list and insert the historical data into the discarded linked list;

[0015] When the second queue meets the compression condition, the discarded linked list is cleared.

[0016] Optionally, the method further includes:

[0017] Receive rollback requests carrying a specified identifier;

[0018] When a specified data with the same identifier as the specified identifier exists in the discarded list, the data with the same identifier as the specified identifier in the target list is removed from the target list and inserted into the discarded list;

[0019] Remove the specified data from the discard list and insert the specified data into the first queue.

[0020] Optionally, the transfer conditions include: the amount of data stored in the first queue is not less than a first specified threshold; or, the size of the data stored in the first queue is not less than a second specified threshold.

[0021] Optionally, the tail of the first queue is connected to the head of the second queue;

[0022] Adjusting the position of the target pointer in the target linked list, and allocating the data at the tail of the first queue to the second queue of the target linked list, specifically includes:

[0023] The position of the target pointer in the target linked list is adjusted so that the data at the tail of the first queue is assigned to the head of the second queue after adjustment.

[0024] Optionally, the compression conditions include: the amount of data stored in the second queue is not less than a third specified threshold; or, the size of the data stored in the second queue is not less than a fourth specified threshold.

[0025] Optionally, the data in the second queue is moved to external storage, specifically including:

[0026] The data in the second queue is compressed to generate a sorted string table file containing the data in the second queue, wherein the storage order of the data in the sorted string table file is the same as the storage order of the data in the second queue;

[0027] Clear the second queue;

[0028] The sorted string table file is stored in external storage.

[0029] This specification provides a data processing apparatus applied to a database system employing a log structure merge tree. The system's memory contains a target linked list, which is divided into a first queue and a second queue by target pointers. The apparatus includes:

[0030] The receiving module receives data write requests and determines the identifier of the data to be written.

[0031] The judgment module determines whether the target linked list contains historical data with the same identifier as the data to be written;

[0032] If the deletion module is present, the historical data is deleted and the data to be written is inserted into the first queue. When the data in the first queue meets the transfer conditions, the position of the target pointer in the target linked list is adjusted, and the data at the end of the first queue is transferred to the second queue of the target linked list.

[0033] If the default module is not specified, the data to be written is inserted into the first queue. When the data in the first queue meets the transfer conditions, the position of the target pointer in the target linked list is adjusted, and the data at the end of the first queue is transferred to the second queue of the target linked list.

[0034] The transfer module moves the data in the second queue to an external storage device when the second queue meets the compression conditions.

[0035] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data processing method.

[0036] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described data processing method.

[0037] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0038] In the data processing method provided in this specification, a data write request is received, and the identifier of the data to be written contained in the data write request is determined. Then, it is determined whether there is data in the target linked list with the same identifier as the data to be written. If so, the data with the same identifier as the data to be written is deleted from the target linked list, and the data to be written is written to a first queue. If not, the data to be written is directly written to the first queue. When the second queue meets the compression condition, the data in the second queue is moved to external storage. When using the data processing method provided in this specification to process data in a system using a log-structured merged tree database, the amount of data to be processed can be effectively reduced by deleting data with duplicate identifiers, avoiding a decrease in system throughput and performance impact caused by repeatedly writing data with the same identifier. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0040] Figure 1 This is a flowchart illustrating a data processing method provided in this specification;

[0041] Figure 2 This is an application diagram of a data processing method provided in this specification;

[0042] Figure 3 This specification provides a schematic diagram of the structure of a first queue and a second queue in the same target linked list;

[0043] Figure 4 This specification provides a schematic diagram of a structure in which the first queue and the second queue are two independent linked lists.

[0044] Figure 5 This is a schematic diagram of a data processing device provided in this specification;

[0045] Figure 6 This specification provides a corresponding Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0047] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0048] Figure 1 This is a flowchart illustrating a data processing method provided in this specification, including the following steps:

[0049] S100: Receives a data write request and determines the identifier of the data to be written.

[0050] In this specification, the execution entity used to implement the data processing method can refer to any device with a system capable of merging a log structure tree database, such as a server or other designated device set up on the business platform. For ease of description, this specification only uses the server as the execution entity as an example to illustrate a code testing method provided in this specification.

[0051] The data processing method provided in this specification is applied to a database system employing a log-structured merge tree. In this system, a target linked list is set in the memory table, and the target linked list is divided into a first queue and a second queue by target pointers. An application diagram of the data processing method provided in this specification is shown below. Figure 2 As shown, the data to be written first enters the memory table, and after passing through the first queue and the second queue in the target linked list, it is finally saved in the form of an SST file in the external storage.

[0052] The specific structures of the first and second queues can be as follows: Figure 3 As shown. In the case of Figure 3 In the target linked list shown, data A1 is at the head of the target linked list, and data B6 is at the tail of the target linked list; the first queue contains data A1-A4, where data A1 is at the head of the first queue and data A4 is at the tail of the first queue; similarly, the second queue contains data B1-B6, where data B1 is at the head of the second queue and data B6 is at the tail of the second queue; the target pointer points to the tail of the first queue.

[0053] For a linked list-style queue, there are two methods for inserting data: head insertion and tail insertion. For ease of description and clarity, this manual assumes head insertion for all data insertions. It's worth noting that the head and tail settings of the queue and the data insertion method are not limited to the embodiment provided in this manual. It's conceivable that swapping the head and tail of the queue and adjusting the data insertion method accordingly can also achieve the linked list effect described in this method; however, this manual will not elaborate further on these points.

[0054] In an LSM tree structure, a memory component called a memory table exists in the system's memory. When new data is written, it is preferentially stored in the memory table. In the data processing method provided in this specification, the memory table contains two queues: a first queue and a second queue. Specifically, the first queue stores the most recently entered data in memory, ordered from last to first; the second queue stores the data evicted from the first queue, ordered from last to first. The sizes of both the first and second queues can be set according to requirements; typically, the amount of data stored in the second queue can be much larger than that in the first queue.

[0055] In the data processing methods provided in this specification, the identifier of data can be the key of the data. Typically, a piece of data can contain both a key and a value; that is, the data can be a key-value pair. For example, the key of a piece of data could be "User A's height," and the corresponding value could be "175"; or the key of a piece of data could be "User B's weight," and the corresponding value could be "60." Each piece of data written will contain both an identifier and a value.

[0056] In this step, when a new data write request occurs, the data write request is received, and the identifier of the data to be written contained in the data write request is determined.

[0057] S102: Determine whether the target linked list contains historical data with the same identifier as the data to be written.

[0058] In essence, a data identifier represents the object it describes, while the data value represents the specific information of that object. When two different data sets have the same identifier, it indicates that the two different data sets describe the same object. Considering time, the moments when these two data sets with the same identifier are obtained must be different. Therefore, for the data obtained first, the data obtained later is an update to the first data; in other words, the newly obtained data is more timely. When the identifier of newly obtained data is the same as the identifier of old data already existing in a memory table, then the old data records invalid data.

[0059] Therefore, in this step, it can be determined whether the target linked list contains data with the same identifier as the data to be written, that is, whether invalid data has appeared in the first queue or the second queue.

[0060] S104: If so, delete the historical data and insert the data to be written into the first queue. When the data in the first queue meets the transfer conditions, adjust the position of the target pointer in the target linked list and transfer the data at the end of the first queue to the second queue of the target linked list.

[0061] If the identifier of the data to be written is the same as the identifier of a data already existing in the target linked list, the data in the target linked list with the same identifier as the data to be written can be regarded as historical data with duplicate identifier. Furthermore, it can be considered that the historical data in the target linked list is already outdated and invalid data, and at this time, the historical data can be deleted from the target linked list.

[0062] It's worth noting that, under normal circumstances, the target linked list will only contain at most one historical data with a duplicate identifier. If neither the first nor the second queue contains data with the same identifier as the data to be written, then there is no historical data. If the first queue contains data with the same identifier as the data to be written, then the historical data exists in the first queue. Similarly, if the second queue contains data with the same identifier as the data to be written, then the historical data exists in the second queue. Since each time new data enters the memory table, the data with the same identifier as that data is deleted from the memory table, the target linked list can contain at most one piece of data with the same identifier as the data to be written at any given time—that is, historical data.

[0063] In practical applications, when deleting historical data with duplicate identifiers, the historical data is not immediately cleared. Instead, it is retained for a period of time to facilitate data recovery. Specifically, the memory table also includes: a discard queue; removing the historical data from the target linked list and inserting the historical data into the discard linked list; and clearing the discard linked list when the second queue meets the compression condition.

[0064] Whenever the second queue meets the compression conditions, the data in the second queue will be compressed and transferred to external storage. At this point, the data in the discard queue can be completely discarded.

[0065] When data exists in the discard queue, it can be restored to the first queue via rollback. Specifically, a rollback request carrying a specified identifier can be received; when specified data with the same identifier as the specified identifier exists in the discard list, the data with the same identifier as the specified identifier in the target list is removed from the target list and inserted into the discard list; the specified data is removed from the discard list and inserted into the first queue.

[0066] It is worth mentioning that when there are multiple pieces of data in the discard queue with the same identifier as the specified identifier, the data that entered the discard queue earliest will usually be rolled back.

[0067] At the same time, the data to be written can be inserted into the first queue.

[0068] S106: If not, insert the data to be written into the first queue. When the data in the first queue meets the transfer conditions, adjust the position of the target pointer in the target linked list and transfer the data at the tail of the first queue to the second queue of the target linked list.

[0069] If there is no data in the target linked list with the same identifier as the data to be written, then the data to be written can be directly written to the first queue.

[0070] In the data processing method provided in this specification, the first queue acts like a "cache." Rather than storing the data, the data in the first queue is more of a temporary storage and does not participate in the LSM tree merging process. Since the data to be written first enters the first queue after being retrieved, all data in the first queue consists of data whose identifiers have been recently accessed. Typically, data whose identifiers have been recently accessed has a high probability of being accessed again. In other words, whenever data to be written is retrieved, there is a high probability that there is data in the first queue with the same identifier as the data to be written. Because the first queue usually stores a relatively small amount of data, in many cases, data operations only need to be performed within the first queue, significantly reducing the amount of data that needs to be processed.

[0071] It is conceivable that the amount of data that the first queue can store is not unlimited. Therefore, when the first queue meets the transfer update conditions, the first specified data in the first queue can be moved to the second queue. Specifically, in this specification, the transfer conditions can be related to the data stored in the first queue. Specifically, the transfer conditions include: the amount of data stored in the first queue is not less than a first specified threshold; or, the size of the data stored in the first queue is not less than a second specified threshold. The first and second specified thresholds can be set according to specific needs.

[0072] The first specified threshold can be a specific number, such as 50, 100, or 200; the second specified threshold can be a specific data volume, such as 100 megabytes (MB) or 1 gigabyte (GB). Both the first and second specified thresholds can be set according to actual needs. In addition, the update condition can be set to a first specified percentage of the total amount of data that the first queue can hold, such as 85% or 100%. Similarly, the first specified percentage can also be set according to requirements.

[0073] In the first queue, the data is sorted in reverse order of when it entered the memory table. For example, data A, data B, and data C are three pieces of data that entered the memory table consecutively at different times, in the order of B, C, and A. Therefore, the first queue can sort these three pieces of data as A, C, and B, that is, sorted in reverse order of when they entered the memory table, exactly the reverse of the order in which they entered memory.

[0074] Meanwhile, since the first queue stores the most recently entered data, when data needs to be moved from the first queue to the second queue, the data that entered the first queue earliest can be removed. Under the above sorting method, it can be seen that the data that entered the first queue earliest, i.e., the data written to the first queue first, is placed at the end of the first queue and can be removed from the first queue and written to the second queue.

[0075] In this method, the first queue and the second queue are contiguous queues separated by a target pointer within the same linked list. Therefore, data transfer from the first queue to the second queue can be achieved simply by changing the position of the target pointer. Specifically, the tail of the first queue is connected to the head of the second queue; the position of the target pointer in the target linked list can be adjusted so that data at the tail of the first queue is allocated to the head of the second queue after adjustment.

[0076] by Figure 2 Taking the target linked list shown as an example, assuming the first queue has already met the transfer conditions, when new data A0 needs to enter, the data at the tail of the first queue, i.e., data A4, needs to be moved to the second queue. After entering the first queue, A0 will be placed at the head of the first queue, i.e., before data A1. Since the target pointer points to the tail of the first queue, it is only necessary to change the target pointer from pointing to data A4 to pointing to data A3 to complete the transfer of data from the first queue to the second queue.

[0077] S108: When the second queue meets the compression condition, the data in the second queue is moved to external memory for storage.

[0078] In this specification, the second queue is used to facilitate interaction between data in memory and external storage. The external storage can be a hard disk, floppy disk, USB flash drive, etc. The second queue sorts the data in reverse order of when it was evicted from the first queue. When the second queue meets compression conditions, the data in the second queue can be moved to the external storage for storage. The compression conditions include: the amount of data stored in the second queue is not less than a third specified threshold; or, the size of the data stored in the second queue is not less than a fourth specified threshold.

[0079] Similarly, the third specified threshold can be a specific number, and the fourth specified threshold can be a specific data volume; the third and fourth specified thresholds can also be set according to specific needs. Likewise, the compression condition can also be when the data volume stored in the second queue reaches a second specified percentage of the data volume that the second queue can store, such as 90%, 100%, etc.

[0080] Because of the off-site storage characteristic of the LSM tree structure, which first stores data in memory and then stores it in external storage, the merging of the LSM tree is triggered when the second queue meets the compression conditions. The merging process of the LSM tree can actually be a compression process, compressing the data and storing it on the hard drive as an SST file. Specifically, the data in the second queue can be compressed to generate a sorted string table file containing the data from the second queue, wherein the storage order of the data in the sorted string table file is the same as the storage order of the data in the second queue; the second queue is then cleared; and the sorted string table file is stored in external storage.

[0081] Once the data in the second queue is stored in external storage as an SST file, the second queue can be cleared, meaning all data in the second queue is removed. The second queue becomes empty and continues to receive data evicted from the first queue. It's worth noting that when the data in the second queue is compressed, the data in the first queue is not compressed simultaneously and remains unchanged; in other words, the first queue does not participate in the LSM tree merging.

[0082] When processing data in a database system using an LSM tree structure using the data processing method provided in this manual, each time new data enters the memory table, the old data with the same identifier as the new data will be deleted accordingly. This prevents the LSM tree merging process from being frequently triggered when there are a large number of writes, which would lead to a decrease in system throughput. This significantly reduces the amount of data that the system needs to process and effectively improves system performance.

[0083] Additionally, in this specification, both the first queue and the second queue only logically connect the data; they do not change the data's storage address in external memory. That is, when operations are performed on the data in the memory table, the data's storage address in external memory remains unchanged. This external memory can be non-volatile memory, such as a hard disk drive (HDD), a solid-state drive (SSD), etc.

[0084] At the same time, the first queue and the second queue may not exist in the same linked list, but rather exist as two separate linked lists, such as... Figure 4 As shown. In Figure 4 In the structure shown, when the data in the first queue meets the transfer condition, the data at the tail node of the first queue is removed and then inserted into the second queue, which can also realize the data processing method provided in this specification. Alternatively, the tail node of the first queue can be connected to the head node of the second queue, that is, the two linked lists are concatenated into one linked list, which will not be elaborated here.

[0085] The above describes one or more methods for implementing data processing as outlined in this specification. Based on the same approach, this specification also provides corresponding data processing devices, such as... Figure 5 As shown.

[0086] Figure 5 A schematic diagram of a data processing apparatus provided in this specification includes:

[0087] The receiving module 200 receives a data write request and determines the identifier of the data to be written.

[0088] The judgment module 202 determines whether the target linked list stores historical data with the same identifier as the data to be written;

[0089] If the deletion module 204 is correct, the historical data is deleted and the data to be written is inserted into the first queue. When the data in the first queue meets the transfer conditions, the position of the target pointer in the target linked list is adjusted, and the data at the end of the first queue is assigned to the second queue of the target linked list.

[0090] If the default module 206 is not specified, the data to be written is inserted into the first queue. When the data in the first queue meets the transfer conditions, the position of the target pointer in the target linked list is adjusted, and the data at the end of the first queue is transferred to the second queue of the target linked list.

[0091] The transfer module 208 moves the data in the second queue to an external memory for storage when the second queue meets the compression conditions.

[0092] Optionally, the memory table may further include: a discard queue;

[0093] The deletion module 204 is specifically used to remove the historical data from the target linked list and insert the historical data into the discarded linked list; when the second queue meets the compression condition, the discarded linked list is cleared.

[0094] Optionally, the apparatus further includes a rollback module 210, specifically configured to receive a rollback request carrying a specified identifier; when specified data with the same identifier as the specified identifier exists in the discarded list, remove the data with the same identifier as the specified identifier from the target list and insert it into the discarded list; remove the specified data from the discarded list and insert the specified data into the first queue.

[0095] Optionally, the transfer conditions include: the amount of data stored in the first queue is not less than a first specified threshold; or, the size of the data stored in the first queue is not less than a second specified threshold.

[0096] Optionally, the tail of the first queue is connected to the head of the second queue;

[0097] The default module 206 is specifically used to adjust the position of the target pointer in the target linked list, so that the data at the tail of the first queue is assigned to the head of the second queue after adjustment.

[0098] Optionally, the compression conditions include: the amount of data stored in the second queue is not less than a third specified threshold; or, the size of the data stored in the second queue is not less than a fourth specified threshold.

[0099] Optionally, the transfer module 208 is specifically used to compress the data in the second queue to generate a sorted string table file containing the data in the second queue, wherein the storage order of the data in the sorted string table file is the same as the storage order of the data in the second queue; clear the second queue; and store the sorted string table file in an external storage device.

[0100] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 This provides a data processing method.

[0101] This instruction manual also provides Figure 6 The one shown corresponds to Figure 1 A schematic diagram of the structure of an electronic device. (e.g.) Figure 6 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The data processing method described herein. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0102] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0103] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0104] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0105] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0106] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0111] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0112] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0113] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0114] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0116] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0117] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A data processing method, the method being applied to a database system employing a log structure merge tree, wherein a target linked list is set in a memory table of the system's memory, the target linked list being divided into a first queue and a second queue by target pointers, wherein, The tail of the first queue is connected to the head of the second queue, and the method includes: Receive a data write request and determine the identifier of the data to be written; Determine whether the target linked list contains historical data with the same identifier as the data to be written; If so, delete the historical data and insert the data to be written into the first queue. When the data in the first queue meets the transfer conditions, adjust the position of the target pointer in the target linked list and transfer the data at the end of the first queue to the second queue of the target linked list. If not, insert the data to be written into the first queue. When the data in the first queue meets the transfer condition, adjust the position of the target pointer in the target linked list and transfer the data at the end of the first queue to the second queue of the target linked list. When the second queue meets the compression conditions, the data in the second queue is moved to external storage.

2. The method as described in claim 1, wherein the memory table further comprises: Discarded linked list; Deleting the historical data specifically includes: Remove the historical data from the target linked list and insert the historical data into the discarded linked list; When the second queue meets the compression condition, the discarded linked list is cleared.

3. The method of claim 2, further comprising: Receive rollback requests carrying a specified identifier; When a specified data with the same identifier as the specified identifier exists in the discarded list, the data with the same identifier as the specified identifier in the target list is removed from the target list and inserted into the discarded list; Remove the specified data from the discard list and insert the specified data into the first queue.

4. The method of claim 1, wherein the transfer conditions include: The number of data stored in the first queue is not less than a first specified threshold; or, the size of the data stored in the first queue is not less than a second specified threshold.

5. The method as described in claim 1, wherein the tail of the first queue is connected to the head of the second queue; Adjusting the position of the target pointer in the target linked list, and allocating the data at the tail of the first queue to the second queue of the target linked list, specifically includes: The position of the target pointer in the target linked list is adjusted so that the data at the tail of the first queue is assigned to the head of the second queue after adjustment.

6. The method of claim 1, wherein the compression conditions include: The amount of data stored in the second queue is not less than a third specified threshold; or, the size of the data stored in the second queue is not less than a fourth specified threshold.

7. The method of claim 1, wherein moving the data of the second queue to external storage specifically includes: The data in the second queue is compressed to generate a sorted string table file containing the data in the second queue, wherein the storage order of the data in the sorted string table file is the same as the storage order of the data in the second queue; Clear the second queue; The sorted string table file is stored in external storage.

8. A data processing apparatus, the apparatus being applied to a database system employing a log-structured merge tree, wherein a target linked list is configured in a memory table of the system's memory, the target linked list being divided into a first queue and a second queue by target pointers, wherein, The tail of the first queue is connected to the head of the second queue, and the device includes: The receiving module receives data write requests and determines the identifier of the data to be written. The judgment module determines whether the target linked list contains historical data with the same identifier as the data to be written; If the deletion module is present, the historical data is deleted and the data to be written is inserted into the first queue. When the data in the first queue meets the transfer conditions, the position of the target pointer in the target linked list is adjusted, and the data at the end of the first queue is transferred to the second queue of the target linked list. If the default module is not specified, the data to be written is inserted into the first queue. When the data in the first queue meets the transfer conditions, the position of the target pointer in the target linked list is adjusted, and the data at the end of the first queue is transferred to the second queue of the target linked list. The transfer module moves the data in the second queue to an external storage device when the second queue meets the compression conditions.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1 to 7.

Images