Data processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data processing method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: storing data into a target copy storage hierarchy in copy storage hierarchies, merging a target sorting character string table in the target copy storage hierarchy with a sorting character string table in a lower copy storage hierarchy, and according to each merged sorting character string table in each copy storage hierarchy, sorting the sorted character string table in each copy storage hierarchy; and obtaining a write-in result of the data according to the sequence character string tables and the unmodified sequence character string tables. Compared with a traditional method of performing data sink writing through a lock mode, multiple versions and the like, the method has the advantages that each copy storage hierarchy of each storage hierarchy in the hard disk is generated, and the data is subjected to generation and sinking of the sorting character string in the copy storage hierarchy corresponding to the storage hierarchy, so that when multiple threads perform concurrent processing on the data, the storage hierarchy of the storage hierarchy is not influenced; and the data consistency can be ensured without locking, so that the data processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Concurrent processing is an effective way to improve computer processing efficiency and performance. Concurrent control of memory data requires handling concurrent reads, writes, refreshes, and merges. Currently, processing memory data typically involves using locks to ensure a single processing operation, or using multi-version models combined with version control strategies to query data across multiple versions. However, processing memory data using locks or multi-version models can reduce data processing efficiency due to lock contention and the large number of versions.

[0003] Therefore, the current method for processing data has the defect of low processing efficiency. Summary of the Invention

[0004] Based on this, it is necessary to provide a data processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve processing efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a data processing method, comprising:

[0006] Acquire data to be written, and acquire, based on the data, each replicated storage layer corresponding to each storage layer in the hard disk;

[0007] Storing the data in a target replication storage layer in each replication storage layer to obtain a target sorted string table;

[0008] Merging the target sorted string table with the sorted string tables in the respective duplicate storage levels of the lower layers to obtain merged sorted string tables;

[0009] The data writing result is obtained according to the merged sorted string tables in the respective duplicate storage levels and the unchanged sorted string tables in the respective duplicate storage levels.

[0010] In this embodiment, the server generates replica storage levels for each storage level in the hard disk, and generates and sinks sorting strings for the data in the replica storage levels corresponding to the storage levels, so that when multiple threads process the data concurrently, data consistency can be guaranteed without locking, thereby improving data processing efficiency.

[0011] In one embodiment, the target replication storage tier includes a first replication storage tier; the first replication storage tier corresponds to a first storage tier in each of the storage tiers; the first storage tier represents a top storage tier in each of the storage tiers;

[0012] The step of storing the data in a target replication storage layer in each replication storage layer to obtain a target sorted string table includes:

[0013] The data is stored in the first replication storage level, and a target sorting string table corresponding to the data in the first replication storage level is obtained.

[0014] In this embodiment, the server flushes the above data and stores the data in the replica storage layer corresponding to the top storage layer to form a target sort string table, instead of modifying the original storage layer. Other threads can still process the storage layer to achieve lock-free concurrent processing, so that the LSM structure fully utilizes processor resources and adopts lock-free concurrency control to avoid blocking caused by synchronization and significantly improve data processing efficiency.

[0015] In one embodiment, merging the target sorted string table with sorted string tables in each of the lower replication storage levels to obtain merged sorted string tables includes:

[0016] obtaining, in the replicated storage tier below the target replicated storage tier, respective sorted string tables associated with the target sorted string table;

[0017] Merging the target sorted string table with each of the associated sorted string tables to obtain a new target replication storage level including the merged target sorted string table;

[0018] Return to executing the step of obtaining the respective sorting string tables associated with the target sorting string table in the replicated storage hierarchy below the target replicated storage hierarchy, until the target replicated storage hierarchy is the lowest target replicated storage hierarchy or there is no sorting string table associated with the target sorting string table, and obtain the merged sorting string tables in the respective replicated storage hierarchies.

[0019] In this embodiment, the server can sink and merge the associated sort string tables based on the replication storage level, so that the storage level can be updated without using a locking mechanism, and each storage level can be processed concurrently, thereby improving the processing efficiency of sinking and merging in the process of processing the above data, and can be well applied to LSM structures of various different modes.

[0020] In one embodiment, the method further comprises:

[0021] When detecting that the data changes to a target state, adding a first recycling marker to one or more of the mutable memory table and the immutable memory table by using a target thread of the data;

[0022] When it is detected that the first recycling identifier exists in one or more of the variable memory table and the immutable memory table and the existence time of the first recycling identifier meets the time threshold, resources of one or more of the variable memory table and the immutable memory table are recycled through the target thread.

[0023] In this embodiment, the server uses a pulse clock mechanism to achieve concurrent processing and recycling of data in the memory without locking, thereby improving the efficiency of data processing in the memory, and achieving the most timely recycling of resources, thereby minimizing unnecessary resource waste.

[0024] In one embodiment, the data includes one or more of a mutable memory table and an immutable memory table;

[0025] The detecting that the data changes to a target state includes:

[0026] When it is detected that the data is converted from the mutable memory table to the immutable memory table and / or the immutable memory table is stored in the target replication storage tier in each of the replication storage tiers, it is determined that the data is changed to the target state.

[0027] In this embodiment, the server detects whether the data meets the recycling conditions by detecting the state changes of the data from the mutable memory table to the immutable memory table, and from the immutable memory table to the target replication storage layer, thereby improving the efficiency of data processing in the memory, and realizing the timely recycling of resources, thereby minimizing unnecessary resource waste.

[0028] In one embodiment, after obtaining the data writing result based on the merged sorted string tables in each of the replicated storage tiers and the unchanged sorted string tables in each of the replicated storage tiers, the method further includes:

[0029] adding a second recycling marker in each of the storage tiers by using a target thread of the data;

[0030] When it is detected that the second recycling mark exists in the storage layer and the existence time of the second recycling mark meets the time threshold, resources of each storage layer are recycled through the target thread.

[0031] In this embodiment, the server generates new versions of each storage layer based on the replicated storage layer. Other threads can still perform read operations during the sinking and merging process of the storage layer, which meets the concurrent processing requirements of each thread for the storage layer and improves data processing efficiency; and uses the pulse clock mechanism to recycle the old storage layers, reducing resource waste.

[0032] In one embodiment, before obtaining the data writing result based on the merged sorted string tables in each of the replicated storage tiers and the unchanged sorted string tables in each of the replicated storage tiers, the method further includes:

[0033] When a query request for the sorted string table is received, each sorted string table in each storage level is queried according to the query request to obtain target data corresponding to the query request.

[0034] In this embodiment, the server updates the version by copying the storage layer, and when the update is not completed, other threads access the storage layers from the old version, ensuring the concurrent processing of reading, writing, merging and refreshing tasks in a lock-free state, thereby improving data processing efficiency.

[0035] In a second aspect, the present application further provides a data processing device, comprising:

[0036] An acquisition module, configured to acquire data to be written, and acquire, based on the data, each replicated storage layer corresponding to each storage layer in the hard disk;

[0037] a storage module, configured to store the data into a target replication storage layer in each of the replication storage layers to obtain a target sorted string table;

[0038] a merging module, configured to merge the target sorted string table with the sorted string tables in the respective replication storage levels of the lower layers to obtain merged sorted string tables;

[0039] The writing module is configured to obtain a writing result of the data according to the merged sorted string tables in the replicated storage levels and the unchanged sorted string tables in the replicated storage levels.

[0040] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0042] In a fifth aspect, the present application also provides a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0043] The above-mentioned data processing method, apparatus, computer equipment, computer-readable storage medium and computer program product obtain a target sort string table by storing data in a target replicated storage layer in a replicated storage layer corresponding to a storage layer in a hard disk, merge the target sort string table with the sort string tables in each replicated storage layer in the lower layer to obtain the merged sort string tables, and obtain the data writing result based on the merged sort string tables in each replicated storage layer and the unchanged sort string tables. Compared with the traditional method of sinking data and writing through lock mode and multiple versions, this solution generates and sinks sort strings for data in the replicated storage layer corresponding to the storage layer by generating each replicated storage layer for each storage layer in the hard disk, so that when multiple threads process data concurrently, data consistency can be guaranteed without locking, thereby improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 1 is a flow chart of a data processing method in one embodiment;

[0046] Figure 2 is a schematic diagram of a merging step in one embodiment;

[0047] Figure 3 is a schematic diagram of a merging step in another embodiment;

[0048] Figure 4 is a schematic diagram of a merging step in yet another embodiment;

[0049] Figure 5 A schematic diagram of a merging step in yet another embodiment;

[0050] Figure 6 is a timing diagram of the pulse management steps in one embodiment;

[0051] Figure 7 Schematic diagram of the recovery step in one embodiment;

[0052] Figure 8 is a schematic flow diagram of the recovery step in another embodiment;

[0053] Figure 9 Schematic diagram of the process of version update in one embodiment;

[0054] Figure 10 is a flow chart of a data processing method in another embodiment;

[0055] Figure 11 is a structural block diagram of a data processing device in one embodiment;

[0056] Figure 12 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0058] In related technologies, the Log-Structured-Merge-Tree (LSM tree) is a data structure designed to optimize write performance. By first writing data to memory and then periodically persisting it to disk, it converts random writes into sequential writes, reducing disk I / O operations and significantly increasing write speeds, albeit slightly reducing read performance. It is often used to address random read and write efficiency issues on hard disks.

[0059] To improve computer processing efficiency and performance, memory data storage and processing can currently be handled concurrently. For concurrency control, LSM needs to handle concurrent read and write operations, as well as concurrent refresh and merge operations. For these processes, lock mode and multi-version mode are currently used according to different requirements.

[0060] The lock mode protects the method for persisting in-memory tables to disk, ensuring that only one persist operation is in progress at a time. The merge of existing SSTables (Sorted String Tables) is also protected by locks to ensure serialization of the merge operation. This ensures data sequentiality and consistency. However, during execution, there is a significant amount of lock contention, resulting in synchronization blockages and reduced concurrency.

[0061] For multi-version mode, LSM trees support the coexistence of multiple versions of data in a concurrent environment. LSM trees achieve high concurrency by maintaining multiple memtables (mutable memory tables) and SSTables, and utilizing WAL (Write-Ahead Logging) and version control. However, as the number of versions increases, read operations may need to check multiple SSTables, resulting in reduced query performance. Furthermore, memtable flushes still require locking protection.

[0062] In view of this, in order to optimize LSM concurrency control, this solution combines technologies such as multi-version, multi-queue, and pulse clock to build an LSM tree structure operation mode that is as lock-free and concurrent as possible. Its main content is divided into two parts: memory table lock-free and version lock-free.

[0063] In one embodiment, Figure 1 As shown, a data processing method is provided. This embodiment uses the method applied to a server as an example. It is understandable that the method can also be applied to terminals, which can be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. It can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server, including the following steps S202 to S208. Among them:

[0064] Step S202 , obtaining data to be written, and obtaining each replicated storage layer corresponding to each storage layer in the hard disk according to the data.

[0065] The data to be written can be data in memory that needs to be written to disk. This process can be based on an LSM tree. The data in memory can first be written to a memory table. This memory table can be a lock-free memory table. To achieve lock-free memory tables, the server can define that only one thread updates the memory data, while other threads are read-only.

[0066] The aforementioned memory tables include Memtable and Immutable MemTable (immutable memtable). A Memtable is an in-memory data structure, typically an ordered red-black tree or similar balanced search tree, and is typically used as a write buffer. When data items are written to the database, they are first written to the MemTable. Because the MemTable is in memory, it provides very fast write performance. When the MemTable reaches a certain size, it is converted into an Immutable MemTable. The Immutable MemTable is an intermediate state between the MemTable and the SSTable. It is primarily used to prevent data updates from being blocked during the transfer process (new writes are handled by the new MemTable). Like the MemTable, the Immutable MemTable is an in-memory data structure of the LSM tree, but it is immutable. Multiple Immutable Memtables can exist, and data in each Immutable Memtable is unordered and may be duplicated. Data is sorted by time and updated when new data is generated.

[0067] When writing, data first enters the current Memtable and is synchronously written to the WAL to ensure data is not lost. When the Memtable is full, the server converts the Memtable to an Immutable MemTable and creates a new Memtable. The background thread writes the Immutable Memtable to the disk to form an SSTable.

[0068] The server's hard drive may include multiple storage tiers, each storing one or more sorted string tables. When writing and updating data, the server can use a COW (Copy-On-Write) approach to write and update data. The server updates the structure of the storage tiers on the hard drive by progressively moving the sorted string tables down through each tier. When writing data to the hard drive, the server can retrieve the corresponding replicated storage tiers for each storage tier on the hard drive based on the data to be written. In other words, the server can replicate the storage tiers on the hard drive to obtain corresponding replicated storage tiers, also known as shadow tiers. When writing data to the hard drive, the data can be stored by progressively moving down and merging the data within each storage tier. Therefore, the replicated storage tiers can be copied only when the sorted string tables for the corresponding storage tiers need to be updated. It should be noted that each replicated storage tier corresponds one-to-one to each storage tier.

[0069] Step S204 : storing the data into a target replication storage level in each of the replication storage levels to obtain a target sorted string table.

[0070] When writing the above data to the hard disk, it can first be stored in the corresponding storage layer. The server writes the data by storing the data in the replica storage layer corresponding to the storage layer and updating the storage layer after storing the data. If there are multiple storage layers, there are also multiple corresponding replica storage layers. The server can store the above data in the target replica storage layer in each replica storage layer to obtain a target sorted string table. The sorted string table can be a persistent, immutable, ordered data file.

[0071] The target replicated storage tier may be the replicated storage tier corresponding to the first storage tier to which the data is first stored when written to the hard disk, and the target sorted string table may be the sorted string table converted from the data in the target replicated storage tier. The first storage tier may be determined based on the distribution of the various storage tiers on the hard disk. For example, the server may select the top storage tier as the first storage tier to which the data is first stored.

[0072] Step S206 : merging the target sorted string table with the sorted string tables in the lower replication storage layers to obtain merged sorted string tables.

[0073] The storage hierarchy has multiple levels. After the server stores the data in the target replication storage hierarchy, the target sorting string table in the target replication storage hierarchy can be merged downward. One or more sorting string tables can be stored in the lower replication storage hierarchy corresponding to the target replication storage hierarchy. The server can merge the target sorting string table with the sorting string tables in the lower replication storage hierarchy based on a data comparison between the target sorting string table and the sorting string tables, thereby obtaining the merged sorting string tables. The above-mentioned merging can be to merge the sorting string table in the upper replication storage hierarchy into the lower replication storage hierarchy.

[0074] In step S208 , a writing result of the data is obtained according to the merged sorted string tables in the replicated storage levels and the unchanged sorted string tables in the replicated storage levels.

[0075] The sorted string tables involved in the merge can be associated sorted string tables. That is, the sorted string tables associated with the data need to be merged, so that the server can obtain the merged sorted string tables in each replicated storage layer. Other unrelated sorted string tables do not need to be merged and still exist in the corresponding replicated storage layers. The server can then obtain the unchanged sorted string tables in each replicated storage layer.

[0076] The server can thus obtain the data write results based on the merged sorted string tables in each of the replicated storage tiers and the unchanged sorted string tables in each of the replicated storage tiers. The write results include data writes and storage tier updates. For example, based on the replicated storage tiers containing the merged sorted string tables and the unchanged sorted string tables, the server can generate new versions of the storage tiers corresponding to the respective storage tiers, thereby storing the data and updating the respective storage tiers.

[0077] In the above data processing method, the target sort string table is obtained by storing the data in the target replicated storage layer in the replicated storage layer corresponding to the storage layer in the hard disk, and the target sort string table is merged with the sort string tables in the replicated storage layers below to obtain the merged sort string tables. The data writing result is obtained based on the merged sort string tables in the replicated storage layers and the unchanged sort string tables. Compared with the traditional method of sinking data through lock mode and multiple versions, this solution generates and sinks the sort string of the data in the replicated storage layer corresponding to the storage layer by generating each replicated storage layer of each storage layer in the hard disk. When multiple threads process the data concurrently, the consistency of the data can be guaranteed without locking, thereby improving the data processing efficiency.

[0078] In one embodiment, the data is stored in a target replication storage layer in each of the replication storage layers to obtain a target sorting string table, including: storing the data in the first replication storage layer to obtain a target sorting string table corresponding to the data in the first replication storage layer.

[0079] In this embodiment, the target replicated storage tier includes a first replicated storage tier. The first replicated storage tier corresponds to the first storage tier in each of the aforementioned storage tiers, and the first storage tier represents the top storage tier in each of the aforementioned storage tiers. That is, the server may first store the data in the replicated storage tier corresponding to the top storage tier in each of the aforementioned storage tiers to obtain the target sorted string.

[0080] Specifically, the server stores the above data in the first replication storage layer, which can be a data flushing process. The server first writes the data in the memory into the Memtable, which is a mutable memory table; when the Memtable reaches a certain size, it converts it into an Immutable memtable as the data to be written. The data write merging process can be as follows: Figure 2 As shown, Figure 2 FIG. 1 is a flow chart of a merging step in an embodiment. level N represents the Nth storage level. Figure 2 It can be seen that the server includes multiple storage levels, and the corresponding SSTable (sorted string) can be generated after the above-mentioned Immutable memtable is flushed. The server uses LSM for version control and manages the SSTables of each storage level on the hard disk. Among them, the LSM version update can be a metadata update, which involves the hierarchical change of SSTable, which conforms to the characteristics of more reading and less writing, and the server uses COW mode in memory to support read and write concurrency. Figure 2 In the process, the updates of LSM structures such as the generation of SSTbale of level 10 by flushing the Immutable memtable and the merging of level 0 into level 1 are all processed in COW mode.

[0081] Among them, the data in the memory is written to the current Memtable through the Write operation, and is synchronously written to the WAL to ensure that the data is not lost. When the Memtable is full, the server converts the Memtable into an Immutable memtable and creates a new Memtable. The background thread writes the Immutable memtable to the hard disk to form an SSTable. The generation and sinking of SSTable will cause the corresponding storage level to be updated. When the storage level is updated, the server will copy the changed storage level, which is called the shadow storage level, shadow level or copied storage level. The server makes modifications on it instead of modifying the original storage level. This achieves lock-free concurrent processing of data in the storage level. Lock-free refers to the use of some technical means in multi-threaded and concurrent programming to avoid or reduce the locking operations of shared resources, thereby improving program concurrency and performance.

[0082] Through this embodiment, the server can flush the above data and store the data in the replica storage layer corresponding to the top storage layer to form a target sort string table, instead of modifying the original storage layer. Other threads can still process the storage layer to achieve lock-free concurrent processing, so that the LSM structure can make full use of processor resources and adopt lock-free concurrency control to avoid blocking caused by synchronization and significantly improve data processing efficiency.

[0083] In one embodiment, the target sort string table is merged with the sort string tables in the lower replicated storage layers to obtain the merged sort string tables, including: obtaining the sort string tables associated with the target sort string table in the replicated storage layer below the target replicated storage layer; merging the target sort string table with the associated sort string tables to obtain a new target replicated storage layer containing the merged target sort string table; returning to execute the step of obtaining the sort string tables associated with the target sort string table in the replicated storage layer below the target replicated storage layer, until the target replicated storage layer is the lowest target replicated storage layer or there is no sort string table associated with the target sort string table, to obtain the merged sort string tables in the replicated storage layers.

[0084] In this embodiment, the aforementioned storage hierarchy can include multiple layers, each of which can generate a replicated storage hierarchy, allowing the server to process data within the replicated storage hierarchy. After the aforementioned data is stored in the target replicated storage hierarchy to form a target sorted string table, data can be merged downwards. Specifically, the target sorted string table is matched with the sorted string tables in the underlying storage hierarchies to determine whether a merge is necessary.

[0085] For example, the target replication storage hierarchy is the replication storage hierarchy corresponding to the top storage hierarchy as a starting point. The server can obtain the sorted string tables associated with the target sorted string table in the replication storage hierarchy below the target replication storage hierarchy, and merge the target string table with the associated sorted string tables to obtain a new target replication storage hierarchy including the merged target string table. The new target replication storage hierarchy can be a new replication storage hierarchy obtained by the server updating the replication storage hierarchy below the target replication storage hierarchy; the new target replication storage hierarchy can include the merged sorted string table and the unchanged sorted string table in the replication storage hierarchy below the target replication storage hierarchy.

[0086] If the server detects that the new target replication storage layer is not the lowest storage layer among the various storage layers, the server can start with the new target replication storage layer, return to execute the above-mentioned steps of obtaining the various sorting string tables associated with the above-mentioned target sorting string table in the above-mentioned replication storage layer below the above-mentioned target replication storage layer, and perform the next round of sinking and merging until the above-mentioned target replication storage layer is the lowest target replication storage layer, or when there is no sorting string table associated with the merged target sorting string table in the lower replication storage layer, the server can obtain the merged sorting string tables in the various replication storage layers. Thus, the various replication storage layers after sinking and merging contain the unchanged sorting string tables and the target sorting string tables obtained after sinking and merging. Then, the various replication storage layers after sinking and merging can form a new version of the storage layer to replace the above-mentioned storage layers.

[0087] Specifically, take the example of N storage layers, where N is a natural number. Figure 3 As shown, Figure 3 This is a schematic diagram of the merging step in another embodiment. For the sinking merge between the top level 0 and the second level 1, the storage level 0 contains the target sorting string table corresponding to the above data and the original sorting string in the level 0 layer, and the level 1 contains the sorting string table associated with the target sorting string table and the original sorting string table in the level 0 layer, such as Figure 3 The server creates a replicated storage layer corresponding to level 0 and level 1, merges each sorted string table in level 0 with the first sorted string table in level 1, and maintains the two unrelated sorted string tables in level 1 unchanged, thus obtaining the merged level 1 layer.

[0088] like Figure 4 As shown, Figure 4This is a schematic diagram of the merging step in another embodiment. For the sinking merge between the second layer level 1 and the third layer level 2, assuming that the first target string in level 1 is associated with the first and second sorting string tables in level 2, the server can generate corresponding replication storage layers for level 1 and level 2, merge the first target sorting string table in level 1 into the first sorting string table and the second sorting string table in level 2, respectively, and maintain the two unrelated sorting string tables in level 1 unchanged in the level 1 layer, thereby obtaining the merged level 2 layer. The merged level 2 layer includes the merged target sorting string tables and the unchanged sorting string table.

[0089] like Figure 5 As shown, Figure 5 This is a schematic diagram of the merging step in another embodiment. If the server detects that there is no association between the sorting string tables between the level2 layer and the levelN layer, the server can stop the sinking and merging of the sorting string tables in the above-mentioned level2 layer. In addition, the sorting string tables in each replicated storage layer are not necessarily updated, only the updates when there is actual data modification. Therefore, the server uses the replicated storage layer after the merge is completed and combines it with other unchanged storage layers to generate a new LSM structure, and can concurrently perform merge or refresh tasks on different storage layers without conflict. Among them, the new LSM structure represents a new version, that is, the server can use the above-mentioned merged replicated sorting string tables to form each storage layer after the version update.

[0090] Through this embodiment, the server can sink and merge the associated sort string tables based on the replication storage level, so that the storage level can be updated without using a locking mechanism, and each storage level can be processed concurrently, thereby improving the processing efficiency of sinking and merging in the process of processing the above data, and can be well applied to LSM structures of various different modes.

[0091] In one embodiment, it also includes: when it is detected that the above-mentioned data changes to the target state, a first recycling marker is added to one or more of the above-mentioned variable memory table and the immutable memory table by using the target thread of the above-mentioned data; when it is detected that the above-mentioned first recycling marker exists in one or more of the above-mentioned variable memory table and the immutable memory table and the existence time of the above-mentioned first recycling marker meets the time threshold, one or more of the above-mentioned variable memory table and the immutable memory table are recycled through the above-mentioned target thread.

[0092] In this embodiment, the target state indicates a state in which recycling can be performed. In some embodiments, the data may be stored in one or more of a mutable memory table and an immutable memory table.

[0093] In one embodiment, detecting that the above-mentioned data changes to the target state includes: when detecting that the above-mentioned data is converted from the mutable memory table to the immutable memory table and / or the above-mentioned immutable memory table is stored in the target replication storage layer in each of the above-mentioned replication storage layers, determining that the above-mentioned data changes to the above-mentioned target state.

[0094] In this embodiment, the above-mentioned data can be a variable memory table or an immutable memory table. Data can first be written to a variable memory table in memory, and when the variable memory table reaches a certain size, it can be converted to an immutable memory table. The immutable memory table can be stored in the target replication storage layer by flushing it down to form a sorted string table. When the variable memory table is converted to an immutable memory table and the immutable memory table is flushed down to the target replication storage layer, the corresponding resources need to be recycled. For example, after the variable memory table is converted to an immutable memory table, the variable memory table is recycled; after the immutable memory table is flushed down to the target replication storage layer, the immutable memory table is recycled. Therefore, through this embodiment, the server detects whether the data meets the recycling conditions by detecting the state changes from the variable memory table to the immutable memory table and the immutable memory table to the target replication storage layer, thereby improving the efficiency of data processing in memory, and realizing the timely recycling of resources as much as possible, thereby minimizing unnecessary resource waste.

[0095] Among them, the server can recycle the variable memory table and the immutable memory table by means of a pulse clock. For example, the server adds a first recycling identifier to one or more of the above-mentioned variable memory table and the immutable memory table by using the target thread of the above-mentioned data. Among them, the target thread can be a thread that performs read and write operations on the above-mentioned data. When the server detects that the above-mentioned first recycling identifier exists in one or more of the above-mentioned variable memory table and the immutable memory table, and the existence time of the above-mentioned first recycling identifier meets the time threshold, the server can recycle resources of one or more of the above-mentioned variable memory table and the immutable memory table through the above-mentioned target thread. That is, the server detects the period of existence of the above-mentioned first recycling identifier through a pulse clock, and then determines whether recycling is required. Among them, the above-mentioned time threshold can be set according to actual conditions.

[0096] Specifically, when a thread performs a read process, it queries the immutable memtable, referencing the structure. In this solution, the read, write, and flush processes can occur in different threads. Therefore, both the immutable and mutable memtables meet the following rules: Memtable memory resources are allocated and released by the read and write threads. After the immutable memtable is solidified and flushed, or after the memtable is converted, only the memory is marked as reclaimable, for example, by adding a first reclaim marker and obtaining a pulse clock, lasteT, which represents the time when the first reclaim marker was added. The read and write threads can periodically check and call the resource release API to obtain the current pulse clock, curT. For example, with a time threshold of 2T, if the server detects a reclaimable marker, such as the first reclaim marker, and curT – lastT >= 2T, the memory is released. 2T represents two pulse periods.

[0097] Among them, the working process of the pulse clock is as follows Figure 6 As shown, Figure 6 The following is a timing diagram of the pulse management steps in one embodiment. Each task in each thread, such as conversion tasks and data processing tasks like sinking, can be managed based on a pulse clock. Here, nT represents the nth pulse clock cycle, and the pulse mechanism is composed of a pulse start point and a pulse end point. A pulse round refers to a set of tasks selected at the pulse start point, which are confirmed to be fully completed at the pulse end point. Only after a pulse round is completed can the next pulse round be initiated. The pulse clock indicates that a certain time interval is added between pulses to promote the quasi-periodic occurrence of pulses, and the clock increments at the end of each pulse round.

[0098] Specific to the processing of Memtable, such as Figure 7 As shown, Figure 7 It is a flowchart of the recycling step in one embodiment. The resource thread can be a thread that uses Memtable and can perform resource initialization, use and recycling thread tasks; the flush thread can be a thread that flushes the Immutable memtable to the target sort string table and can mark the Memtable as recyclable; other threads can be other read-only threads. After the resource thread converts the Memtable to Immutable memtable and flushes it to the target sort string table, it can mark the Memtable and the Immutable memtable, for example, by adding a first recycling mark, which can mark the above-mentioned "in-use" resources as recyclable. When the server recognizes that the resources are recyclable through the resource thread and the pulse clock condition is met, the server can recycle the corresponding resources through the resource thread.

[0099] Among them, such as Figure 8 As shown, Figure 8 This is a flowchart of the recycling step in another embodiment. The target Memtable may be a variable memory table that needs to be determined whether it can be recycled. The server detects the presence of a first recycling identifier and further determines whether the existence duration of the first recycling identifier meets a time threshold, such as whether it is greater than or equal to two pulse clock cycles. If not, the Memtable is not recycled; if so, the server can recycle the Memtable through the resource thread.

[0100] Therefore, when the server ensures that writing and resource recycling are in the same thread, it can ensure that there is no concurrency in resource application or release at the same time without locking. Among them, write operations include but are not limited to unique resource application and initialization triggering methods. At the same time, because each clock push represents that the task set at the starting point of the previous clock has been completed, it is set that each time the data is queried, the Memtable currently marked as recyclable will not be queried, thereby ensuring that after two or more clocks that the Memtable is marked as recycled, there will be no more executing or newly generated tasks using itself. At this time, the server release program can safely reclaim the resources occupied by the Memtable without locking.

[0101] When threads other than the one using the memtable query data, they always retrieve the latest table and do not modify related resources. This ensures safe, lock-free concurrent execution of tasks. For read operations by threads other than the one using the memtable, data in the immutable memtable can be accurately and correctly queried, ensuring data timeliness and consistency in a lock-free concurrency environment.

[0102] Through this embodiment, the server uses a pulse clock mechanism to achieve concurrent processing and recycling of data in the memory without locking, thereby improving the efficiency of data processing in the memory, and achieving the most timely recycling of resources, thereby minimizing unnecessary resource waste.

[0103] In one embodiment, after obtaining the write result of the above-mentioned data based on the merged sorted string tables in each of the above-mentioned replicated storage layers and the unchanged sorted string tables in each of the above-mentioned replicated storage layers, it also includes: adding a second recycling mark in each of the above-mentioned storage layers by using the target thread of the above-mentioned data; when it is detected that the above-mentioned second recycling mark exists in the above-mentioned storage layer and the existence time of the above-mentioned second recycling mark meets the time threshold, reclaiming resources of each of the above-mentioned storage layers through the above-mentioned target thread.

[0104] In this embodiment, for each storage layer after sinking and merging, the server generates new versions of each storage layer based on each copy storage layer after merging the sorted string table and each unchanged storage layer, and then the server can recycle each old storage layer.

[0105] The above recycling can also be performed using a pulse clock mechanism. For example, the server can use the target thread of the data to add a second recycling mark in each of the above storage tiers. When the server detects the presence of the second recycling mark in the above storage tier and the existence time of the second recycling mark meets the duration threshold, the target thread will reclaim the resources of each of the above storage tiers. The duration threshold can be set according to actual conditions, for example, it can be two pulse clock cycles.

[0106] Specifically, if Figure 9 As shown, Figure 9 This is a flowchart illustrating the version update process in one embodiment. When a version is updated, changes to the sorted string table in the storage tier are applied to each replicated storage tier of the new version. Once the change is complete, a new version is generated. After the change is complete, the new version replaces the old version, and the old version is added to the list of pending versions. Pending versions are recycled using a pulse clock method, creating a lock-free management structure.

[0107] Through this embodiment, the server generates new versions of each storage layer based on the replicated storage layer, and each other thread can still perform read operations during the sinking and merging process of the storage layer, which meets the concurrent processing requirements of each thread for the storage layer and improves data processing efficiency; and uses the pulse clock mechanism to recycle the old storage layers, reducing resource waste.

[0108] In one embodiment, before obtaining the writing result of the above-mentioned data based on the merged above-mentioned sorting string tables in each of the above-mentioned replicated storage levels and the unchanged sorting string tables in each of the above-mentioned replicated storage levels, it also includes: when a query request for the above-mentioned sorting string table is received, querying each of the above-mentioned sorting string tables in each of the above-mentioned storage levels according to the above-mentioned query request to obtain the target data corresponding to the above-mentioned query request.

[0109] In this embodiment, the server realizes the sinking and merging of the data in each sorting string table by copying the storage hierarchy, and generates new versions of each storage hierarchy. When the sinking and merging and new version generation process is not completed, other threads can make query requests for the sorting string tables in the storage hierarchy, and then what other threads may query at this time may be the sorting string tables in each storage hierarchy that have not been updated. For example, when the server receives a query request for the above-mentioned sorting string table, it can query each of the above-mentioned sorting string tables in each of the above-mentioned storage hierarchies according to the above-mentioned query request to obtain the target data corresponding to the above-mentioned query request. Among them, the above-mentioned storage hierarchies represent the storage hierarchies that have not been updated, and the sorting string tables in each of the storage hierarchies that have not been updated are also sorting string tables that have not been updated.

[0110] Specifically, the version update and recycling tasks of the above-mentioned storage levels can be completed by one thread. Before the version change is completed, the read requests of other threads will enter the sorted string tables of each level from the storage level of the old version to query the corresponding original version information, and cannot query the version that is changing; after the version update is completed, the server updates the current version to the new version, so that other threads can access it through the new version.

[0111] Through this embodiment, the server updates the version by copying the storage layer, and when the update is not completed, other threads access the storage layers from the old version, ensuring the concurrent processing of reading, writing, merging and refreshing tasks in a lock-free state, thereby improving data processing efficiency.

[0112] In an exemplary embodiment, Figure 10 As shown, Figure 10 FIG. 1 is a flow chart of a data processing method in another embodiment. The method includes the following steps:

[0113] In step S302, the memory resources of the Memtable (mutable memory table) are all requested and released by the reading and writing threads. After the Immutable MemTable is solidified and flushed, or after the Memtable is converted, only the memory is marked as reclaimable, and a pulse clock lasteT is obtained at the same time. LasteT represents the time when the above-mentioned first reclaim mark is added.

[0114] Step S304: The user actively checks and calls the resource release interface in the reading and writing threads to obtain the current pulse clock curT. If there is a recyclable flag and curT-lastT>=2T, the memory is released.

[0115] In step S306, after the resource thread converts the Memtable into an Immutable memtable and flushes it to the target sort string table, the Memtable and the Immutable memtable can be marked. When the server recognizes that the resource thread is recyclable and meets the pulse clock condition, the server can recycle the corresponding resources through the resource thread.

[0116] Step S308, detecting the existence of a first recycling mark, further determining whether the existence duration of the first recycling mark meets the time threshold, such as whether it is greater than or equal to two pulse clock cycles. If not, it will not be recycled; if so, the server can recycle the above Memtable through the above resource thread.

[0117] Step S310, write the data in the memory into Memtable, which is a mutable memory table; when the Memtable reaches a certain size, convert it into Immutable memtable as the data to be written; after the Immutable memtable is flushed, the corresponding SSTable can be generated.

[0118] Step S312: When the storage level is updated, the server copies the changed storage level and makes modifications thereon instead of modifying the original storage level.

[0119] Step S314: Based on the replicated storage level, each associated SSTable is merged down to each storage level one by one to form each storage level of the new version.

[0120] Step S316: After the version change is completed, the new version replaces the old version, and the old version is added to the list of versions to be recycled. The versions to be recycled are recycled using a pulse clock method.

[0121] Step S318: Before the version change is completed, the read requests of other threads will enter the sorted string tables of each level from the storage level of the old version and query the corresponding original version information, but cannot query the version that is changing; after the version update is completed, the server updates the current version to the new version, so that other threads can access it through the new version.

[0122] Through the above embodiment, the server generates each replica storage level of each storage level in the hard disk, and generates and sinks the sorting string of the data in the replica storage level corresponding to the storage level, so that when multiple threads process the data concurrently, the consistency of the data can be guaranteed without locking, and a management mechanism for concurrent reading, writing and flushing operations without locking the memory table is implemented, thereby improving the data processing efficiency.

[0123] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0124] Based on the same inventive concept, the present application also provides a data processing device for implementing the aforementioned data processing method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in one or more data processing device embodiments provided below can be found in the above-mentioned limitations on the data processing method and will not be repeated here.

[0125] In an exemplary embodiment, Figure 11 As shown, a data processing device is provided, including: an acquisition module 500, a storage module 502, a merging module 504 and a writing module 506, wherein:

[0126] The acquisition module 500 is used to acquire the data to be written, and acquire each replication storage layer corresponding to each storage layer in the hard disk according to the data.

[0127] The storage module 502 is configured to store the data into a target replication storage layer in each of the replication storage layers to obtain a target sorted string table.

[0128] The merging module 504 is configured to merge the target sorted string table with the sorted string tables in the lower replication storage layers to obtain merged sorted string tables.

[0129] The writing module 506 is configured to obtain a writing result of the data according to the merged sorted string tables in the replicated storage levels and the unchanged sorted string tables in the replicated storage levels.

[0130] In one embodiment, the storage module 502 is configured to store the data into the first replication storage layer, and obtain a target sorting string table corresponding to the data in the first replication storage layer.

[0131] In one embodiment, the above-mentioned merging module 504 is used to obtain the various sorting string tables associated with the above-mentioned target sorting string table in the above-mentioned replication storage layer below the above-mentioned target replication storage layer; merge the above-mentioned target sorting string table with the various associated sorting string tables to obtain a new target replication storage layer containing the merged target sorting string table; return to execute the above-mentioned step of obtaining the various sorting string tables associated with the above-mentioned target sorting string table in the above-mentioned replication storage layer below the above-mentioned target replication storage layer, until the above-mentioned target replication storage layer is the lowest target replication storage layer or there is no sorting string table associated with the above-mentioned target sorting string table, and obtain the merged various sorting string tables in each replication storage layer.

[0132] In one embodiment, the above-mentioned device also includes: a first recycling module, which is used to add a first recycling identifier in one or more of the above-mentioned variable memory table and immutable memory table by using the target thread of the above-mentioned data when detecting that the above-mentioned data changes to the target state; when it is detected that the above-mentioned first recycling identifier exists in one or more of the above-mentioned variable memory table and immutable memory table and the existence time of the above-mentioned first recycling identifier meets the time threshold, reclaim resources of one or more of the above-mentioned variable memory table and immutable memory table through the above-mentioned target thread.

[0133] In one embodiment, the first recycling module is used to determine that the data changes to the target state when it detects that the data is converted from a mutable memory table to an immutable memory table and / or the immutable memory table is stored in a target replication storage layer in each of the replication storage layers.

[0134] In one embodiment, the above-mentioned device also includes: a second recycling module, which is used to add a second recycling identifier in each of the above-mentioned storage layers by using the target thread of the above-mentioned data; when it is detected that the above-mentioned second recycling identifier exists in the above-mentioned storage layer and the existence time of the above-mentioned second recycling identifier meets the time threshold, the resources of each of the above-mentioned storage layers are recycled through the above-mentioned target thread.

[0135] In one embodiment, the apparatus further comprises: a query module configured to, upon receiving a query request for the sorted string table, query each of the sorted string tables in each of the storage levels according to the query request to obtain target data corresponding to the query request.

[0136] Each module in the above-mentioned data processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0137] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 12 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data in the memory. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0138] Those skilled in the art will understand that Figure 12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0139] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned data processing method when executing the computer program.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data processing method is implemented.

[0141] In one embodiment, a computer program product is provided, comprising a computer program, which implements the above-mentioned data processing method when executed by a processor.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0143] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable logic unit (PLC), a data processing logic unit based on quantum computing, an artificial intelligence (AI) processor, and the like.

[0144] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0145] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire data to be written, and acquire, based on the data, each replicated storage layer corresponding to each storage layer in the hard disk; Storing the data in a target replication storage layer in each replication storage layer to obtain a target sorted string table; Merging the target sorted string table with the sorted string tables in the respective duplicate storage levels of the lower layers to obtain merged sorted string tables; The data writing result is obtained according to the merged sorted string tables in the respective duplicate storage levels and the unchanged sorted string tables in the respective duplicate storage levels.

2. The method according to claim 1, characterized in that The target replication storage tier includes a first replication storage tier; The first replication storage tier corresponds to a first storage tier among the storage tiers; The first storage tier represents a top storage tier among the storage tiers; The step of storing the data in a target replication storage layer in each replication storage layer to obtain a target sorted string table includes: The data is stored in the first replication storage level, and a target sorting string table corresponding to the data in the first replication storage level is obtained.

3. The method according to claim 1, characterized in that The step of merging the target sorted string table with the sorted string tables in the respective replication storage levels of the lower layers to obtain the merged sorted string tables includes: obtaining, in the replicated storage tier below the target replicated storage tier, respective sorted string tables associated with the target sorted string table; Merging the target sorted string table with each of the associated sorted string tables to obtain a new target replication storage level including the merged target sorted string table; Return to executing the step of obtaining the respective sorting string tables associated with the target sorting string table in the replicated storage hierarchy below the target replicated storage hierarchy, until the target replicated storage hierarchy is the lowest target replicated storage hierarchy or there is no sorting string table associated with the target sorting string table, and obtain the merged sorting string tables in the respective replicated storage hierarchies.

4. The method according to claim 1, wherein The method further comprises: When detecting that the data changes to a target state, adding a first recycling marker to one or more of the mutable memory table and the immutable memory table by using a target thread of the data; When it is detected that the first recycling identifier exists in one or more of the variable memory table and the immutable memory table and the existence time of the first recycling identifier meets the time threshold, resources of one or more of the variable memory table and the immutable memory table are recycled through the target thread.

5. The method according to claim 4, characterized in that The data includes one or more of a mutable memory table and an immutable memory table; The detecting that the data changes to a target state includes: When it is detected that the data is converted from the mutable memory table to the immutable memory table and / or the immutable memory table is stored in the target replication storage tier in each of the replication storage tiers, it is determined that the data is changed to the target state.

6. The method according to claim 1, characterized in that After obtaining the data writing result according to the merged sorted string tables in the respective replicated storage levels and the unchanged sorted string tables in the respective replicated storage levels, the method further includes: adding a second recycling marker in each of the storage tiers by using a target thread of the data; When it is detected that the second recycling mark exists in the storage layer and the existence time of the second recycling mark meets the time threshold, resources of each storage layer are recycled through the target thread.

7. The method according to any one of claims 1 to 6, characterized in that Before obtaining the data writing result according to the merged sorted string tables in the replicated storage levels and the unchanged sorted string tables in the replicated storage levels, the method further includes: When a query request for the sorted string table is received, each sorted string table in each storage level is queried according to the query request to obtain target data corresponding to the query request.

8. A data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire data to be written, and acquire, based on the data, each replicated storage layer corresponding to each storage layer in the hard disk; a storage module, configured to store the data into a target replication storage layer in each of the replication storage layers to obtain a target sorted string table; a merging module, configured to merge the target sorted string table with the sorted string tables in the respective replication storage levels of the lower layers to obtain merged sorted string tables; The writing module is configured to obtain a writing result of the data according to the merged sorted string tables in the replicated storage levels and the unchanged sorted string tables in the replicated storage levels.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images