Data processing method and device, computer device, readable storage medium and program product

Abstract

The application relates to a data processing method and device, computer equipment, a readable storage medium and a program product. Data is stored in a target replication storage level in a replication storage hierarchy, a target sorting string table in the target replication storage level is combined with a sorting string table in a replication storage level in a lower layer, and a write result of the data is obtained according to each combined sorting string table in each replication storage level and each unaltered sorting string table. Compared with traditional data sinking writing through a lock mode and multiple versions, each replication storage level of each storage level in a hard disk is generated in the present scheme, the data is subjected to sorting string generation and sinking in the corresponding replication storage level of the storage level, the data consistency can be ensured without locking when multiple threads concurrently process the data, and the data processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Concurrency processing is currently an effective way to improve computer processing efficiency and performance. For concurrent control of memory data, concurrent read / write, flush, and merge operations need to be handled. Currently, memory data processing typically uses locking to ensure only one processing operation occurs at a time, or uses multi-versioning (MPV) combined with version control strategies to query data across multiple versions. However, processing memory data using locking or MPV methods can lead to reduced processing efficiency due to lock contention and numerous versions.

[0003] Therefore, current data processing methods suffer from low processing efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve processing efficiency in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a data processing method, including:

[0006] Obtain the data to be written, and obtain the corresponding copy storage levels for each storage level in the hard disk based on the data;

[0007] The data is stored in the target replication storage level of each replication storage level to obtain the target sorted string table;

[0008] The target sorted string table is merged with the sorted string tables in each of the lower-level copy storage levels to obtain the merged sorted string tables.

[0009] The data writing result is obtained based on the merged sorted string tables in each of the replicated storage levels and the unchanged sorted string tables in each of the replicated storage levels.

[0010] In this embodiment, the server generates and sinks the data into the corresponding replica storage level of each storage level in the hard disk by generating each replica storage level of each storage level. This allows the data to be consistent without locking when multiple threads process the data concurrently, thus improving the data processing efficiency.

[0011] In one embodiment, the target replication storage level includes a first replication storage level; the first replication storage level corresponds to a first storage level in each of the storage levels; the first storage level represents the top storage level in each of the storage levels;

[0012] The step of storing the data into the target replication storage level in each of the replication storage levels to obtain the target sorted string table includes:

[0013] The data is stored in the first replicated storage level to obtain the target sorted string table corresponding to the data in the first replicated storage level.

[0014] In this embodiment, the server flushes the data and stores it in the copy storage level corresponding to the top-level storage level to form the target sorted string table, instead of modifying the original storage level. Other threads can still process the storage level, achieving lock-free concurrent processing. This allows the LSM structure to make full use of processor resources, and the use of lock-free concurrency control avoids blocking caused by synchronization, significantly improving data processing efficiency.

[0015] In one embodiment, merging the target sorted string table with the sorted string tables in each of the lower-level replicated storage levels to obtain merged sorted string tables includes:

[0016] In the copy storage layer below the target copy storage layer, obtain each sort string table associated with the target sort string table;

[0017] The target sorted string table is merged with each of the associated sorted string tables to obtain a new target copy storage level containing the merged target sorted string table.

[0018] Return to the step of obtaining each sorted string table associated with the target sorted string table in the copy storage level below the target copy storage level, until the target copy storage level is the lowest target copy storage level or there is no sorted string table associated with the target sorted string table, and obtain the merged sorted string tables in each copy storage level.

[0019] In this embodiment, the server can merge related sorted string tables by replicating the storage hierarchy, thereby updating the storage hierarchy without using a locking mechanism. It can also process each storage hierarchy concurrently, improving the efficiency of the sinking and merging process during the data processing. This method can be well applied to various LSM structures.

[0020] In one embodiment, the method further includes:

[0021] When the data is detected to have changed to the target state, a first recycling flag is added to one or more of the mutable and immutable memory tables by using the target thread of the data.

[0022] When the presence of the first reclamation flag is detected in one or more of the variable memory table and the immutable memory table, and the duration of the presence of the first reclamation flag meets the duration threshold, the target thread reclaims resources from one or more of the variable memory table and the immutable memory table.

[0023] In this embodiment, the server utilizes a pulse clock mechanism to achieve concurrent processing and reclamation of data in memory without locking, thereby improving the efficiency of data processing in memory and enabling resources to be reclaimed as promptly as possible, thus minimizing unnecessary resource waste.

[0024] In one embodiment, the data includes one or more of a variable memory table and an immutable memory table;

[0025] The detection of the data change to the target state includes:

[0026] When it is detected that the data has been transformed from a mutable memory table to an immutable memory table and / or the immutable memory table is stored in the target replicated storage level of each of the replicated storage levels, it is determined that the data has changed to the target state.

[0027] In this embodiment, the server detects changes in the state of data from a mutable memory table to an immutable memory table, and from an immutable memory table to the target replication storage level. This allows the server to detect whether the data meets the recycling conditions, improving the efficiency of data processing in memory and enabling resources to be recycled as promptly as possible, thus minimizing unnecessary resource waste.

[0028] In one embodiment, after obtaining the data writing result based on the merged sorted string tables in each of the replicated storage levels and the unchanged sorted string tables in each of the replicated storage levels, the method further includes:

[0029] By using the target thread of the data, a second recycling flag is added to each of the storage levels;

[0030] When the existence of the second reclamation identifier in the storage level is detected and the duration of the existence of the second reclamation identifier meets the duration threshold, the resources of each storage level are reclaimed through the target thread.

[0031] In this embodiment, the server generates new versions of each storage level based on the replicated storage level. Other threads can still perform read operations during the sinking and merging process of the storage levels, which meets the concurrent processing needs of each thread for the storage levels and improves data processing efficiency. Furthermore, the old storage levels are reclaimed using a pulse clock mechanism, which reduces resource waste.

[0032] In one embodiment, before obtaining the data writing result based on the merged sorted string tables in each of the replicated storage levels and the unchanged sorted string tables in each of the replicated storage levels, the method further includes:

[0033] When a query request for the sorted string table is received, the query is performed in each of the sorted string tables in each of the storage levels according to the query request to obtain the target data corresponding to the query request.

[0034] In this embodiment, the server updates the version by replicating the storage hierarchy. While the update is not complete, other threads access each storage hierarchy from the old version, ensuring concurrent processing of read, write, merge, and refresh tasks without locks, thus improving data processing efficiency.

[0035] Secondly, this application also provides a data processing apparatus, comprising:

[0036] The acquisition module is used to acquire the data to be written and, based on the data, acquire the corresponding copy storage levels for each storage level in the hard disk.

[0037] The storage module is used to store the data into the target replication storage level in each of the replication storage levels to obtain the target sorted string table;

[0038] The merging module is used to merge the target sorted string table with the sorted string tables in each of the lower-level copy storage levels to obtain the merged sorted string tables.

[0039] The writing module is used to obtain the writing result of the data based on the merged sorted string tables in each of the replicated storage levels and the unmodified sorted string tables in each of the replicated storage levels.

[0040] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0041] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0042] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0043] The aforementioned data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product obtain a target sorted string table by storing data in the target copied storage level corresponding to the storage level on the hard disk. This target sorted string table is then merged with the sorted string tables in each of the lower-level copied storage levels to obtain merged sorted string tables. Based on the merged sorted string tables in each copied storage level and the unchanged sorted string tables, the data writing result is obtained. Compared to traditional data sinking writes using locking modes and multi-versioning, this solution generates and sinks sorted strings within each copied storage level of the hard disk, ensuring data consistency even when multiple threads process data concurrently without the need for locking, thus improving data processing efficiency. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating a data processing method in one embodiment;

[0046] Figure 2 This is a schematic diagram of the merging steps in one embodiment;

[0047] Figure 3 This is a schematic diagram of the merging steps in another embodiment;

[0048] Figure 4 This is a schematic diagram of the merging steps in yet another embodiment;

[0049] Figure 5 This is a schematic diagram of the merging steps in another embodiment;

[0050] Figure 6 This is a timing diagram of the pulse management steps in one embodiment;

[0051] Figure 7 This is a flowchart illustrating the recycling steps in one embodiment;

[0052] Figure 8 This is a flowchart illustrating the recycling steps in another embodiment;

[0053] Figure 9 This is a flowchart illustrating the version update steps in one embodiment;

[0054] Figure 10 This is a flowchart illustrating the data processing method in another embodiment;

[0055] Figure 11 This is a structural block diagram of a data processing device in one embodiment;

[0056] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] In related technologies, the LSM (Log-Structured-Merge-Tree) is a data structure designed to optimize write performance. It converts random writes into sequential writes by first writing data to memory and then periodically persisting it to disk, reducing disk I / O operations and thus significantly improving write speed, but slightly reducing read performance. It is commonly used to solve the problem of random read / write efficiency on hard drives.

[0059] To improve computer processing efficiency and performance, in-memory data storage and processing can currently be performed concurrently. For concurrency control, LSM (Limited-Side Memory Management) needs to handle concurrent reads and writes, as well as concurrent refresh and merge operations. Currently, these processes are handled using either locking or multi-versioning mechanisms, depending on the specific requirements.

[0060] In lock mode, the method of persisting the memory table to disk is protected by locks, ensuring that only one persistence operation occurs at a time. The merging of existing SSTables (Sorted String Tables) is also protected by locks to ensure that the merge operation is serialized. This ensures data ordering and consistency. However, significant lock contention occurs during execution, causing synchronization blocking and reducing concurrency performance.

[0061] For multi-version concurrency (MPC) mode, LSM trees support the simultaneous existence of multiple versions of data in concurrent environments. LSM trees achieve high-concurrency read and write operations by maintaining multiple Memtables (mutable in-memory tables) and SSTables, and by utilizing WAL (Write-Ahead Logging) and version control. However, as the number of versions increases, read operations may need to check multiple SSTables, leading to a decrease in query performance. Furthermore, Memtable flushing still requires locking mechanisms for protection.

[0062] In view of this, in order to optimize LSM concurrency control, this solution combines multiple versions, multiple queues and pulse clocks to build an LSM tree structure operation mode that is as lock-free as possible. Its main contents are divided into two parts: lock-free memory table and lock-free version.

[0063] In one embodiment, such as Figure 1 As shown, a data processing method is provided. This embodiment illustrates the application of this method to a server. It is understood that this method can also be applied to a terminal, which can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart vehicle devices, etc. It can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server, including the following steps S202 to S208. Wherein:

[0064] Step S202: Obtain the data to be written, and obtain the corresponding copy storage levels in the hard disk based on the above data.

[0065] The data to be written can be data in memory that needs to be written to disk, and this process can be based on an LSM tree. The data in memory can first be written to a memory table. This memory table can be a lock-free type. For lock-free memory tables, the server can define that only one thread updates the memory data, while other threads are read-only.

[0066] The aforementioned memory tables include Memtable and Immutable memtable. A Memtable is an in-memory data structure, typically an ordered red-black tree or a similar balanced search tree, often used as a write buffer. When data items are written to the database, they are first written to the Memtable. Because the Memtable resides in memory, it provides very fast write performance. When the Memtable reaches a certain size, it is converted into an Immutable memtable. An Immutable memtable is an intermediate state between a Memtable and an SSTable. It is mainly used to prevent data update operations from being blocked during the conversion process (new written data is processed by a new Memtable). Like the Memtable, an Immutable memtable is an in-memory data structure of an LSM tree, but it is immutable. There can be multiple Immutable memtables, and the data between them is unordered and may contain duplicates. They are sorted by time and updated as new data is generated.

[0067] During writing, data first enters the current Memtable and is synchronously written to the Write-Ahead Log (WAL) to ensure no data loss. When the Memtable is full, the server converts the Memtable into an Immutable Memtable and creates a new Memtable. A background thread writes the Immutable Memtable to disk to form an SSTable.

[0068] The server's hard drive can include multiple storage levels, each storing one or more sorted string tables. During data writing and updating, the server can use a Copy-On-Write (COW) approach. The server updates the storage hierarchy structure by progressively merging the sorted string tables across each level. When writing to the hard drive, for the data to be written, the server can retrieve the corresponding replica storage levels based on the data. In other words, the server can copy the storage levels on the hard drive to obtain the corresponding replica storage levels, also known as shadow levels. The data can be stored by progressively merging across different storage levels, meaning the replica storage levels are only copied when an update to the sorted string table is needed. It should be noted that each replica storage level corresponds one-to-one with the existing storage levels.

[0069] Step S204: Store the above data into the target replication storage level in each of the above replication storage levels to obtain the target sorted string table.

[0070] When writing the aforementioned data to the hard drive, it can first be stored in the corresponding storage level. The server then stores the data in a replica storage level corresponding to that storage level, and updates the storage level after storage, thus completing the data writing process. Since there are multiple storage levels, there are also multiple replica storage levels. The server can store the data in the target replica storage level within each replica storage level, obtaining a target sorted string table. The sorted string table can be a persistent, immutable, and ordered data file.

[0071] The target replication storage level can be the replication storage level corresponding to the first storage level where data is stored when it is written to the hard disk, and the target sorting string table can be the sorting string table obtained from the data in the target replication storage level. The first storage level can be determined based on the distribution of storage levels on the hard disk. For example, the server can choose the top-level storage level as the first storage level where the data is stored.

[0072] Step S206: Merge the target sorted string table with the sorted string tables in the lower-level copy storage levels to obtain merged sorted string tables.

[0073] The aforementioned storage hierarchy has multiple levels. After the server stores the data in the target replication storage level, the target sorted string table in the target replication storage level can be merged downwards. The lower replication storage level corresponding to the target replication storage level can store one or more sorted string tables. The server can compare the target sorted string table with the data in each of the lower replication storage levels and merge the target sorted string table with the sorted string tables in each of the lower replication storage levels to obtain merged sorted string tables. This merging can involve merging the sorted string tables from the upper replication storage level to the lower replication storage level.

[0074] Step S208: Based on the merged sorted string tables in each of the aforementioned replicated storage levels and the unchanged sorted string tables in each of the aforementioned replicated storage levels, the write result of the aforementioned data is obtained.

[0075] The sorted string tables involved in the merging process can be related sorted string tables. That is, sorted string tables related to the aforementioned data need to be merged, so that the server obtains the merged sorted string tables in each replication storage level. Other unrelated sorted string tables do not need to be merged and remain in their respective replication storage levels. Therefore, the server can also obtain the unchanged sorted string tables in each replication storage level.

[0076] Therefore, the server can obtain the data writing results based on the merged sorted string tables in each of the aforementioned replicated storage levels, and the unchanged sorted string tables in each of the aforementioned replicated storage levels. The writing results include both data writing and storage level updates. For example, the server can generate new versions of each of the aforementioned storage levels based on each replicated storage level containing both the merged and unchanged sorted string tables, thereby enabling the storage of the aforementioned data and the updating of the aforementioned storage levels.

[0077] In the above data processing method, a target sorted string table is obtained by storing data in the target copy storage level corresponding to the storage level on the hard disk. This target sorted string table is then merged with the sorted string tables in each of the lower-level copy storage levels to obtain merged sorted string tables. Based on the merged sorted string tables in each copy storage level and the unchanged sorted string tables, the data writing result is obtained. Compared to traditional data sinking writes using locking modes and multi-versioning, this solution generates and sinks sorted strings within each copy storage level of the hard disk, ensuring data consistency even when multiple threads process data concurrently without the need for locking, thus improving data processing efficiency.

[0078] In one embodiment, storing the data into the target replication storage level of each of the aforementioned replication storage levels to obtain a target sorted string table includes: storing the data into the first replication storage level to obtain a target sorted string table corresponding to the data in the first replication storage level.

[0079] In this embodiment, the target replication storage level includes a first replication storage level. This first replication storage level corresponds to the first storage level among all the aforementioned storage levels, and represents the top-level storage level among all the aforementioned storage levels. That is, the server can first store the data in the replication storage level corresponding to the top-level storage level among all the aforementioned storage levels to obtain the target sorted string.

[0080] Specifically, the server stores the aforementioned data in the first replication storage level, which can be a data flushing process. The server first writes the data in memory to a Memtable, an immutable in-memory table; when the Memtable reaches a certain size, it converts it into an Immutable memtable, which serves as the data to be written. The data write-and-merge process can be as follows: Figure 2 As shown, Figure 2 This is a flowchart illustrating the merging steps in one embodiment. `levelN` represents the Nth storage level. Figure 2 As can be seen, the server comprises multiple storage tiers. After the aforementioned Immutable memtable is flushed, a corresponding SSTable (sorted string) can be generated. The server uses LSM for version control, managing the SSTables at each storage tier on the hard drive. LSM version updates can be metadata updates, involving hierarchical changes to the SSTables, conforming to the characteristics of read-heavy and write-light architecture. Furthermore, the server uses a Copy-on-Write (COW) mode in memory to support concurrent read and write operations. Figure 2 In the process, updates to the LSM structure, such as the generation of the SSTbale for level 10 by flushing the Immutable memtable down, and the sinking and merging of level 0 into level 1, are all handled through COW mode.

[0081] In this process, data in memory is written to the current Memtable via Write operations and simultaneously written to the Write Allocation (WAL) to ensure no data loss. When the Memtable is full, the server converts it to an Immutable Memtable and creates a new Memtable. A background thread writes the Immutable Memtable to disk, forming an SSTable. The creation and sinking of SSTables leads to updates to the corresponding storage levels. When a storage level is updated, the server copies the changed storage level, called a shadow storage level or replicated storage level. The server modifies the storage level on this copy instead of modifying the original storage level. This achieves lock-free concurrent processing of data in the storage level. Lock-free processing refers to the use of techniques in multithreaded and concurrent programming to avoid or reduce locking operations on shared resources, thereby improving program concurrency and performance.

[0082] In this embodiment, the server can flush the data and store it in the copy storage level corresponding to the top-level storage level to form the target sorted string table, instead of modifying the original storage level. Other threads can still process the storage level, achieving lock-free concurrent processing. This allows the LSM structure to make full use of processor resources, and the use of lock-free concurrency control avoids blocking caused by synchronization, significantly improving data processing efficiency.

[0083] In one embodiment, merging the target sorted string table with the sorted string tables in the lower-level replication storage layers to obtain merged sorted string tables includes: obtaining the sorted string tables associated with the target sorted string table in the replication storage layers below the target replication storage layer; merging the target sorted string table with the associated sorted string tables to obtain a new target replication storage layer containing the merged target sorted string table; returning to the step of obtaining the sorted string tables associated with the target sorted string table in the replication storage layers below the target replication storage layer, until the target replication storage layer is the lowest-level target replication storage layer or there is no sorted string table associated with the target sorted string table, thus obtaining the merged sorted string tables in each replication storage layer.

[0084] In this embodiment, the aforementioned storage hierarchy may include multiple layers, and each storage hierarchy can generate a replica storage hierarchy, allowing the server to process data within the replica storage hierarchy. After the data is stored in the target replica storage hierarchy to form a target sorted string table, data can be merged downwards, that is, the target sorted string table is matched with the sorted string tables in each of the lower storage hierarchies to determine whether merging is necessary.

[0085] For example, taking the target replication storage level as the replication storage level corresponding to the aforementioned top-level storage level as the starting point, the server can retrieve the various sorted string tables associated with the aforementioned target sorted string table from the replication storage levels below the target replication storage level, and merge the target string table with the associated sorted string tables to obtain a new target replication storage level containing the merged target string table. This new target replication storage level can be a new replication storage level obtained by updating the aforementioned lower-level replication storage level; the new target replication storage level can include the merged sorted string table, and can also include the unchanged sorted string table from the aforementioned lower-level replication storage level.

[0086] If the server detects that the new target replication storage level is not the lowest storage level among all storage levels, the server can start with the new target replication storage level and return to the steps described above, in the replication storage levels below the target replication storage level, to obtain the sorted string tables associated with the target sorted string table, and perform the next round of sinking and merging until the target replication storage level is the lowest target replication storage level, or there is no sorted string table associated with the merged target sorted string table in the lower replication storage level. At this point, the server can obtain the merged sorted string tables in each replication storage level. Therefore, each replication storage level after sinking and merging contains the unmodified sorted string tables and the target sorted string table obtained after sinking and merging. Thus, each replication storage level after sinking and merging can form a new version of the storage level, replacing the previous storage levels.

[0087] Specifically, taking a storage hierarchy of N levels as an example, where N is a natural number. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of the merging steps in another embodiment. For the sinking merge between the top-level 0 and the second-level 1, storage level 0 contains the target sorted string table corresponding to the aforementioned data, as well as the original sorted strings in level 0. Level 1 contains sorted string tables associated with both the target sorted string table and the original sorted string table in level 0, such as... Figure 3 The server generates a replica storage layer for both level 0 and level 1, merging the various replica storage layers in level 0 with the first replica storage layer in level 1, while maintaining the two unrelated replica storage layers in level 1 unchanged, thus obtaining the merged level 1 layer.

[0088] like Figure 4 As shown, Figure 4This is a schematic diagram of the merging steps in another embodiment. For the sinking merge between the second level (level1) and the third level (level2), assuming the first target string in level1 is associated with the first and second sorted string tables in level2, the server can generate corresponding replication storage levels for level1 and level2 respectively. The first target sorted string table in level1 is merged into the first and second sorted string tables in level2, while maintaining the two unrelated sorted string tables in level1 unchanged, thus obtaining the merged level2. The merged level2 includes the merged target sorted string tables and the unmodified sorted string tables.

[0089] like Figure 5 As shown, Figure 5 This is a schematic diagram of the merging step in another embodiment. If the server detects that there is no association between the sorted string tables in level 2 and level N, the server can stop the sinking and merging of the sorted string tables in level 2. Furthermore, the sorted string tables in each replicated storage level may not necessarily be updated; only updates where data modifications actually occur are possible. Thus, the server uses the merged replicated storage level combined with other unchanged storage levels to generate a new LSM structure. Merging or refreshing tasks can be performed concurrently on different storage levels without conflict. The new LSM structure represents a new version; that is, the server can use the merged replicated sorted string tables to form updated storage levels.

[0090] Through this embodiment, the server can sink and merge various related sorted string tables based on the replicated storage hierarchy, thereby completing the update of the storage hierarchy without using a locking mechanism. It can also perform concurrent processing on various storage hierarchies, improving the processing efficiency of sinking and merging during the processing of the above data. It can be well applied to various LSM structures.

[0091] In one embodiment, the method further includes: when the data changes to the target state, adding a first recycling flag in one or more of the variable memory table and the immutable memory table using the target thread of the data; when the existence of the first recycling flag in one or more of the variable memory table and the existence duration of the first recycling flag meets the duration threshold, reclaiming resources in one or more of the variable memory table and the immutable memory table through the target thread.

[0092] In this embodiment, the target state refers to a state where data can be reclaimed. In some embodiments, the data can be one or more of a variable memory table and an immutable memory table.

[0093] In one embodiment, detecting that the data has changed to the target state includes: determining that the data has changed to the target state when the data is detected to have been converted from a variable memory table to an immutable memory table and / or the immutable memory table is stored in the target replication storage level of each of the replication storage levels.

[0094] In this embodiment, the aforementioned data can be either a variable memory table or an immutable memory table. Data can first be written to a variable memory table in memory, and when the variable memory table reaches a certain size, it is converted to an immutable memory table. The immutable memory table can then be flushed down to the target replication storage level, forming a sorted string table. When a variable memory table is converted to an immutable memory table, or when an immutable memory table is flushed down to the target replication storage level, the corresponding resources need to be reclaimed. For example, after a variable memory table is converted to an immutable memory table, the variable memory table is reclaimed; after an immutable memory table is flushed down to the target replication storage level, the immutable memory table is reclaimed. Thus, through this embodiment, the server detects changes in the state of data from a variable memory table to an immutable memory table, and from an immutable memory table to storage in the target replication storage level, thereby detecting whether the data meets the reclamation conditions. This improves the efficiency of data processing in memory and achieves timely resource reclamation, minimizing unnecessary resource waste.

[0095] The server can reclaim resources from mutable and immutable memory tables using a pulse clock. For example, the server can add a first reclamation flag to one or more of the mutable and immutable memory tables using a target thread that performs read and write operations on the data. When the server detects the presence of the first reclamation flag in one or more of the mutable and immutable memory tables, and the duration of the first reclamation flag meets a duration threshold, the server can reclaim resources from one or more of the mutable and immutable memory tables using the target thread. In other words, the server uses a pulse clock to detect the duration of the first reclamation flag and thus determines whether reclamation is necessary. The duration threshold can be set according to actual conditions.

[0096] Specifically, during the read process, which involves searching the Immutable memtable, the read, write, and flush processes can reside in different threads. Therefore, the immutable and mutable memory tables can satisfy the following rules: The memory resources of the Memtable are allocated and released by the threads performing the read and write operations; after the Immutable memtable is permanently flushed or converted, only the memory is marked as reclaimable, for example, by adding a first reclamation flag, and a pulse clock `lasteT` is obtained, where `lasteT` represents the time the first reclamation flag was added; the threads performing the read and write operations are actively checked periodically by the user, calling the resource release interface to obtain the current pulse clock `curT`. Taking a time threshold of 2T as an example, if the server detects a reclaimable flag, such as the first reclamation flag, and `curT – lastT >= 2T`, the memory is released. Here, 2T represents two pulse cycles.

[0097] The working process of the pulse clock is as follows: Figure 6 As shown, Figure 6 This is a timing diagram of the pulse management steps in one embodiment. Each task in each thread, such as conversion tasks and data processing tasks like sinking, can be managed based on a pulse clock. Here, nT represents the nth pulse clock cycle, and the pulse mechanism consists of a pulse start point and a pulse end point. A pulse round refers to the pulse start point selecting a task set and the pulse end point confirming that all tasks are completed. Only after one pulse round is completed can the next pulse round be initiated. The pulse clock represents adding a certain time interval between pulses to drive the quasi-periodic occurrence of pulses, with the clock incrementing at the end of each pulse round.

[0098] Specifically regarding the processing of Memtable, such as Figure 7 As shown, Figure 7 This is a flowchart illustrating the recycling steps in one embodiment. The resource thread can be a thread using a Memtable, capable of performing resource initialization, usage, and recycling tasks; the flushing thread can be a thread that flushes the Immutable memtable to the target sorted string table and can mark the Memtable as recyclable; other threads can be other read-only threads. After the resource thread converts the Memtable to an Immutable memtable and flushes it to the target sorted string table, it can mark the Memtable and Immutable memtable, for example, by adding a first recycling flag, thus marking the aforementioned "in use" resources as recyclable. When the server identifies recyclable resources through the resource thread and the pulse clock condition is met, the server can recycle the corresponding resources through the resource thread.

[0099] Among them, such as Figure 8 As shown, Figure 8 This is a flowchart illustrating the reclamation step in another embodiment. The target Memtable can be a variable memory table whose reclamation is uncertain. The server detects the existence of a first reclamation flag and further determines whether the duration of the first reclamation flag meets a time threshold, such as whether it is greater than or equal to two clock cycles. If not, it is not reclaimed; if so, the server can reclaim the Memtable through the aforementioned resource thread.

[0100] Therefore, by ensuring that writing and resource reclamation occur in the same thread, the server can guarantee that there is no concurrency in resource allocation or release at any given time, without requiring locks. Write operations include, but are not limited to, unique resource allocation and initialization triggering methods. Simultaneously, since each clock cycle represents the completion of the task set from the previous clock cycle, each data query is designed to avoid finding currently marked Memtables as reclaimable. This ensures that after two or more clock cycles, no tasks using the Memtable are running or newly generated, allowing the server's release procedure to safely reclaim the resources occupied by the Memtable without locks.

[0101] For threads other than those using Memtable, data queries always retrieve the latest table information without modifying related resources. This ensures safe, lock-free concurrent execution of all tasks. For read operations by threads other than those using Memtable, the data in the Immutable memtable can be retrieved accurately and without errors, guaranteeing data timeliness and consistency in a lock-free concurrent environment.

[0102] In this embodiment, the server utilizes a pulse clock mechanism to achieve concurrent processing and reclamation of data in memory without locking, thereby improving the efficiency of data processing in memory and enabling resources to be reclaimed as promptly as possible, thus minimizing unnecessary resource waste.

[0103] In one embodiment, after obtaining the data writing result based on the merged sorted string tables in each of the aforementioned replicated storage levels and the unmodified sorted string tables in each of the aforementioned replicated storage levels, the method further includes: adding a second reclamation flag in each of the aforementioned storage levels using the target thread of the aforementioned data; and reclamation of resources in each of the aforementioned storage levels by using the target thread when the existence of the second reclamation flag in the aforementioned storage level is detected and the existence duration of the second reclamation flag meets the duration threshold.

[0104] In this embodiment, for each storage level after sinking and merging, the server generates a new version of each storage level based on each copied storage level after merging the sorted string tables and each storage level that has not been modified, so that the server can reclaim the old storage levels.

[0105] The aforementioned resource reclamation can also be performed using a pulse clock mechanism. For example, the server can add a second reclamation flag in each of the aforementioned storage levels through a target thread that uses the aforementioned data. When the server detects the existence of the second reclamation flag in a storage level and the duration of the second reclamation flag meets a duration threshold, the target thread will reclaim resources from each of the aforementioned storage levels. The aforementioned duration threshold can be set according to actual conditions, for example, it can be two pulse clock cycles.

[0106] Specifically, such as Figure 9 As shown, Figure 9 This is a flowchart illustrating the version update steps in one embodiment. When a version is changed, the changes to the sorting string table in the storage hierarchy are applied to all replica storage levels of the new version. After the changes are complete, a new version is generated. Once the changes are finished, the old version is replaced with the new version, and the old version is added to the list of versions to be reclaimed. Versions to be reclaimed are reclaimed using a pulse clock method, thus constructing a lock-free management structure.

[0107] In this embodiment, the server generates new versions of each storage level based on the replicated storage level. Other threads can still perform read operations during the sinking and merging process of the storage levels, which meets the concurrent processing needs of each thread for the storage levels and improves data processing efficiency. Furthermore, the old storage levels are reclaimed using a pulse clock mechanism, which reduces resource waste.

[0108] In one embodiment, before obtaining the write result of the data based on the merged sorted string tables in each of the aforementioned replicated storage levels and the unmodified sorted string tables in each of the aforementioned replicated storage levels, the method further includes: when a query request for the sorted string tables is received, querying the sorted string tables in each of the aforementioned storage levels according to the query request to obtain the target data corresponding to the query request.

[0109] In this embodiment, the server replicates storage levels to merge and sink data in each sorted string table, generating new versions of each storage level. While the merging and new version generation processes are incomplete, other threads can query the sorted string tables within the storage levels. In this case, the queries retrieved by other threads may be the unupdated sorted string tables in each storage level. For example, when the server receives a query request for the aforementioned sorted string tables, it can query the respective sorted string tables in each of the aforementioned storage levels to obtain the target data corresponding to the query request. Here, each of the aforementioned storage levels refers to the unupdated storage levels, and the sorted string tables in each unupdated storage level are also unupdated sorted string tables.

[0110] Specifically, the version update and recycling tasks of each of the above storage levels can be completed by a single thread. Before the version change is completed, read requests from other threads will enter the sorted string table of each level from the old version storage level and find the corresponding original version information. They will not be able to find the version that is changing. After the version update is completed, the server updates the current version to the new version, so that other threads can access it through the new version.

[0111] In this embodiment, the server updates the version by replicating the storage hierarchy. While the update is not complete, other threads can access each storage hierarchy from the old version, ensuring concurrent processing of read, write, merge, and refresh tasks without locks, thus improving data processing efficiency.

[0112] In one exemplary embodiment, such as Figure 10 As shown, Figure 10 This is a flowchart illustrating a data processing method in another embodiment. It includes the following steps:

[0113] In step S302, the memory resources of the Memtable (mutable memory table) are all requested and released by the threads where reading and writing are performed; after the Immutable memtable is solidified and flushed, or after the Memtable is converted, only the memory is marked as reclaimable, and a pulse clock lasteT is obtained, where lasteT represents the time when the first reclamation mark was added.

[0114] In step S304, the user actively checks the resource release interface periodically for the thread that is reading and writing, obtains the current pulse clock curT, and releases the memory if there is a reclaimable flag and curT – lastT >= 2T.

[0115] In step S306, after the resource thread converts the Memtable into an Immutable memtable and flushes it to the target sorted string table, it can mark the Memtable and the Immutable memtable. When the server identifies recyclable resources that meet the pulse clock conditions through the resource thread, the server can reclaim the corresponding resources through the resource thread.

[0116] Step S308: A first recycling flag is detected. Further determine whether the duration of the first recycling flag meets a time threshold, such as whether it is greater than or equal to two clock pulses. If not, it will not be recycled; if so, the server can recycle the above Memtable through the above resource thread.

[0117] Step S310: Write the data in memory into the Memtable, which is a memtable; when the Memtable reaches a certain size, convert it into an Immutable memtable as the data to be written; after the Immutable memtable is flushed, the corresponding SSTable can be generated.

[0118] In step S312, when updating the storage level, the server copies the changed storage level and makes modifications on it instead of modifying the original storage level.

[0119] Step S314: Based on the replication storage hierarchy, the associated SSTables are merged down one storage hierarchy at a time to form the new version of each storage hierarchy.

[0120] Step S316: After the version change is completed, replace the old version with the new version, and add the old version to the list of versions to be recycled. Versions to be recycled are recycled using a pulse clock method.

[0121] In step S318, before the version change is completed, read requests from other threads will enter the sorted string tables of each level from the storage level of the old version and find the corresponding original version information. They will not be able to find the version that is changing. After the version update is completed, the server updates the current version to the new version, so that other threads can access it through the new version.

[0122] Through the above embodiments, the server generates each replicated storage level of each storage level in the hard disk, and generates and sinks the sorted string in the replicated storage level corresponding to the storage level. This allows data consistency to be guaranteed without locking when multiple threads process the data concurrently. It realizes a management mechanism to protect the concurrent read, write and flush operations of the memory table without locking, and improves the data processing efficiency.

[0123] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the steps or stages of other steps.

[0124] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0125] In one exemplary embodiment, such as Figure 11 As shown, a data processing apparatus is provided, including: an acquisition module 500, a storage module 502, a merging module 504, and a writing module 506, wherein:

[0126] The acquisition module 500 is used to acquire the data to be written and to acquire the corresponding copy storage levels in the hard disk based on the data.

[0127] The storage module 502 is used to store the above data into the target replication storage level in each of the above replication storage levels to obtain the target sorted string table.

[0128] The merging module 504 is used to merge the target sorted string table mentioned above with the sorted string tables in each of the lower-level replicated storage levels to obtain the merged sorted string tables.

[0129] The writing module 506 is used to obtain the writing result of the above data based on the merged sorted string tables in each of the above-mentioned replicated storage levels and the unmodified sorted string tables in each of the above-mentioned replicated storage levels.

[0130] In one embodiment, the storage module 502 is used to store the data into the first replication storage layer to obtain the target sorted string table corresponding to the data in the first replication storage layer.

[0131] In one embodiment, the merging module 504 is configured to: obtain each sorted string table associated with the target sorted string table in the copy storage layer below the target copy storage layer; merge the target sorted string table with each associated sorted string table to obtain a new target copy storage layer containing the merged target sorted string table; return to the step of obtaining each sorted string table associated with the target sorted string table in the copy storage layer below the target copy storage layer, until the target copy storage layer is the lowest target copy storage layer or there is no sorted string table associated with the target sorted string table, thus obtaining the merged sorted string tables in each copy storage layer.

[0132] In one embodiment, the apparatus further includes: a first recycling module, configured to, when detecting that the data has changed to a target state, add a first recycling identifier in one or more of the variable memory table and the immutable memory table by using the target thread of the data; and when detecting that one or more of the variable memory table and the immutable memory table have the first recycling identifier and the duration of the existence of the first recycling identifier meets a duration threshold, reclaim resources from one or more of the variable memory table and the immutable memory table through the target thread.

[0133] In one embodiment, the first recycling module is used to determine that the data has changed to the target state when it detects that the data has been transformed from a variable memory table to an immutable memory table and / or the immutable memory table is stored in the target replication storage level of each of the replication storage levels.

[0134] In one embodiment, the above apparatus further includes: a second recycling module, configured to add a second recycling identifier in each of the above storage levels by using a target thread that uses the above data; and when the existence of the second recycling identifier in the above storage level is detected and the existence duration of the second recycling identifier meets a duration threshold, to reclaim resources in each of the above storage levels through the target thread.

[0135] In one embodiment, the apparatus further includes a query module, configured to, upon receiving a query request for the sorted string table, perform a query in each of the sorted string tables in each of the storage levels according to the query request, and obtain the target data corresponding to the query request.

[0136] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0137] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data in memory. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0138] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0139] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the data processing method described above.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the data processing method described above.

[0141] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the data processing method described above.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0143] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0145] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the data to be written, and obtain the corresponding copy storage levels for each storage level in the hard disk based on the data; The data is stored in the target replication storage level of each replication storage level to obtain the target sorted string table; The target sorted string table is merged with the sorted string tables in each of the lower-level copy storage levels to obtain the merged sorted string tables. Based on the merged sorted string tables in each of the replicated storage levels and the unchanged sorted string tables in each of the replicated storage levels, a new version of each of the storage levels is generated. Before updating the current version to the new version, when a read request from another thread is received, the sorted string tables of each storage level of the current version are queried, and the query results are returned. The current version is then updated to the new version, and the current version before the update is added to the list of versions to be recycled, thus completing the writing of the data.

2. The method according to claim 1, characterized in that, The target replication storage hierarchy includes a first replication storage hierarchy; The first replication storage level corresponds to the first storage level in each of the storage levels; The first storage level represents the top storage level among all the storage levels; The step of storing the data into the target replication storage level in each of the replication storage levels to obtain the target sorted string table includes: The data is stored in the first replicated storage level to obtain the target sorted string table corresponding to the data in the first replicated storage level.

3. The method according to claim 1, characterized in that, The step of merging the target sorted string table with the sorted string tables in each of the lower-level replicated storage levels to obtain merged sorted string tables includes: In the copy storage layer below the target copy storage layer, obtain each sort string table associated with the target sort string table; The target sorted string table is merged with each of the associated sorted string tables to obtain a new target copy storage level containing the merged target sorted string table. Return to the step of obtaining each sorted string table associated with the target sorted string table in the copy storage level below the target copy storage level, until the target copy storage level is the lowest target copy storage level or there is no sorted string table associated with the target sorted string table, and obtain the merged sorted string tables in each copy storage level.

4. The method according to claim 1, characterized in that, The method further includes: When the data is detected to have changed to the target state, a first recycling flag is added in one or more of the mutable and immutable memory tables by using the target thread of the data. When the presence of the first reclamation flag is detected in one or more of the variable memory table and the immutable memory table, and the duration of the presence of the first reclamation flag meets the duration threshold, the target thread reclaims resources from one or more of the variable memory table and the immutable memory table.

5. The method according to claim 4, characterized in that, The data includes one or more of variable memory tables and immutable memory tables; The detection of the data change to the target state includes: When it is detected that the data has been transformed from a mutable memory table to an immutable memory table and / or the immutable memory table is stored in the target replicated storage level of each of the replicated storage levels, it is determined that the data has changed to the target state.

6. The method according to claim 1, characterized in that, After obtaining the data writing result based on the merged sorted string tables in each of the replicated storage levels and the unchanged sorted string tables in each of the replicated storage levels, the method further includes: By using the target thread of the data, a second recycling flag is added to each of the storage levels; When the existence of the second reclamation identifier in the storage level is detected and the duration of the existence of the second reclamation identifier meets the duration threshold, the resources of each storage level are reclaimed through the target thread.

7. The method according to any one of claims 1 to 6, characterized in that, Before obtaining the data writing result based on the merged sorted string tables in each of the replicated storage levels and the unchanged sorted string tables in each of the replicated storage levels, the method further includes: When a query request for the sorted string table is received, the query is performed in each of the sorted string tables in each of the storage levels according to the query request to obtain the target data corresponding to the query request.

8. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the data to be written and, based on the data, acquire the corresponding copy storage levels for each storage level in the hard disk. The storage module is used to store the data into the target replication storage level in each of the replication storage levels to obtain the target sorted string table; The merging module is used to merge the target sorted string table with the sorted string tables in each of the lower-level copy storage levels to obtain the merged sorted string tables. The writing module is used to generate a new version of each storage level based on the merged sorted string tables in each of the replicated storage levels and the unchanged sorted string tables in each of the replicated storage levels; before updating the current version to the new version, when a read request from another thread is received, the module queries the sorted string tables of each storage level of the current version, returns the query results, updates the current version to the new version, and adds the current version before the update to the list of versions to be recycled, thus completing the writing of the data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images