Data processing method, computing device, storage medium and computer program product

Abstract

Embodiments of this specification relate to a data processing method, a computing device, a storage medium, and a computer program product, wherein the data processing method includes: obtaining at least two historical versions of data to be processed, and data files corresponding to the at least two historical versions of data; determining, based on a deletion mark carried by any of the at least two historical versions of data, in the data files corresponding to the at least two historical versions of data, an associated data file of the data to be processed and data association information corresponding to the associated data file; and deleting the deletion mark and historical version data contained in the associated data file when it is determined, based on the data association information, that the associated data file meets a preset deletion condition.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of database technology, and in particular to data processing methods, computing devices, storage media, and computer program products. Background Art

[0002] In the field of data storage, data written by users can usually be cached in memory. When the data cached in memory reaches a preset amount of data, the cached data in memory is merged and written to disk. When the written data is updated, the data update can usually be achieved by appending new data. Then, when the written data is updated frequently, a data will generate multiple historical versions stored in the same or different data files, thereby affecting data query efficiency. Based on this, historical versions of written data can usually be deleted. When deleting data, a deletion mark can be written to the data to be deleted. When the data is subsequently merged, the data corresponding to the deletion mark is filtered to achieve the deletion of the historical version of the data.

[0003] However, when users frequently update and delete data or when a large amount of data is stored, a large number of deletion marks and historical versions of data will accumulate, causing a large amount of deletion marks and historical versions of data to occupy storage space, thereby affecting data query performance. Therefore, an effective technical solution is urgently needed to solve the above problem. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a data processing method is provided, which is applied to a database based on an LSM storage engine, comprising:

[0006] Obtaining at least two historical versions of data to be processed, and data files corresponding to the at least two historical versions of data;

[0007] Determining, based on a deletion mark carried by any of the at least two historical versions of data, in the data files corresponding to the at least two historical versions of data, an associated data file of the data to be processed and data association information corresponding to the associated data file;

[0008] When it is determined according to the data association information that the associated data file meets a preset deletion condition, the deletion mark and the historical version data contained in the associated data file are deleted.

[0009] According to a second aspect of an embodiment of this specification, a data processing device is provided, which is applied to a database based on an LSM storage engine, including:

[0010] An acquisition module is configured to acquire at least two historical versions of the data to be processed and data files corresponding to the at least two historical versions of the data;

[0011] a determination module configured to determine, based on a deletion mark carried by any of the at least two historical versions of data, an associated data file of the data to be processed and data association information corresponding to the associated data file in the data files corresponding to the at least two historical versions of data;

[0012] The deletion module is configured to delete the deletion mark and historical version data contained in the associated data file when it is determined according to the data association information that the associated data file meets a preset deletion condition.

[0013] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0014] memory and processor;

[0015] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above method are implemented.

[0016] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and the computer program / instruction implements the steps of the above method when executed by a processor.

[0017] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0018] One embodiment of the present specification provides a data processing method, including: obtaining at least two historical versions of data to be processed, and data files corresponding to the at least two historical versions of data; determining, based on a deletion mark carried by any of the at least two historical versions of data, in the data files corresponding to the at least two historical versions of data, an associated data file of the data to be processed and data association information corresponding to the associated data file; and deleting the deletion mark and historical version data contained in the associated data file when it is determined, based on the data association information, that the associated data file meets a preset deletion condition.

[0019] The above method obtains at least two historical versions of the data to be processed and the data files corresponding to each historical version of the data, and determines the associated data files of the data to be processed and the data association information corresponding to the associated data files from the data files corresponding to the at least two historical versions of the data based on the deletion marks carried by any version of the at least two historical versions of the data. When it is determined based on the data association information that the associated data files meet the preset deletion conditions, the deletion marks and historical version data contained in the associated data files are deleted, thereby achieving timely deletion of the deletion marks and historical version data, avoiding the accumulation of a large number of undelete deletion marks and historical version data in the storage space, and further avoiding affecting the data query performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic diagram of an application scenario of a data processing method provided by an embodiment of this specification;

[0021] Figure 2 is a flow chart of a data processing method provided by one embodiment of this specification;

[0022] Figure 3 This is a flowchart of determining associated data files in a data processing method provided by one embodiment of this specification;

[0023] Figure 4 This is a flowchart of a data processing method provided by one embodiment of this specification;

[0024] Figure 5 This is a schematic diagram of the structure of a data processing device provided by one embodiment of this specification;

[0025] Figure 6 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0026] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0027] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0028] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0029] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] First, the terms involved in one or more embodiments of this specification are explained.

[0031] LSM (Log-Structured Merge Tree) is a data structure used in storage systems, particularly suitable for scenarios requiring efficient write processing. LSM trees improve write efficiency by converting random write operations into sequential write operations and manage data through a hierarchical storage structure.

[0032] KV: Key-Value, a key-value pair is a simple data storage model in which each data item consists of a unique identifier (key) and associated data (value). This model is very suitable for fast lookup and update operations because it allows direct access to the corresponding value through the key.

[0033] Map: is a data structure or mechanism used to manage and optimize data storage and access.

[0034] Bloom filter: Placed in the metadata area of each data file, it is used to quickly check whether a key value exists in the current data file. False positives may occur, indicating that the key value exists in the current data file, but false negatives cannot occur, indicating that the key value does not exist in the current data file.

[0035] In practical applications, in a database designed based on LSM, user-written data is first cached in memory, then sorted and written to disk. Data already written to disk cannot be modified. Therefore, updates and deletions of data (i.e., KV data) require writing new data to update or delete it. During queries, the Bloom filter in each data file determines whether to scan the file. For all data files to be scanned, the Bloom filter creates an iterator and manages the iterator using a priority queue, ensuring that newer versions of data are scanned first. This allows queries to retrieve all historical versions of the scanned data.

[0036] When data on disk is frequently updated, new data is constantly written. This can lead to multiple versions of a KV data being scattered across different data files, severely impacting data query efficiency. To address this issue, the database can periodically trigger a minor compaction process to consolidate and consolidate data files. This process selects several data files of similar size, cleans and sorts the data within them, and merges them into a new file. For larger data files, this process is typically not used regularly. Instead, a major compaction process is triggered. This process is triggered at a longer interval than the previous one, and it selects all data files and merges them into a single, larger file. This process is typically triggered at a longer interval, such as 20 or 30 days.

[0037] In this database, for data deletion operations, a delete marker will be written to the data. During the data query process, if data carrying this delete marker is scanned, all previous historical versions of the data will be ignored, thus achieving the deletion effect. However, these delete markers can usually only be deleted through the above-mentioned data merging process, because the data sorting process cannot guarantee that all data files containing all historical versions of a data will be selected. If the delete marker is cleared based on the data sorting process, some historical version data that has not been deleted will be queried in subsequent data queries, which is contrary to the delete operation called previously. If users frequently update and delete data in the database, a large number of delete markers and historical version data will accumulate in the database, which will greatly affect query performance. Therefore, there is an urgent need for an effective method to solve the above technical problems.

[0038] In this specification, a data processing method is provided. This specification also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0039] See also Figure 1 , Figure 1 A schematic diagram of an application scenario of a data processing method provided according to an embodiment of this specification is shown. The data processing method includes:

[0040] Obtaining at least two historical versions of data to be processed, and data files corresponding to the at least two historical versions of data;

[0041] Determining, based on a deletion mark carried by any of the at least two historical versions of data, in the data files corresponding to the at least two historical versions of data, an associated data file of the data to be processed and data association information corresponding to the associated data file;

[0042] When it is determined according to the data association information that the associated data file meets a preset deletion condition, the deletion mark and the historical version data contained in the associated data file are deleted.

[0043] Specifically, Figure 1 It includes a terminal device 102 and a distributed database 104.

[0044] During specific implementation, the user can send a data query request to the distributed database 104 through the end-side device 102. The distributed database 104 responds to the data query request, determines the data to be processed corresponding to the data query request, and obtains at least two historical versions of the data to be processed, and data files corresponding to at least two historical versions of the data. According to the deletion mark carried by any historical version of the at least two historical versions of the data, the associated data file of the data to be processed and the data association information corresponding to the associated data file are determined in the data files corresponding to the at least two historical versions of the data. When it is determined according to the data association information that the associated data file meets the preset deletion condition, the deletion mark and historical version data contained in the associated data file are deleted, thereby realizing timely deletion of the deletion mark and historical version data, avoiding the accumulation of a large number of undelete deletion marks and historical version data in the storage space, and further avoiding affecting the data query performance.

[0045] The end-side device 102 may include a browser, an application (APP), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program, a type of lightweight application), or a cloud application. The end-side device may be developed based on a software development kit (SDK) for the corresponding service provided by the server, such as a real-time communication (RTC) SDK. The end-side device may be deployed in an electronic device and may rely on the device or certain apps in the device to operate. The electronic device may have a display and support information browsing, such as a personal mobile terminal such as a mobile phone, tablet computer, or personal computer. Various other types of applications may also be configured in the electronic device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0046] See also Figure 2 , Figure 2 A flowchart of a data processing method provided according to an embodiment of the present specification is shown, which is applied to a database based on an LSM storage engine and specifically includes the following steps.

[0047] Step 202: Obtain at least two historical versions of the data to be processed, and data files corresponding to the at least two historical versions.

[0048] Specifically, the data processing method provided in the embodiments of this specification can be applied to a database, and specifically to a distributed database, which can be built based on an LSM storage engine.

[0049] Therefore, the data to be processed can be understood as data stored in a distributed database. Furthermore, the data to be processed can be understood as data stored on a disk that cannot be modified. The at least two historical versions of the data to be processed can be understood as all versions of the data after the data to be processed has been updated. For example, for data to be processed A, the user wrote data A1 on January 2, the user wrote data A2 on January 3, and the user wrote data A3 on January 5. Data A1, A2, and A3 are all historical versions of data A. It can be understood that there is a write order relationship between the at least two historical versions of the data to be processed, and the data version written first is smaller than the data version written later. When querying data, the new version of the data to be processed will be scanned first. The data files corresponding to the at least two historical versions of the data can be understood as the data files storing each historical version of the at least two historical versions of the data. For example, data A1 is stored in data file F1, data A2 is stored in data file F2, and data A3 is stored in data file F3. Then, data files F1, F2, and F3 are the data files corresponding to the at least two historical versions of the data. It is understandable that among at least two historical versions of data, there may also be multiple historical versions of data stored in the same data file, for example, data A1 and data A2 are stored in data file F1, and data A3 is stored in data file F2.

[0050] Based on this, the distributed database can be scanned to obtain at least two historical versions of the data to be processed, and the data files storing each of the at least two historical versions of the data can be obtained.

[0051] In practical applications, before obtaining at least two historical versions of the data to be processed and the data files corresponding to the at least two historical versions of the data, the process further includes:

[0052] In response to a data query request, to-be-processed data corresponding to the data query request is determined.

[0053] The data query request may be understood as a request received by a distributed database to query data to be processed.

[0054] Specifically, in order to ensure that deletion marks that affect data reading performance are cleared, in response to a data query request, the pending data that the data query request wants to query can be determined, and then at least two historical versions of the pending data and the corresponding data files can be obtained.

[0055] In summary, by performing subsequent deletion mark statistics and determining the associated data files for the pending data that the user wants to query in response to a data query request, and placing the deletion mark statistics at the time when the user reads, the deletion marks that affect the user's reading performance can be cleared without deleting all deletion marks and wasting computing resources.

[0056] Step 204: According to the deletion mark carried by any historical version data in the at least two historical versions of data, determine the associated data file of the to-be-processed data and the data association information corresponding to the associated data file in the data files corresponding to the at least two historical versions of data.

[0057] Among them, the deletion mark can be understood as a mark written to the data stored in the distributed database. The data carrying the deletion mark can indicate that the data is marked for deletion, and all historical versions of the data in the distributed database will be deleted. When querying the data, if the data carrying the deletion mark is scanned, then all historical versions of the data will not be queried. For example, data A1, A2, and A3 are all historical versions of data A, and the data version of A3 is greater than the data version of A2, which is greater than the data version of A1. That is, A3 is the new version of data A. In response to a data query request for data A3, if it is determined that data A3 carries a deletion mark, then data A1 and data A2 will not be returned as query results. The associated data files of the data to be processed can be understood as data files that store historical versions of data that can be deleted by a deletion marker. For example, data A1 is stored in data file F1, data A2 is stored in data file F2, and data A3 is stored in data file F3. If data A3 carries a deletion marker, it means that both data A2 and data A1 will be deleted by the deletion marker. In this case, the associated data files of data file F3 are data files F1 and F2, the associated data file of data file F1 is data file F3, and the associated data file of data file F1 is data file F3. Therefore, data files F1, F2, and F3 are all associated data files of the data to be processed A. The data association information corresponding to the associated data files can be understood as the file information and file number of the data files that are associated with the associated data files. For example, for data file F3, its associated data files are F1 and F2. The data association information of data file F3 can be "F1+1, F2+1", with the file information being F1 and F2 and the file number being 2.

[0058] Specifically, based on the deletion marks carried by any one or more historical versions of data in at least two historical versions of data, data files with associated relationships can be determined in the data files corresponding to at least two historical versions of data, thereby determining the associated data files of the data to be processed and the data association information corresponding to the associated data files.

[0059] In a specific implementation, determining, based on the deletion mark carried by any historical version data of the at least two historical versions of data, in the data files corresponding to the at least two historical versions of data, the associated data file of the data to be processed and the data association information corresponding to the associated data file includes:

[0060] Determining, according to a deletion mark carried by any historical version data of the at least two historical versions of data, in the data files corresponding to the at least two historical versions of data, an associated data file of the data to be processed;

[0061] Determining, based on the deletion mark, an association relationship between the arbitrary historical version data and other historical version data;

[0062] According to the association relationship, data association information corresponding to the associated data file is recorded.

[0063] The "other historical version data" can be understood as the other historical version data among the at least two historical version data, excluding the historical version data carrying the deletion mark. The association relationship between any historical version data and other historical version data can be understood as the relationship between whether the other historical version data can be deleted by the deletion mark carried by any historical version data.

[0064] Specifically, based on the deletion marks carried by any historical version data in at least two historical versions of data, the associated data files of the data to be processed can be determined in the data files corresponding to the at least two historical versions of data, and based on the deletion marks, the association relationship between the any historical version data and other historical version data can be determined, and the data association information corresponding to the associated data file can be recorded based on the association relationship.

[0065] Furthermore, when determining the associated data files, the data files corresponding to each historical version of data can be sequentially determined based on the order of the data versions of at least two historical versions of data. If so, the associated data files with which the associated data files are associated can be further determined. For example, the associated data files corresponding to the latest version of the historical version of data can be used for determination.

[0066] For example, data A1, A2 and A3 are all historical version data of data A, and the data version of A3 is greater than the data version of A2, which is greater than the data version of A1, that is, A3 is the new version data of data A, data A1 is stored in data file F1, data A2 is stored in data file F2, and data A3 is stored in data file F3. When A3 carries a deletion mark, the deletion mark carried by data A3 can delete data A1 and data A2. Then, there is an association relationship between data A3 and data A1 and A2. Then, there is an association relationship between data file F3 stored in data A3 and data files F1 and F2 stored in data A1 and A2. Then, data file F3 is an associated data file, and the data association information corresponding to the associated data file F3 can include file information and file quantity of data file F1 and data file F2, that is, "F1+1, F2+1".

[0067] In summary, by counting the deletion marks of the data to be processed, the associated data files associated with the data to be processed are obtained, which provides information for the subsequent deletion of the deletion marks.

[0068] Furthermore, determining, based on the deletion mark carried by any of the at least two historical versions of data, the associated data file of the data to be processed in the data files corresponding to the at least two historical versions of data, includes:

[0069] If it is determined that any historical version data among the at least two historical version data carries a deletion mark, determining the data file corresponding to the any historical version data as the first associated data file;

[0070] determining, according to the deletion mark, a second associated data file associated with the first associated data file from the data files corresponding to the at least two historical versions of the data;

[0071] The first associated data file and the second associated data file are determined as associated data files of the data to be processed.

[0072] Specifically, after determining the data file corresponding to any historical version data as the first associated data file, the first associated data file can be placed in a queue for temporary storage, and the second associated data file determined subsequently is also placed in the queue for temporary storage. All associated data files temporarily stored in the queue are associated data files of the data to be processed.

[0073] For example, data A1, A2 and A3 are all historical version data of data A, and the data version of A3 is greater than the data version of A2, which is greater than the data version of A1, that is, A3 is the new version data of data A. Data A1 is stored in data file F1, data A2 is stored in data file F2, and data A3 is stored in data file F3. When A3 carries a deletion mark, data file F3 stored in data A3 is determined as the first associated data file. The deletion mark carried by data A3 can delete data A1 and data A2. Then, there is an association relationship between data A3 and data A1 and A2. Then, there is an association relationship between data file F3 stored in data A3 and data files F1 and F2 stored in data A1 and A2. Then, data files F1 and F2 can be determined as second associated data files, and the first associated data file F3 and the second associated data files F1 and F2 can be determined as associated data files of the data to be processed.

[0074] In summary, by determining the associated data files of the data to be processed based on the deletion mark, it is convenient to subsequently record data association information according to the associated data files, and further facilitate the subsequent judgment of whether to delete the deletion marks and historical version data contained in the associated data files.

[0075] Step 206: When it is determined according to the data association information that the associated data file meets the preset deletion condition, the deletion mark and historical version data included in the associated data file are deleted.

[0076] Specifically, it is possible to determine whether the associated data file meets the preset deletion condition based on the data association information. If it is determined that the associated data file meets the preset deletion condition, the deletion mark and historical version data contained in the associated data file are deleted.

[0077] In specific implementation, when it is determined that the data association information meets a preset association threshold and / or the number of associated data files meets a preset number threshold, it can be determined that the associated data file meets the preset deletion condition.

[0078] In practical applications, there are multiple associated data files;

[0079] After determining the associated data file of the data to be processed and the data association information corresponding to the associated data file, the method further includes:

[0080] Creating a set of associated data files according to the plurality of associated data files of the data to be processed and data association information corresponding to each associated data file;

[0081] Calculating the set association information of the set of associated data files according to the data association information corresponding to each associated data file;

[0082] The step of deleting the deletion mark and historical version data contained in the associated data file when it is determined according to the data association information that the associated data file meets a preset deletion condition includes:

[0083] When it is determined according to the set association information that the associated data file set meets a preset deletion condition, the deletion marks and historical version data included in the plurality of associated data files are deleted.

[0084] The associated data file set may include associated data files and data association information corresponding to the associated data files. The set association information may be understood as the sum of the data association information corresponding to each associated data file included in the associated data file set.

[0085] Specifically, a set of associated data files may be created based on multiple associated data files of the data to be processed and the data association information corresponding to each associated data file in the multiple associated data files. Based on the data association information of the multiple associated data files in the associated data file set, the sum of the data association information of the multiple associated data files may be calculated as the set association information of the associated data file set. If it is determined that the associated data file set meets a preset deletion condition based on the set association information, all deletion marks and historical version data contained in the multiple associated data files in the associated data file set may be deleted.

[0086] In practical applications, the collection of related data files can be a file queue, and a map can be used to store data association information. Figure 3 , Figure 3 FIG. 1 shows a flow chart of determining associated data files in a data processing method according to an embodiment of the present specification. Figure 3 As shown, when a user performs a data query, the query of the pending data KV is used as an example to illustrate. During the data query, the four historical versions of the pending data KV, V4, V3, V2 and V1, are scanned, among which the version of V4 is greater than the version of V3, which is greater than the version of V2, which is greater than the version of V1.

[0087] First, use the map to store data association information, and use a file queue to temporarily store the data files of historical version data. At this time, the file queue is empty. First, scan the historical version data V4 and determine that the historical version data V4 carries a deletion mark. Then, put the data file D corresponding to the historical version data V4 into the file queue, indicating that the data file D is the associated data file of the data KV to be processed.

[0088] Next, the historical version data V3 is scanned. The historical version data V3 does not carry a deletion mark. At this time, the file queue is checked and it is found that there is data file D in the file queue. Since the deletion mark carried by the historical version data V4 will delete the historical version data V3, there is an association between the historical version data V4 and V3, and the historical version data V3 is stored in the data file C, which means that there is also an association between data file D and data file C. Then, data file C is also put into the file queue, and the association of historical version data V3 is recorded in the map of data file D, that is, the association of data file D with data file C is recorded, and the number of associated files is 1. Then, the data association information of data file D is "C+1" at this time. At the same time, the association of historical version data V4 is recorded in the map of data file C. At this time, the data association information of data file C is "D+1".

[0089] Then, the historical version data V2 stored in data file B is scanned. The historical version data V2 carries a deletion mark. The file queue is checked and it is found that data file D and data file C exist in the file queue. The deletion mark carried by the historical version data V4 stored in data file D will delete the historical version data V2. That is to say, there is an association between historical version data V4 and historical version data V2, that is to say, there is also an association between data file D and data file B. Then, data file B is also put into the file queue, and the association of historical version data V2 is recorded in the map of data file D. Then, the data association information of data file D is now "C+1; B+1", and the association of historical version data V4 is recorded in the map of data file B. Then, the data association information of data file B is now "D+1".

[0090] Then, the historical version data V1 stored in data file A is scanned. The historical version data V1 does not carry a deletion mark. The file queue is checked and it is found that there are data files D, data file C and data file B in the file queue. Among them, the deletion mark carried by the historical version data V4 stored in data file D will delete the historical version data V1, that is, there is an association relationship between the historical version data V4 and the historical version data V1, that is, there is also an association relationship between data file D and data file A. Then, data file A is put into the file queue and the association of the historical version data V1 is recorded in the map of data file D. Then the data association information of data file D is "C+1; B+1; A+1". According to the association of historical version data V4 recorded in the map of file A, the data association information of data file A is now "D+1"; the deletion mark carried by historical version data V2 stored in data file B will also delete the historical version data V1, that is, there is an association relationship between historical version data V2 and historical version data V1, that is, there is also an association relationship between data file B and data file A, then, the association of historical version data V1 is recorded in the map of data file B, then the data association information of data file B is now "D+1; A+1", and the association of historical version data V2 is recorded in the map of data file A, then the data association information of data file A is now "D+1; B+1".

[0091] Then, the data files A, B, C, and D contained in the file queue are the associated data files of the data KV to be processed, and the set of associated data files is the file queue.

[0092] In summary, by scanning historical version data in sequence, the correlation relationships between historical version data can be statistically analyzed, which makes it easier to determine whether a clearing operation is needed based on the data correlation information, further ensuring that data query performance is not affected.

[0093] In specific implementation, after determining the above-mentioned associated data files, a data file selection strategy can be executed according to the associated data files and data association information. The data file selection strategy is the process of determining the above-mentioned associated data files. It can be understood that the larger the sum of the associated values of the data association information recorded in the map of an associated data file, the more deletion marks and historical version data associated with the associated data file, and the higher the cleaning priority of the associated data file. Since each associated data file maintains a map, all data files in the distributed database can be grouped according to the association of these maps. Combined with the above Figure 3In the example shown, the associated data files A, B, C, and D may belong to the same group, i.e., the associated data file set. The associated data files included in the group with the largest sum of associated values may be marked for deletion and their historical version data may be cleaned up. The specific implementation is as follows.

[0094] The step of deleting the deletion marks and historical version data contained in the plurality of associated data files when it is determined according to the set association information that the associated data file set meets a preset deletion condition comprises:

[0095] If it is determined that the set association information meets a preset association threshold, determining that the set of associated data files meets a preset deletion condition, and deleting the deletion marks and historical version data included in the plurality of associated data files; or

[0096] When it is determined that the number of the associated data files meets a preset number threshold, it is determined that the associated data file set meets a preset deletion condition, and the deletion marks and historical version data included in the plurality of associated data files are deleted.

[0097] In one embodiment of the present specification, the sum of data association information of all associated data files in an associated data file set may be calculated as set association information. When it is determined that the set association information meets a preset association threshold, it is determined that the associated data file set meets a preset deletion condition, and the deletion marks and historical version data contained in multiple associated data files in the associated data file set are deleted.

[0098] In another embodiment of the present specification, the number of associated data files included in an associated data file set can be determined. When it is determined that the number of files meets a preset number threshold, it can be determined that the associated data file set meets a preset deletion condition, and the deletion marks and historical version data included in multiple associated data files in the associated data file set can be deleted.

[0099] In practical applications, the preset association threshold can be obtained by multiplying the total number of KV data in all associated data files contained in the associated data file set by the ratio threshold R, and the ratio threshold R can be set manually and flexibly adjusted. If it is determined that the user frequently updates the data in the distributed database, the ratio threshold R can be set larger, otherwise the ratio threshold R can be set smaller.

[0100] Then, further, for a distributed database, the total correlation numerical score of the maps of all data files in the distributed database can be periodically calculated. When it is determined that the total correlation numerical score is greater than the preset correlation threshold, the data file selection strategy is executed according to the above process, and the data files are grouped according to the maps of all data files. According to the above process, multiple sets of related data files are obtained, and the set of related data files with the largest correlation numerical score (i.e., set correlation information) is selected, and the deletion marks and historical version data of all related data files in the set of related data files are deleted.

[0101] In summary, by selecting the set of associated data files with the most associated deletion marks, the deletion marks in the set of associated data files can be deleted in a timely manner without having to wait until the data merge process is triggered to clear the deletion marks, thereby ensuring the data reading performance of the distributed database.

[0102] Furthermore, deleting the deletion marks and historical version data contained in the plurality of associated data files includes:

[0103] Obtain all data files in the data partition of the data storage to be processed;

[0104] According to the data to be processed, all the data files are filtered to obtain data filtering results, and according to the data filtering results, a reference data file containing the data to be processed is obtained;

[0105] In the case where it is determined that the associated data file set includes the reference data file, deletion marks and historical version data included in the plurality of associated data files are deleted.

[0106] Among them, data partition can be understood as the basic unit of data management in a distributed database.

[0107] Specifically, after determining the associated data file set, before deleting the deletion marks and historical version data contained in the associated data files in the associated data file set, the associated data file set can be checked to ensure that the associated data file set contains data files corresponding to all historical versions of the data to be processed, thereby avoiding omission of historical version data. Then, all data files in the data partition storing the data to be processed can be obtained, and data filtering can be performed on all data files based on the data to be processed to obtain data filtering results. Based on the data filtering results, reference data files containing the data to be processed can be obtained. If it is determined that all reference data files are contained in the associated data file set, it indicates that no historical version data is missing, and the deletion operation of the deletion marks and historical version data can be performed.

[0108] For example, the associated data file set includes associated data file A, associated data file B, associated data file C and associated data file D. Data filtering is performed on all data files in the data partition, and the data filtering results determine that reference data file A, reference data file B and reference data file C contain the data to be processed. At this time, the associated data file set contains all reference data files, indicating that there is no omission, and a deletion operation is performed at this time; if the data filtering results determine that reference data file A, reference data file B, and reference data file E contain the data to be processed, since the associated data file set does not contain the reference data file E, it means that there is an omission at this time, and the deletion mark and historical version data deletion operation are not performed.

[0109] In actual applications, the Bloom filters of all data files in the data partition can be checked, and the data files that return positive results (i.e., reference data files) can be recorded to check whether these positive data files are included in the associated data file set. If so, the deletion marks and historical version data in the associated data file set can be cleared. If not, the deletion marks and historical version data in the associated data file set cannot be cleared, because this indicates that there is historical version data of the data to be processed that is not included in the associated data file set.

[0110] In summary, by performing data checking before the deletion operation, it is ensured that the deleted associated data file set can contain all historical version data of the data to be processed, avoiding the omission of historical version data, and further avoiding the error of querying historical version data due to the deletion of the deletion mark but the historical version data not being deleted.

[0111] In addition, after deleting the deletion marks and historical version data contained in the multiple associated data files, the method further includes:

[0112] Obtaining the remaining data contained in the plurality of associated data files;

[0113] The remaining data are merged to obtain a target data file.

[0114] The remaining data can be understood as other KV data in the associated data file except the historical version data.

[0115] Specifically, after deleting the deletion marks and historical version data contained in the multiple associated data files, the other KV data except the historical version data in the multiple associated data files can be merged to obtain the target data file, thereby achieving file merging.

[0116] In practical applications, the method further includes:

[0117] Determine the data partition for storing the data to be processed;

[0118] When it is determined that the data storage capacity of the data partition is greater than a preset storage threshold, the data partition is split to obtain a target data partition obtained after the split.

[0119] The target data partition obtained after the split may be at least two data partitions, for example, one data partition may be split into two data partitions.

[0120] Specifically, the data partition of the data to be processed can be determined, and the data storage capacity of the data partition can be monitored. When it is determined that the data storage capacity of the data partition is greater than the preset storage threshold, the data partition is split to obtain the target data partition after the split, so that the distributed database can cope with massive data with high flexibility and ensure data storage performance.

[0121] However, in actual applications, the preset storage threshold can be a fixed-size threshold, that is, whenever the data storage capacity of a data partition reaches a fixed-size threshold (for example, 8GB), the data partition is split into two new data partitions. This method is suitable for storing larger data tables, that is, using multiple data partitions to store data in shards for a larger data table. However, for smaller data tables, this method will result in a smaller number of data partitions stored in the smaller data table, which is not conducive to database load balancing. Moreover, if the smaller data table is updated frequently, even if the above-mentioned data processing method is used to delete the mark and clear the historical version data, the clearing granularity will be larger, affecting resources and performance. Based on this, the preset storage threshold can be dynamically calculated according to the number of partitions of the data partition corresponding to the data table through the following data partition splitting strategy. The specific implementation method is as follows.

[0122] After determining the data partition for storing the data to be processed, the method further includes:

[0123] Determine a data table corresponding to the data to be processed, and determine the number of partitions of the data partitions stored in the data table;

[0124] A preset storage threshold is calculated according to the number of partitions.

[0125] The data table corresponding to the data to be processed can be understood as the data table to which the data to be processed belongs, and the data table can be understood as the data table for which the above-mentioned data file selection strategy is enabled.

[0126] Specifically, for the data table corresponding to the data to be processed, the number of partitions of the data partition of the data table in the data storage node in the distributed database can be determined, and the preset storage threshold can be calculated based on the number of partitions to facilitate subsequent data partition splitting operations based on the preset storage threshold.

[0127] In practical applications, the goal of this data partition splitting strategy is to quickly split a data table's data partitions into a certain number, and the preset storage threshold can be differentiated according to the above-mentioned data processing method. If the data file selection strategy of the above-mentioned data processing method is implemented, the preset storage threshold can be set larger to give the data partitions a smaller granularity, further expediting the deletion of deletion marks and historical version data. Specifically, the number of partitions of the data table in the current data storage node can be determined, and the preset storage threshold is obtained by multiplying the number of partitions by a custom constant and then multiplying it by 128MB. If it is determined that the data storage capacity of the data partition is greater than the preset storage threshold, it is determined that the data partition needs to be split.

[0128] It is understandable that the above storage capacity monitoring can be performed on data partitions stored in all data tables in the distributed database.

[0129] In summary, by calculating the preset storage threshold according to the number of partitions, it is convenient to perform the data partition splitting operation according to the preset storage threshold, thereby ensuring that the computing resources and data storage performance of the distributed database are not affected.

[0130] Furthermore, determining the number of partitions of the data partition stored in the data table includes:

[0131] Determining the number of partitions of the data partition stored in the data table according to a preset time interval;

[0132] Calculating a preset storage threshold based on the number of partitions includes:

[0133] Calculating a first storage threshold according to the preset time interval and the number of partitions;

[0134] A second storage threshold calculated at a previous time interval is obtained, and a preset storage threshold is determined according to a comparison result between the first storage threshold and the second storage threshold.

[0135] The preset time interval can be understood as the time interval for calculating the preset storage threshold. For example, the distributed database can calculate the preset storage threshold every 10 minutes, or every hour or day. This specification does not limit this. Then, the first storage threshold can be understood as the storage threshold calculated at the current time interval, and the second storage threshold can be understood as the storage threshold calculated at the previous time interval.

[0136] Specifically, the number of partitions of the data partition stored in the data table for the current time interval can be determined according to a preset time interval, and a first storage threshold can be calculated in the current time interval based on the number of partitions. A second storage threshold calculated in the time interval before the current time interval can be obtained, and the first storage threshold and the second storage threshold can be compared to obtain a comparison result. If the comparison result determines that the first storage threshold is greater than the second storage threshold, the first storage threshold can be determined as the preset storage threshold; if the comparison result determines that the first storage threshold is less than the second storage threshold, the second storage threshold can be determined as the preset storage threshold. In other words, the larger of the first storage threshold and the second storage threshold can be determined as the preset storage threshold.

[0137] It can be understood that in actual applications, the smaller storage threshold between the first storage threshold and the second storage threshold can also be determined as the preset storage threshold according to actual needs, that is, when the first storage threshold is determined to be greater than the second storage threshold according to the comparison result, the second storage threshold is determined to be the preset storage threshold; when the first storage threshold is determined to be less than the second storage threshold according to the comparison result, the first storage threshold is determined to be the preset storage threshold.

[0138] In addition, since the preset storage threshold is positively correlated with the number of partitions of the data partition in the data storage node, when the distributed database expands the nodes, some data partitions will be moved to the expanded data storage node, resulting in a decrease in the preset storage threshold. Based on this, each time the preset storage threshold is calculated, the calculated preset storage threshold can be recorded in a variable. Then, the second storage threshold calculated in the previous time interval can be obtained from the variable. Moreover, when the preset storage threshold is calculated for the first time, it indicates that the data partition is restarted, that is, the data partition has undergone load balancing migration or migration caused by expanding the data storage node. At this time, the preset storage threshold can be calculated according to the following formula: (data storage capacity of the data partition / 128MB×constant+1.0)×128MB×constant.

[0139] In summary, by dynamically calculating the preset storage threshold, we can prevent the distributed database from splitting and avalanching after node expansion, and ensure load balancing of data partitions.

[0140] In a practical application, a high-frequency update scenario was simulated for a distributed database using the aforementioned data processing method. The secondary index table in the distributed database was constantly queried and updated, generating a large number of delete markers. Since the initial data volume of the secondary index table was 10 million rows, without the aforementioned data processing method, this initial data volume would be concentrated in a 600MB data file. Multiple delete markers would then remain in this data file for a long time. However, due to the large size of the data file, the data cleanup process could not select it, forcing the data file to be cleared and merged by the data merge process triggered some time later. However, when the aforementioned data processing method was implemented with the ratio threshold R set to 2, delete markers and historical version data were detected every half hour, keeping the data read response time (RT) under control. When the aforementioned data processing method was implemented with the ratio threshold R set to 1, delete markers and historical version data were detected every six minutes, triggering the delete operation more quickly and keeping the data read response time low, thereby ensuring data read performance.

[0141] To summarize, the above method obtains at least two historical versions of the data to be processed, and the data files corresponding to each historical version of the data, and determines the associated data files of the data to be processed and the data association information corresponding to the associated data files from the data files corresponding to the at least two historical versions of the data according to the deletion marks carried by any version of the at least two historical versions of the data. When it is determined according to the data association information that the associated data files meet the preset deletion conditions, the deletion marks and historical version data contained in the associated data files are deleted, thereby achieving timely deletion of the deletion marks and historical version data, avoiding the accumulation of a large number of undelete deletion marks and historical version data in the storage space, and further avoiding affecting the data query performance.

[0142] The following combined Figure 4 , taking the application of the data processing method provided in this specification in a distributed database as an example, the data processing method is further explained. Figure 4 A flowchart of a data processing method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0143] Step 402: In response to the data query request, determine the to-be-processed data corresponding to the data query request.

[0144] Step 404: Obtain at least two historical versions of the data to be processed and data files corresponding to the at least two historical versions.

[0145] Step 406: When it is determined that any historical version data in the at least two historical version data carries a deletion mark, the data file corresponding to the any historical version data is determined as the first associated data file.

[0146] Step 408: Determine, based on the deletion mark, a second associated data file associated with the first associated data file from data files corresponding to at least two historical versions of data.

[0147] Step 410: Determine the first associated data file and the second associated data file as associated data files of the data to be processed.

[0148] Step 412: Determine the association relationship between any historical version data and other historical version data based on the deletion mark, and record data association information corresponding to the associated data file based on the association relationship.

[0149] Step 414: When it is determined that the data association information meets the preset association threshold, it is determined that the associated data file meets the preset deletion condition, and the deletion mark and historical version data included in the associated data file are deleted.

[0150] Step 416: Obtain the remaining data contained in the multiple associated data files, merge the remaining data, and obtain the target data file.

[0151] To summarize, the above method obtains at least two historical versions of the data to be processed, and the data files corresponding to each historical version of the data, and determines the associated data files of the data to be processed and the data association information corresponding to the associated data files from the data files corresponding to the at least two historical versions of the data according to the deletion marks carried by any version of the at least two historical versions of the data. When it is determined according to the data association information that the associated data files meet the preset deletion conditions, the deletion marks and historical version data contained in the associated data files are deleted, thereby achieving timely deletion of the deletion marks and historical version data, avoiding the accumulation of a large number of undelete deletion marks and historical version data in the storage space, and further avoiding affecting the data query performance.

[0152] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, which is applied to a database based on an LSM storage engine. Figure 5 FIG1 shows a schematic diagram of the structure of a data processing device provided by an embodiment of this specification. Figure 5 As shown, the device includes:

[0153] An acquisition module 502 is configured to acquire at least two historical versions of the data to be processed and data files corresponding to the at least two historical versions of the data;

[0154] A determination module 504 is configured to determine, based on a deletion mark carried by any of the at least two historical versions of data, an associated data file of the data to be processed and data association information corresponding to the associated data file in the data files corresponding to the at least two historical versions of data;

[0155] The deletion module 506 is configured to delete the deletion mark and historical version data contained in the associated data file if it is determined according to the data association information that the associated data file meets a preset deletion condition.

[0156] In an optional embodiment, the determining module 504 is further configured to:

[0157] Determining, according to a deletion mark carried by any historical version data of the at least two historical versions of data, in the data files corresponding to the at least two historical versions of data, an associated data file of the data to be processed;

[0158] Determining, based on the deletion mark, an association relationship between the arbitrary historical version data and other historical version data;

[0159] According to the association relationship, data association information corresponding to the associated data file is recorded.

[0160] In an optional embodiment, the determining module 504 is further configured to:

[0161] If it is determined that any historical version data among the at least two historical version data carries a deletion mark, determining the data file corresponding to the any historical version data as the first associated data file;

[0162] determining, according to the deletion mark, a second associated data file associated with the first associated data file from the data files corresponding to the at least two historical versions of the data;

[0163] The first associated data file and the second associated data file are determined as associated data files of the data to be processed.

[0164] In an optional embodiment, there are multiple associated data files;

[0165] The determining module 504 is further configured to:

[0166] Creating a set of associated data files according to the plurality of associated data files of the data to be processed and data association information corresponding to each associated data file;

[0167] Calculating the set association information of the set of associated data files according to the data association information corresponding to each associated data file;

[0168] The deletion module 506 is further configured to:

[0169] When it is determined according to the set association information that the associated data file set meets a preset deletion condition, the deletion marks and historical version data included in the plurality of associated data files are deleted.

[0170] In an optional embodiment, the deletion module 506 is further configured to:

[0171] If it is determined that the set association information meets a preset association threshold, determining that the set of associated data files meets a preset deletion condition, and deleting the deletion marks and historical version data included in the plurality of associated data files; or

[0172] When it is determined that the number of the associated data files meets a preset number threshold, it is determined that the associated data file set meets a preset deletion condition, and the deletion marks and historical version data included in the plurality of associated data files are deleted.

[0173] In an optional embodiment, the deletion module 506 is further configured to:

[0174] Obtain all data files in the data partition of the data storage to be processed;

[0175] According to the data to be processed, all the data files are filtered to obtain data filtering results, and according to the data filtering results, a reference data file containing the data to be processed is obtained;

[0176] In the case where it is determined that the associated data file set includes the reference data file, deletion marks and historical version data included in the plurality of associated data files are deleted.

[0177] In an optional embodiment, the apparatus further includes a merging module configured to:

[0178] Obtaining the remaining data contained in the plurality of associated data files;

[0179] The remaining data are merged to obtain a target data file.

[0180] In an optional embodiment, the acquisition module 502 is further configured to:

[0181] In response to a data query request, to-be-processed data corresponding to the data query request is determined.

[0182] In an optional embodiment, the apparatus further includes a splitting module configured to:

[0183] Determine the data partition for storing the data to be processed;

[0184] When it is determined that the data storage capacity of the data partition is greater than a preset storage threshold, the data partition is split to obtain a target data partition obtained after the split.

[0185] In an optional embodiment, the splitting module is further configured to:

[0186] Determine a data table corresponding to the data to be processed, and determine the number of partitions of the data partitions stored in the data table;

[0187] A preset storage threshold is calculated according to the number of partitions.

[0188] In an optional embodiment, the splitting module is further configured to:

[0189] Determining the number of partitions of the data partition stored in the data table according to a preset time interval;

[0190] Calculating a first storage threshold according to the preset time interval and the number of partitions;

[0191] A second storage threshold calculated at a previous time interval is obtained, and a preset storage threshold is determined according to a comparison result between the first storage threshold and the second storage threshold.

[0192] To summarize, the above-mentioned device obtains at least two historical version data of the data to be processed and the data files corresponding to each historical version data, and determines the associated data file of the data to be processed and the data association information corresponding to the associated data file from the data files corresponding to the at least two historical version data according to the deletion mark carried by any version data in the at least two historical version data. When it is determined according to the data association information that the associated data file meets the preset deletion condition, the deletion mark and historical version data contained in the associated data file are deleted, thereby realizing timely deletion of the deletion mark and historical version data, avoiding the accumulation of a large number of undelete deletion marks and historical version data in the storage space, and further avoiding affecting the data query performance.

[0193] The above is a schematic diagram of a data processing device according to this embodiment. It should be noted that the technical solution of the data processing device and the technical solution of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical solution of the data processing device, please refer to the description of the technical solution of the above-mentioned data processing method.

[0194] Figure 66 shows a block diagram of a computing device 600 according to one embodiment of the present disclosure. Components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0195] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface controller (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0196] In one embodiment of the present application, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 6 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0197] Computing device 600 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 600 can also be a mobile or stationary server.

[0198] The processor 620 is configured to execute the following computer program / instruction, which implements the steps of the above-mentioned data processing method when executed by the processor.

[0199] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the computing device embodiment is generally similar to the data processing method embodiment, so the description is relatively simple. For relevant parts, refer to the description of the data processing method embodiment.

[0200] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0201] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from the other embodiments. In particular, the computer-readable storage medium embodiment is generally similar to the data processing method embodiment, so its description is relatively simple. For relevant portions, refer to the description of the data processing method embodiment.

[0202] An embodiment of the present specification further provides a computer program product, comprising a computer program / instruction, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0203] The above is a schematic solution of a computer program product of this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above-mentioned data processing method.

[0204] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0205] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium may be appropriately increased or decreased based on the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0206] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0207] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0208] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, applied to a database based on an LSM storage engine, comprising: Obtaining at least two historical versions of data to be processed, and data files corresponding to the at least two historical versions of data; Determining, based on a deletion mark carried by any of the at least two historical versions of data, in the data files corresponding to the at least two historical versions of data, an associated data file of the data to be processed and data association information corresponding to the associated data file, wherein the associated data file is a data file storing historical version data that can be deleted by the deletion mark, and the data association information is file information and the number of data files associated with the associated data file, where there are multiple associated data files; Creating a set of associated data files according to the plurality of associated data files of the data to be processed and data association information corresponding to each associated data file; Calculating set association information of the set of associated data files based on the data association information corresponding to each associated data file, wherein the set association information is the sum of the data association information corresponding to each associated data file included in the set of associated data files; When it is determined according to the set association information that the associated data file set meets a preset deletion condition, obtaining all data files in the data partition of the data storage to be processed; According to the data to be processed, all the data files are filtered to obtain data filtering results, and according to the data filtering results, a reference data file containing the data to be processed is obtained; In the case where it is determined that the associated data file set includes the reference data file, deletion marks and historical version data included in the plurality of associated data files are deleted.

2. The method according to claim 1, wherein determining, based on the deletion mark carried by any of the at least two historical versions of data, in the data files corresponding to the at least two historical versions of data, the associated data files of the data to be processed and the data association information corresponding to the associated data files comprises: Determining, according to a deletion mark carried by any historical version data of the at least two historical versions of data, in the data files corresponding to the at least two historical versions of data, an associated data file of the data to be processed; Determining, based on the deletion mark, an association relationship between the arbitrary historical version data and other historical version data; According to the association relationship, data association information corresponding to the associated data file is recorded.

3. The method of claim 2, wherein determining, based on a deletion mark carried by any of the at least two historical versions of data, the associated data file of the to-be-processed data in the data files corresponding to the at least two historical versions of data comprises: If it is determined that any historical version data among the at least two historical version data carries a deletion mark, determining the data file corresponding to the any historical version data as the first associated data file; determining, according to the deletion mark, a second associated data file associated with the first associated data file from the data files corresponding to the at least two historical versions of the data; The first associated data file and the second associated data file are determined as associated data files of the data to be processed.

4. The method of claim 1 , wherein, when it is determined according to the set association information that the set of associated data files satisfies a preset deletion condition, deleting the deletion marks and historical version data contained in the plurality of associated data files comprises: If it is determined that the set association information meets a preset association threshold, determining that the set of associated data files meets a preset deletion condition, and deleting the deletion marks and historical version data included in the plurality of associated data files; or When it is determined that the number of the associated data files meets a preset number threshold, it is determined that the associated data file set meets a preset deletion condition, and the deletion marks and historical version data included in the plurality of associated data files are deleted.

5. The method according to claim 1, further comprising, after deleting the deletion marks and historical version data contained in the plurality of associated data files: Obtaining the remaining data contained in the plurality of associated data files; The remaining data are merged to obtain a target data file.

6. The method according to any one of claims 1 to 5, before obtaining at least two historical versions of the data to be processed and the data files corresponding to the at least two historical versions of the data, further comprising: In response to a data query request, to-be-processed data corresponding to the data query request is determined.

7. The method according to any one of claims 1 to 5, further comprising: Determine the data partition for storing the data to be processed; When it is determined that the data storage capacity of the data partition is greater than a preset storage threshold, the data partition is split to obtain a target data partition obtained after the split.

8. The method according to claim 7, further comprising, after determining the data partition of the data storage to be processed: Determine a data table corresponding to the data to be processed, and determine the number of partitions of the data partitions stored in the data table; A preset storage threshold is calculated according to the number of partitions.

9. The method according to claim 8, wherein determining the number of partitions of the data partition stored in the data table comprises: Determining the number of partitions of the data partition stored in the data table according to a preset time interval; Calculating a preset storage threshold based on the number of partitions includes: Calculating a first storage threshold according to the preset time interval and the number of partitions; A second storage threshold calculated at a previous time interval is obtained, and a preset storage threshold is determined according to a comparison result between the first storage threshold and the second storage threshold.

10. A computing device comprising: memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 9 are implemented.

11. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

12. A computer program product comprising a computer program / instruction, which implements the steps of the method according to any one of claims 1 to 9 when executed by a processor.

Images