Data expiration processing method and related equipment for distributed systems

Abstract

This application provides a data expiration handling method for distributed systems. The method includes: creating a target coroutine for a target storage table; obtaining the target lifespan of the target storage table and the timestamps of each index in the target index table through the target coroutine; obtaining the remaining lifespan of the target data that expires earliest based on the target lifespan and the timestamps of each index; suspending the target coroutine until the target data expires based on the remaining lifespan of the target data; and initiating an expiration notification and / or deletion notification through the target coroutine when the target data expires. The technical solution of this application embodiment can immediately initiate expiration notifications and / or deletion notifications for target data through the target coroutine when the target data expires, without asynchronous scanning, thus improving the speed of initiating expiration notifications and / or deletion notifications.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and computer-readable storage medium for handling data expiration in a distributed system. Background Technology

[0002] A distributed system is a computer system composed of multiple independent computer nodes interconnected through a network to collaborate on a task or provide a service. Distributed systems, such as key-value (KV) storage systems, can provide data storage services. When data expires, the distributed system notifies the business to execute corresponding post-processing logic. However, the notification operation performed by the distributed system is slow and lacks timeliness.

[0003] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention

[0004] This application provides a method and related equipment for handling data expiration in a distributed system, in order to solve the problems existing in related technologies. The technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a data expiration handling method for a distributed system. The distributed system stores data through multiple storage tables, each storage table being associated with an index table. The index table stores an index for each data entry in the corresponding storage table, and each data entry in the same storage table has the same lifespan. The method includes:

[0006] Create a target coroutine for a target storage table, which is one of multiple storage tables and associated with a target index table;

[0007] The target coroutine is used to obtain the target lifespan of the target storage table and the timestamps of each index in the target index table.

[0008] Based on the target lifecycle and the timestamps of each index, obtain the remaining lifecycle of the target data that expires first;

[0009] Based on the remaining lifespan of the target data, suspend the target coroutine until the target data expires; and

[0010] In the event that the target data expires, an expiration notification and / or deletion notification for the target data is initiated through the target coroutine.

[0011] Secondly, embodiments of this application provide a data expiration processing device for a distributed system. The distributed system stores data through multiple storage tables, each storage table being associated with an index table. The index table stores an index for each data entry in the corresponding storage table, and each data entry in the same storage table has the same lifespan. The device includes:

[0012] The first creation module is used to create a target coroutine for a target storage table, wherein the target storage table is one of multiple storage tables and is associated with a target index table;

[0013] The first acquisition module is used to acquire the target lifecycle of the target storage table and the timestamps of each index in the target index table through the target coroutine.

[0014] The second acquisition module is used to acquire the remaining lifespan of the target data that expires first, based on the target lifespan period and the timestamps of each index.

[0015] The suspension module is used to suspend the target coroutine until the target data expires, based on the remaining lifespan of the target data; and

[0016] The first initiation module is used to initiate an expiration notification and / or deletion notification for the target data through a target coroutine when the target data expires.

[0017] Thirdly, embodiments of this application provide an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to execute the above-described data expiration processing method for a distributed system.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium that stores computer instructions, wherein when the computer instructions are executed on a computer, the methods in any of the above-described embodiments are performed.

[0019] The advantages or beneficial effects of the above technical solutions include at least the following:

[0020] A target coroutine is created for the target storage table. This coroutine retrieves the remaining time-to-live (TTL) of the earliest expiring target data. Based on the TTL, the coroutine is then suspended to wait for the data to expire. When the target data expires, the coroutine can immediately initiate expiration and / or deletion notifications without asynchronous scanning, thus improving the speed of initiating these notifications. Attached Figure Description

[0021] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0022] Figure 1 The diagram illustrates the operating environment of a data expiration handling method for a distributed system according to an embodiment of this application.

[0023] Figure 2 A flowchart illustrating a data expiration handling method for a distributed system according to an embodiment of this application is shown schematically.

[0024] Figure 3 A schematic diagram of the index table and storage table is shown.

[0025] Figure 4 The diagram illustrates a new flowchart of a data expiration handling method for a distributed system according to an embodiment of this application.

[0026] Figure 5 The diagram illustrates a new flowchart of a data expiration handling method for a distributed system according to an embodiment of this application.

[0027] Figure 6 The diagram illustrates a new flowchart of a data expiration handling method for a distributed system according to an embodiment of this application.

[0028] Figure 7 Schematic illustration Figure 2 Sub-step of step S208;

[0029] Figure 8 Schematic illustration Figure 2 Sub-step of step S208;

[0030] Figure 9 A block diagram of a data expiration processing apparatus for a distributed system according to another embodiment of this application is schematically shown; and

[0031] Figure 10 A schematic diagram of the hardware architecture of an electronic device according to another embodiment of this application is shown. Detailed Implementation

[0032] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0033] First, a definition of the terminology used in this application is provided:

[0034] TTL (Time To Live): The lifespan of data in a storage system (such as a KV storage system).

[0035] RPC (Remote Procedure Call): Allows a computer program (which can be a client) to call a program or function on another computer (which can be a server) without explicitly handling the details of network communication.

[0036] Table (Storage Table): A collection of data with the same attributes, where TTL is one of the attributes of a table.

[0037] Coroutine: A special type of function that can be suspended at a certain point and can resume execution from where it was suspended.

[0038] Suspend: This pauses the execution of a coroutine, returning control to other coroutines, the main program, or the caller, while preserving the current execution state so that execution can resume from where it was suspended later.

[0039] Raft: A consensus algorithm that can be applied to distributed systems. Through Raft, nodes in a distributed cluster can reach a consensus on certain information or states.

[0040] Key-Value Storage System: A distributed storage system that stores data in the form of key-value pairs.

[0041] Secondly, to facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are described below:

[0042] A distributed system is a computer system composed of multiple independent computer nodes. These nodes are interconnected through a network to collaborate and complete a task or provide a service. Distributed systems, such as key-value (KV) storage systems, can provide data storage services. Taking a KV storage system as an example, it stores data in the form of key-value pairs. This storage method is simple and easily scalable, making it suitable for large-scale data storage scenarios. KV storage systems feature high availability, scalability, and fault tolerance, meeting the needs of massive data storage. In real-world production scenarios, some business-stored KV records need to expire and be deleted, and there is post-processing logic after expiration. Therefore, the KV storage system needs to manage the expiration time of KV records and notify the business to perform post-processing logic upon expiration.

[0043] The inventors have learned that distributed systems (such as KV storage systems) have the following drawbacks: (1) Asynchronous thread scanning is required to determine whether the data (such as Key) lifecycle has expired. (2) Notification of data lifecycle expiration is not timely. (3) The core read / write functions of the distributed system (such as KV storage system) are coupled with the data (such as Key) expiration notification function, and their performance will affect each other.

[0044] Therefore, this application provides a data expiration handling technology solution for distributed systems. In this solution, (1) a coroutine is allocated to each storage table, and the coroutine is suspended to wait for the data (such as a key) that expires first in the storage table, without the need for asynchronous scanning. (2) Notification is given immediately after data expiration, reducing the delay caused by the time interval of asynchronous thread scanning. (3) An independent lifecycle management service system is added to decouple the lifecycle management-related functions from the core read and write functions of the distributed system (such as a KV storage system). See below for details.

[0045] like Figure 1 As shown in the diagram, the runtime environment includes: a distributed system, a lifecycle management service system, and a client. Among them:

[0046] The distributed system can be a key-value (KV) storage system. In this KV storage system, data is stored through multiple storage tables (Table-1, Table-2, ..., Table-n), each associated with an index table. The index table stores the index of each data entry in the corresponding storage table, and each data entry in the same storage table has the same lifespan. The KV storage system is also configured with a corresponding execution module, which interacts with external nodes (such as clients or lifecycle management service systems) to read and write relevant data.

[0047] Distributed systems can be coupled with lifecycle management service systems through communication interfaces (such as RPC).

[0048] The lifecycle management service system is configured with an execution module, through which a coroutine is configured for each storage table. For example, coroutine 1 is configured for Table-1, coroutine 2 for Table-2, and so on. The lifecycle management service system can also be configured with a local database to store relevant data.

[0049] The lifecycle management service system also provides notification and subscription functions. These functions facilitate interaction with the distributed system and clients.

[0050] The technical solutions of this application are described below through several embodiments. It should be understood that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein.

[0051] Figure 2 A flowchart illustrating a data expiration handling method for a distributed system according to an embodiment of this application is shown schematically.

[0052] The distributed system stores data through multiple storage tables, each associated with an index table. The index table stores an index for each data entry in the corresponding storage table, and each data entry within the same storage table has the same lifespan. Figure 2 As shown, the data expiration handling method for distributed systems may include steps S200 to S208, wherein:

[0053] Step S200: Create a target coroutine for the target storage table, which is one of multiple storage tables and associated with a target index table.

[0054] Step S202: Obtain the target lifecycle of the target storage table and the timestamps of each index in the target index table through the target coroutine.

[0055] Step S204: Based on the target lifecycle and the timestamps of each index, obtain the remaining lifecycle of the target data that expires first.

[0056] Step S206: Based on the remaining lifespan of the target data, suspend the target coroutine until the target data expires.

[0057] Step S208: If the target data expires, initiate an expiration notification and / or deletion notification for the target data through the target coroutine.

[0058] In this embodiment, a target coroutine is created for the target storage table. The remaining time-to-live (TTL) of the earliest expiring target data is obtained through the target coroutine. Then, based on the remaining TTL of the target data, the target coroutine is suspended to wait for the target data to expire. When the target data expires, the target coroutine can immediately initiate an expiration notification and / or deletion notification, eliminating the need for asynchronous scanning and improving the speed of initiating these notifications.

[0059] The following combination Figure 2 The steps in steps S200 to S208, as well as other optional steps, are described in detail.

[0060] Step S200: Create a target coroutine for the target storage table, which is one of multiple storage tables and associated with a target index table.

[0061] The storage table is a collection of multiple data sets with the same attributes, which may include: data type, lifespan, etc. The target storage table can be one of multiple storage tables, such as... Figure 3 As shown, the target storage table can be Table-1, and correspondingly, the target index table can be index table 1.

[0062] Each storage table corresponds to a coroutine, such as Figure 1 As shown, Table-1 corresponds to coroutine-1, Table-2 corresponds to coroutine-2, Table-3 corresponds to coroutine-3, and so on, with Table-n corresponding to coroutine-n.

[0063] Step S202: Obtain the target lifecycle of the target storage table and the timestamps of each index in the target index table through the target coroutine.

[0064] The lifespan refers to the period during which data is stored and retained in a distributed system. The lifespan can be set based on factors such as the importance of the data in the storage table, storage costs, and security. For example, if the data in the target storage table is important, the target lifespan can be set to a longer period, such as one year or two years; no specific limitation is made here.

[0065] The target index table corresponds to the target storage table, and each index in the index table corresponds to a piece of data in the target storage table. Through the index, specific data rows or data items in the storage table can be quickly located and accessed, thereby improving the speed and efficiency of data retrieval operations. The type of index can be selected according to the actual situation, such as: (1) Hash index: using a hash function to map data to the index position. A hash index can be represented by a hash table, which can contain the storage of key-value pairs. (2) Primary key index: The primary key index is used to uniquely identify each data record, ensuring that the values ​​in the primary key column are unique. For example, in a relational database, the primary key index of a table is used to uniquely identify each row, and the primary key index can be an auto-incrementing integer or a globally unique identifier.

[0066] The specific format of the index can be selected according to the actual situation, for example, such as... Figure 3As shown, in a KV storage system, the index format can be "TableName@Timestamp@Key", where TableName is the name of the storage table corresponding to the indexed data, Timestamp is the storage time of the data corresponding to the index, and Key is the key in the key-value pair corresponding to the data. For example, the index can be Table-1@1685760001@(key1). That is, when creating an index for data in a storage table, it is created based on the data storage time information, and the created indexes are arranged in order according to the storage time information, so that the data corresponding to the first index in the index table is the data with the earliest storage time, and then the order increases sequentially. In practical applications, the data corresponding to the first index in the index table is the data that expires earliest. By retrieving the first index in the index table, the data that expires earliest in the corresponding storage table can be obtained. It should be noted that the storage of the target data and the creation of the target index corresponding to the target data maintain atomicity, that is, both must either succeed simultaneously or fail simultaneously.

[0067] Step S204: Based on the target lifecycle and the timestamps of each index, obtain the remaining lifecycle of the target data that expires first.

[0068] The remaining lifespan refers to the period during which the data can continue to be stored in the storage table. Once the remaining lifespan is exceeded, the data will be cleaned up or deleted.

[0069] The timestamp refers to the time when the corresponding data was stored in the distributed system. Based on the target lifespan and the timestamps of each index, the remaining lifespan of the data corresponding to each index can be obtained. For example, assuming the target storage table has a target lifespan of 20 days and contains 3 data entries, the target index table has three indexes. The timestamp of index 1 (corresponding to data 1) is 0:00 on September 1, 2023; the timestamp of index 2 (corresponding to data 2) is 0:00 on September 2, 2023; and the timestamp of index 3 (corresponding to data 3) is 0:00 on September 3, 2023. The current time is 0:00 on September 15, 2023. Therefore, using the target goroutine, the remaining lifespan of data 1 can be calculated to be 5 days, data 2 to be 6 days, and data 3 to be 7 days. Thus, data 1 is the first data to expire, meaning data 1 is the first target data to expire, and its remaining lifespan is 5 days.

[0070] Step S206: Based on the remaining lifespan of the target data, suspend the target coroutine until the target data expires.

[0071] If the remaining time to live of the target data is greater than 0, meaning the target data has not yet expired, the target coroutine will be suspended until it expires. For example, if data 1 is the target data and its remaining time to live is 5 days, the target coroutine will be suspended until 5 days later, when data 1 expires, at which point the target coroutine will restart.

[0072] Step S208: If the target data expires, initiate an expiration notification and / or deletion notification for the target data through the target coroutine.

[0073] The expiration notice can be a notification used to remind relevant parties that the target data has reached a predetermined end time. By issuing an expiration notice to the corresponding recipients, the recipients can take appropriate measures regarding the target data or related data.

[0074] The deletion notification can indicate that target data has reached the end of its lifecycle and can be deleted or cleaned up. Specifically, a deletion notification can be published to a distributed system. After receiving the deletion notification, the distributed system performs a deletion operation on the target data or related data.

[0075] The lifespan of data in a distributed system can be adjusted in some cases. How to handle this adjusted data will be discussed later.

[0076] In one embodiment of this application, as Figure 4 As shown, the method may further include:

[0077] If the target lifespan of the target storage table increases, perform the following operations:

[0078] Step 400: Destroy the target goroutine.

[0079] Step 402: Create a new target coroutine for the target storage table.

[0080] Step 404: Obtain the target lifecycle after the target storage table is enlarged and the timestamps of each index in the target index table through the new target coroutine.

[0081] Step 406: Based on the enlarged target lifespan and the timestamps of each index, obtain the remaining lifespan of the target data that expires first.

[0082] Step 408: Based on the remaining lifespan of the target data, suspend the new target coroutine until the target data expires.

[0083] Step 410: If the target data expires, initiate an expiration notification and / or deletion notification for the target data through the new target coroutine.

[0084] For example, assume the target storage table is Table-1, the corresponding target index table is Index Table 1, and the target coroutine is coroutine-1. When the target lifespan of the target storage table increases, such as from 20 days to 25 days, the remaining lifespan of each piece of data in the target storage table will correspondingly increase. Therefore, it is necessary to recalculate the remaining lifespan of each piece of data in the target storage table in order to correctly initiate expiration and / or deletion notifications for the target data. Specifically: the target coroutine (coroutine-1) can be destroyed, and a new target coroutine (new coroutine-1') can be created for Table-1. Through the new coroutine-1', the increased target lifespan of Table-1 and the timestamps of each index in Index Table 1 are obtained. Based on the increased target lifespan and the timestamps of each index, the remaining lifespan of the target data that expires first is obtained. Based on the remaining lifespan of the target data, the new coroutine-1' is suspended until the target data expires. In the case of target data expiration, the expiration and / or deletion notifications for the target data are initiated through the new coroutine-1'.

[0085] In the above implementation, when the target lifespan of the target storage table increases, the target coroutine is destroyed and a new target coroutine is created for the target storage table. The remaining lifespan of the earliest expiring target data is recalculated using the new target coroutine, and expiration and / or deletion notifications for the target data can then be initiated through the new target coroutine. The advantage of this approach is that it ensures that the lifecycle management of data in the target storage table can be performed correctly and effectively when the target lifespan of the target storage table increases.

[0086] In one embodiment of this application, as Figure 5 As shown, the method may further include:

[0087] If the target lifespan of the target storage table becomes shorter, perform the following operations:

[0088] Step 500: Destroy the target goroutine.

[0089] Step 502: Create a new target coroutine for the target storage table.

[0090] Step 504: Obtain the reduced target lifespan and the survival time of each data entry in the target storage table through the new target coroutine.

[0091] Step 506: Based on the reduced target lifespan and the lifespan of each data entry, determine the expired data. The expired data is the data in the target storage table whose lifespan is greater than the reduced target lifespan.

[0092] Step 508: Initiate batch expiration notifications and / or batch deletion notifications for the expired data through the new target coroutine.

[0093] For example, when the target lifespan of the target storage table decreases, such as from 20 days to 15 days, the lifespan of each piece of data in the target storage table will decrease accordingly. At this point, some data in the target storage table may have already lived for more than 15 days (the reduced lifespan), while some data may have lived for less than 15 days. For data that has lived for more than 15 days, this data has expired and cannot be retained in the target storage table. Therefore, it is necessary to determine the expired data based on the reduced target lifespan and the lifespan of each piece of data. Based on the determination of the expired data, a batch expiration notification and / or batch deletion notification for the expired data is initiated through a new target coroutine.

[0094] In the above implementation, batch processing of expired data (expired data) caused by the shortening of the target lifespan of the target storage table can promptly delete these expired data to improve the processing efficiency of expired data, thereby ensuring the data quality of the target storage table and ensuring that all data in the target data table is non-expired data.

[0095] In one embodiment of this application, as Figure 6 As shown, the method may further include:

[0096] Step 600: Obtain the timestamps of each index in the target index table through the new target coroutine.

[0097] Step 602: Based on the reduced target lifespan and the timestamps of each index, obtain the remaining lifespan of the data that expires first among the unexpired data. The unexpired data refers to the data in the target storage table whose lifespan is less than the reduced target lifespan.

[0098] Step 604: Based on the remaining lifespan of the first expiring data among the unexpired data, suspend the new target coroutine until the expiring data expires.

[0099] Step 606: If the data to be expired is due, initiate an expiration notification and / or deletion notification for the data to be expired through the new target coroutine.

[0100] For example, when the target lifespan of the target storage table decreases, such as from 20 days to 15 days, the remaining lifespan of each piece of data in the target storage table will correspondingly decrease. For data whose lifespan is less than 15 days, the remaining lifespan of the earliest expiring data among the unexpired data needs to be recalculated through a new target goroutine. Upon the expiration of this expiring data, an expiration notification and / or deletion notification for the target data is initiated through the new target goroutine.

[0101] In the above implementation, when the target lifespan of the target storage table decreases, the remaining lifespan of the data to be expired is recalculated through a new target goroutine, and expiration notifications and / or deletion notifications for the data to be expired are initiated based on the remaining lifespan of the data to be expired. The advantage of this is that even when the target lifespan of the target storage table decreases, lifecycle management of the unexpired data in the target storage table can be performed correctly and effectively, that is, expiration notifications and / or deletion notifications for the unexpired data in the target storage table can be initiated correctly and effectively.

[0102] In one embodiment of this application, the method is applied to a lifecycle management service system, which is coupled to the distributed system through a communication interface. It should be noted that the communication interface can be selected according to actual needs, such as an RPC (Remote Procedure Call) interface, a RESTful (Representational State Transfer Application Programming Interface) interface, or a GRPC (Google Remote Procedure Call) interface.

[0103] In the above implementation, an independent lifecycle management service system is added to decouple the lifecycle management service system from the distributed system. This decouples the data lifecycle management function from the core functions of the distributed system (such as read and write functions), mitigating the performance degradation caused by the coupling between the two functions and thereby improving the efficiency of executing the corresponding operations of the lifecycle management function and the core functions of the distributed system (such as read and write functions).

[0104] In one embodiment of this application, the lifecycle management service system may include a master node and multiple slave nodes. Wherein:

[0105] The master node is used to create coroutines to obtain information about each storage table and to manage the lifecycle of data based on the information about each storage table. The storage table information includes the lifecycle and the timestamps of each index in the corresponding index table.

[0106] The slave node is used to synchronize the information of each storage table obtained by the master node.

[0107] The method further includes performing the following operations in the event of a master node switchover or restart:

[0108] By switching or restarting the master node, a new coroutine is created for each storage table based on the information of each storage table.

[0109] In practical applications, lifecycle management service systems can achieve high availability through the Raft consensus algorithm. Specifically, the Raft consensus algorithm enables slave nodes to synchronize the various storage table information obtained from the master node, maintaining the consistency of storage table information across nodes. This allows for the creation of corresponding coroutines based on the storage table information in the nodes. Furthermore, if the master node in the lifecycle management service system fails due to a fault or other reasons, the Raft algorithm can trigger a new round of election to select a new master node, enabling the lifecycle management service system to continue operating.

[0110] When the master node restarts, the coroutines within it are destroyed. Therefore, new coroutines need to be created for each storage table based on the information of each storage table in the master node, such as creating a target coroutine for the target storage table. When a master node switch occurs, i.e., a slave node in the node group is designated as the master node, coroutines are only set on the master node. When a slave node is first designated as the master node, no corresponding coroutines are set on that node. Therefore, new coroutines need to be created for each storage table. After creating new coroutines for each storage table, the target storage table is now associated with the new target coroutine. Through the new target coroutine, the target lifespan of the target storage table and the timestamps of each index in the target index table can be obtained. Based on the target lifespan and the timestamps of each index, the remaining lifespan of the target data that expires first is obtained. Based on the remaining lifespan of the target data, the new target coroutine is suspended until the target data expires. When the target data expires, an expiration notification and / or deletion notification for the target data is initiated through the new target coroutine.

[0111] In the above implementation, when the master node is switched or restarted, a new coroutine is created for each storage table through the new master node after the switch or restart, so that the data in the storage table can be managed for lifecycle according to the new coroutine. This ensures that the lifecycle management service system can still correctly and effectively manage the data in each storage table for lifecycle when the master node is switched or restarted.

[0112] In one embodiment of this application, the lifecycle management service system is further configured to provide a client subscription service. Correspondingly, step 208 includes:

[0113] In the event that the target data expires, the expiration notification of the target data is sent to the target client via the target coroutine, wherein the target client is a client that has subscribed to the expiration notification of the target storage table.

[0114] In the above implementation, the target coroutine sends an expiration notification to the target client to notify the target client that the target data has reached the end of its predetermined lifespan, thereby preventing the target client from requesting the expired data and reducing unnecessary network traffic.

[0115] In one embodiment of this application, the deletion notification is used to instruct the simultaneous deletion of target data and target index. For example... Figure 7 As shown, step 208 may further include:

[0116] Step 700: Initiate a deletion notification for the target data through the target coroutine to delete the target data and the target index.

[0117] Step 702: If the deletion of the target data and the target index fails, the deletion notification is sent again through the target coroutine.

[0118] In an exemplary application, a failure in the distributed system, such as a network problem or storage media failure, may cause the initial deletion operation of the target data and target index to fail. Therefore, to delete the target data and target index, a second deletion notification can be sent via the target goroutine to perform a second deletion of the target data and target index.

[0119] In the above implementation, if the deletion of target data and target index fails, a deletion notification is sent again to delete the target data and target index again. This ensures that if the deletion of target data and target index fails the first time, the deletion operation can be performed again, thereby increasing the probability that expired target data and target index will be deleted. This allows for timely cleanup of data that is no longer needed, freeing up storage resources, saving storage costs, and ensuring the quality of data in the target storage table.

[0120] In one embodiment of this application, as Figure 8 As shown, step 208 may further include:

[0121] Step 800: If the deletion of the target data and the target index fails again, the target data and the target index are recorded in the local database.

[0122] Step 802: Periodically retrieve records of the target data and the target index from the local database to establish an offline deletion task, so as to delete the target data and the target index from the target storage table through the offline deletion task.

[0123] In this exemplary application, assume the target data is data 1 and the target index is index 1. When data 1 expires and the deletion of data 1 and index 1 fails again (for the second time), data 1 and index 1 are recorded in the local database of the lifecycle management service system. After recording, the records of data 1 and index 1 are retrieved according to a predetermined time schedule or interval. For example, the records in the local database can be retrieved at 00:00 every day, and an offline task can be created based on the records to delete data 1 and index 1.

[0124] In some embodiments, when the master node is switched or restarted, the new coroutine can filter out expired but failed-to-delete data in each storage table based on the records in the local database when re-acquiring the information of each storage table, so as to avoid repeatedly sending expiration notifications and / or deletion notifications to expired but failed-to-delete data in the future.

[0125] In the above implementation, the target data and target index that failed to be deleted again are recorded in the local database. The target data and target index are then deleted using an offline task to ensure they are completely removed. It can be seen that periodically deleting and cleaning up expired data through offline tasks can free up storage resources, save storage costs, ensure that there is no expired data in the target storage table, and guarantee the data quality in the target storage table.

[0126] In some embodiments, the target data may include multiple data items with the same storage time, where the storage time is the time the data was stored in the distributed system. Correspondingly, the timestamps of the multiple indexes corresponding to the multiple data items with the same storage time are the same. In an exemplary application, if there are multiple data items with the same storage time in the target storage table, for example, if there are three data items (data 1, data 2, and data 3) with the same storage time in the target storage table Table-1, and the storage time of all three data items is 0:00 on September 1, 2023. Since each data item in the target storage table has the same lifespan (the target lifespan), the multiple data items (data 1, data 2, and data 3) will expire at the same time. In order to correctly and quickly manage the lifecycle of the multiple data items, when they expire, a batch expiration notification and / or batch deletion notification can be initiated through the target coroutine to process the multiple data items in batches, that is, a batch expiration notification and / or batch deletion notification can be initiated through the target coroutine for data 1, data 2, and data 3.

[0127] Figure 9 The diagram schematically illustrates a block diagram of a data expiration processing apparatus for a distributed system according to an embodiment of this application. This apparatus can be divided into one or more program modules, which are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program module referred to in the embodiments of this application refers to a series of computer program instruction segments capable of performing a specific function. The following description will specifically introduce the functions of each program module in this embodiment. Figure 9 As shown, the distributed system stores data through multiple storage tables, each associated with an index table. The index table stores the index of each data entry in the corresponding storage table. Each data entry in the same storage table has the same lifespan. The data expiration processing device 900 for the distributed system may include: a first creation module 910, a first acquisition module 920, a second acquisition module 930, a suspension module 940, and a first initiation module 950, wherein:

[0128] The first creation module 910 is used to create a target coroutine for a target storage table, wherein the target storage table is one of a plurality of storage tables and is associated with a target index table;

[0129] The first acquisition module 920 is used to acquire the target lifecycle of the target storage table and the timestamps of each index in the target index table through the target coroutine.

[0130] The second acquisition module 930 is used to acquire the remaining lifespan of the target data that expires first, based on the target lifespan period and the timestamps of each index.

[0131] Suspension module 940 is used to suspend the target coroutine until the target data expires, based on the remaining lifespan of the target data; and

[0132] The first initiation module 950 is used to initiate an expiration notification and / or deletion notification of the target data through a target coroutine when the target data expires.

[0133] In one embodiment of this application, the data expiration processing device for a distributed system may further include a second initiating module, the second initiating module being used for:

[0134] If the target lifespan of the target storage table increases, perform the following operations:

[0135] Destroy the target goroutine;

[0136] Create a new target coroutine for the target storage table;

[0137] The new target coroutine is used to obtain the target lifecycle after the target storage table is enlarged and the timestamps of each index in the target index table.

[0138] Based on the enlarged target lifespan and the timestamps of each index, obtain the remaining lifespan of the target data that expires first;

[0139] Based on the remaining lifespan of the target data, suspend the new target coroutine until the target data expires; and

[0140] If the target data expires, the expiration notification and / or deletion notification of the target data shall be initiated through the new target coroutine.

[0141] In one embodiment of this application, the data expiration processing device for a distributed system may further include the third initiating module, the third initiating module being used for:

[0142] If the target lifespan of the target storage table becomes shorter, perform the following operations:

[0143] Destroy the target goroutine;

[0144] Create a new target coroutine for the target storage table;

[0145] The reduced target lifespan and the survival time of each data entry in the target storage table are obtained through the new target coroutine.

[0146] Based on the reduced target lifespan and the existing lifespan of each data entry, expired data is determined. The expired data is data in the target storage table whose existing lifespan is greater than the reduced target lifespan.

[0147] The new target coroutine initiates batch expiration notifications and / or batch deletion notifications for the expired data.

[0148] In one embodiment of this application, the data expiration processing device for a distributed system may further include the fourth initiation module, which is used for:

[0149] The timestamps of each index in the target index table are obtained through the new target coroutine;

[0150] Based on the reduced target lifespan and the timestamps of each index, the remaining lifespan of the data that expires first among the unexpired data is obtained. The unexpired data refers to the data in the target storage table whose lifespan is less than the reduced target lifespan.

[0151] Based on the remaining lifespan of the earliest expiring data among the unexpired data, suspend the new target coroutine until that expiring data expires; and

[0152] If the data to be expired is due, the expiration notification and / or deletion notification for the data to be expired will be initiated through the new target coroutine.

[0153] In one embodiment of this application, the method is applied to a lifecycle management service system, which is coupled to the distributed system through a communication interface.

[0154] In one embodiment of this application, the lifecycle management service system includes a master node and multiple slave nodes; wherein:

[0155] The master node is used to create coroutines to obtain information about each storage table and to manage the lifecycle of data based on the information about each storage table. The storage table information includes the lifecycle and the timestamps of each index in the corresponding index table.

[0156] The slave node is used to synchronize the information of each storage table obtained by the master node;

[0157] The data expiration processing device for distributed systems may further include a second creation module, wherein the second initiation module is used for:

[0158] In the event of a master node switchover or restart, perform the following operations:

[0159] By switching or restarting the master node, a new coroutine is created for each storage table based on the information of each storage table.

[0160] In an optional embodiment, the lifecycle management service system is further configured to provide a client subscription service;

[0161] Correspondingly, the first initiating module 950 is also used for:

[0162] In the event that the target data expires, the expiration notification of the target data is sent to the target client via the target coroutine, wherein the target client is a client that has subscribed to the expiration notification of the target storage table.

[0163] In one embodiment of this application, the deletion notification is used to instruct the simultaneous deletion of target data and target index;

[0164] Correspondingly, the first initiating module 950 is also used for:

[0165] The target coroutine initiates a deletion notification for the target data to delete the target data and the target index.

[0166] If the deletion of the target data and the target index fails, the deletion notification will be sent again through the target coroutine.

[0167] In one embodiment of this application, the data expiration processing device for a distributed system may further include the pull module, the pull module being used for:

[0168] If the deletion of the target data and the target index fails again, the target data and the target index will be recorded in the local database.

[0169] Records of the target data and the target index are periodically retrieved from the local database to create an offline deletion task, which is then used to delete the target data and the target index from the target storage table.

[0170] Figure 10 A structural block diagram of an electronic device according to an embodiment of this application is shown. Figure 10 As shown, the electronic device includes a memory 1010 and a processor 1020. The memory 1010 stores instructions that can be executed on the processor 1020. When the processor 1020 executes the instructions, it implements the data expiration handling method for a distributed system described in the above embodiments. The number of memories 1010 and processors 1020 can be one or more. This electronic device is intended to represent various forms of digital computers, such as workbenches, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0171] The electronic device may also include a communication interface 1030 for communicating with external devices and exchanging data. The devices are interconnected using different buses and can be mounted on a common motherboard or otherwise as needed. The processor 1020 can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). The bus can be divided into address buses, data buses, control buses, etc. For ease of illustration, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0172] Optionally, in a specific implementation, if the memory 1010, processor 1020 and communication interface 1030 are integrated on a single chip, then the memory 1010, processor 1020 and communication interface 1030 can communicate with each other through an internal interface.

[0173] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting the Advanced Reduced Instruction Set Computing (RISC) machine (ARM) architecture.

[0174] One embodiment of this application provides a computer-readable storage medium (such as the memory 1010 described above) that stores computer instructions that, when executed by a processor, implement the methods provided in the embodiments of this application.

[0175] Optionally, memory 1010 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device for data expiration processing in a distributed system. Furthermore, memory 1010 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 1010 may optionally include memory remotely located relative to processor 1020, and these remote memories may be connected via a network to the electronic device for data expiration processing in a distributed system. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0176] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0177] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0178] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more (two or more) executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0179] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0180] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0181] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0182] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for handling data expiration in a distributed system, characterized in that, The distributed system stores data through multiple storage tables, each associated with an index table. The index table stores the index of each data entry in the corresponding storage table, and each data entry in the same storage table has the same lifespan. The method includes: Create a target coroutine for a target storage table, which is one of multiple storage tables and associated with a target index table; The target coroutine is used to obtain the target lifespan of the target storage table and the timestamps of each index in the target index table. Based on the target lifecycle and the timestamps of each index, obtain the remaining lifecycle of the target data that expires first; Based on the remaining lifespan of the target data, suspend the target coroutine until the target data expires; and In the event that the target data expires, an expiration notification and / or deletion notification for the target data will be initiated through the target coroutine. This also includes: If the target lifespan of the target storage table increases, perform the following operations: Destroy the target goroutine; Create a new target coroutine for the target storage table; The new target coroutine is used to obtain the target lifecycle after the target storage table is enlarged and the timestamps of each index in the target index table; Based on the enlarged target lifespan and the timestamps of each index, obtain the remaining lifespan of the target data that expires first; Based on the remaining lifespan of the target data, suspend the new target coroutine until the target data expires; and If the target data expires, the expiration notification and / or deletion notification of the target data shall be initiated through the new target coroutine.

2. The method according to claim 1, characterized in that, The method further includes: If the target lifespan of the target storage table becomes shorter, perform the following operations: Destroy the target goroutine; Create a new target coroutine for the target storage table; The reduced target lifespan and the survival time of each data entry in the target storage table are obtained through the new target coroutine. Based on the reduced target lifespan and the existing lifespan of each data entry, expired data is determined. The expired data is data in the target storage table whose existing lifespan is greater than the reduced target lifespan. The new target coroutine initiates batch expiration notifications and / or batch deletion notifications for the expired data.

3. The method according to claim 2, characterized in that, The method further includes: The timestamps of each index in the target index table are obtained through the new target coroutine; Based on the reduced target lifespan and the timestamps of each index, the remaining lifespan of the data that expires first among the unexpired data is obtained. The unexpired data refers to the data in the target storage table whose lifespan is less than the reduced target lifespan. Based on the remaining lifespan of the earliest expiring data among the unexpired data, suspend the new target coroutine until that expiring data expires; and If the data to be expired is due, the expiration notification and / or deletion notification for the data to be expired will be initiated through the new target coroutine.

4. The method according to any one of claims 1 to 3, characterized in that, The method is applied to a lifecycle management service system, which is coupled to the distributed system through a communication interface.

5. The method according to claim 4, characterized in that, The lifecycle management service system includes a master node and multiple slave nodes; among which: The master node is used to create coroutines to obtain information about each storage table and to manage the lifecycle of data based on the information about each storage table. The storage table information includes the lifecycle and the timestamps of each index in the corresponding index table. The slave node is used to synchronize the information of each storage table obtained by the master node; The method further includes performing the following operations in the event of a master node switchover or restart: By switching or restarting the master node, a new coroutine is created for each storage table based on the information of each storage table.

6. The method according to claim 4, characterized in that, The lifecycle management service system is also used to provide client subscription services; Correspondingly, in the event that the target data expires, an expiration notification and / or deletion notification for the target data is initiated through the target coroutine, including: In the event that the target data expires, the expiration notification of the target data is sent to the target client via the target coroutine, wherein the target client is a client that has subscribed to the expiration notification of the target storage table.

7. The method according to any one of claims 1 to 3, characterized in that, The deletion notification is used to instruct the simultaneous deletion of the target data and the target index; Correspondingly, in the event that the target data expires, an expiration notification and / or deletion notification for the target data is initiated through the target coroutine, including: The target coroutine initiates a deletion notification for the target data to delete the target data and the target index. If the deletion of the target data and the target index fails, the deletion notification will be sent again through the target coroutine.

8. The method according to claim 7, characterized in that, Also includes: If the deletion of the target data and the target index fails again, the target data and the target index will be recorded in the local database. Records of the target data and the target index are periodically retrieved from the local database to create an offline deletion task, which is then used to delete the target data and the target index from the target storage table.

9. A data expiration processing device for a distributed system, characterized in that, The device includes: The first creation module is used to create a target coroutine for a target storage table, wherein the target storage table is one of multiple storage tables and is associated with a target index table; The first acquisition module is used to acquire the target lifecycle of the target storage table and the timestamps of each index in the target index table through the target coroutine. The second acquisition module is used to acquire the remaining lifespan of the target data that expires first, based on the target lifespan period and the timestamps of each index. The suspension module is used to suspend the target coroutine until the target data expires, based on the remaining lifespan of the target data; and The first initiation module is used to initiate an expiration notification and / or deletion notification of the target data through a target coroutine when the target data expires. This also includes: If the target lifespan of the target storage table increases, perform the following operations: Destroy the target goroutine; Create a new target coroutine for the target storage table; The new target coroutine is used to obtain the target lifecycle after the target storage table is enlarged and the timestamps of each index in the target index table; Based on the enlarged target lifespan and the timestamps of each index, obtain the remaining lifespan of the target data that expires first; Based on the remaining lifespan of the target data, suspend the new target coroutine until the target data expires; and If the target data expires, the expiration notification and / or deletion notification of the target data shall be initiated through the new target coroutine.

10. An electronic device, characterized in that, include: At least one processor; and A memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 8.

Images