A method and device for regularly cleaning out expired data in a time series database
By using the retention keyword in the timing database to define the data retention period and periodically scan the schedule system table through the background coroutine to trigger the data cleaning task, the problem of data expiration in the timing database is solved, and automated data cleaning and effective disk space management are achieved.
Patent Information
- Application Number
- CN202411834067.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-04-04
AI Technical Summary
The data value in the timing database decreases with time, resulting in a huge amount of data, making it difficult for users to manually delete useless timing data to free up disk space.
Define the retention period of data by introducing the retention keyword in the timing database, and check whether there are lifecycle-related tasks in the schedule system table when the database service starts, and create a default lifecycle task if it does not exist. Then start the background coroutine to scan the schedule system table regularly to obtain the data cleaning task to be executed, and trigger the operation to clean up expired data when the current time exceeds the scheduled execution time of the task.
It realizes the function of automatically cleaning out expiration data in a time-series database, simplifies user management and maintenance costs, saves disk usage space, and improves user efficiency.
Smart Images
Figure CN119759885A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of time series databases, and in particular to a method and device for periodically cleaning out expired data in a time series database. Background Art
[0002] The characteristic of time series data is that there will be a huge amount of data, but the value of the data will decrease with the increase of time. When using the database, it is not convenient for users to manually delete useless time series data again and again to free up disk space. At this time, the database needs to be able to regularly clear out expired and unnecessary time series data to free up disk space. Summary of the invention
[0003] The present invention provides a method and device for periodically clearing expired time series data in a time series database so as to enable a user to conveniently clear expired time series data and release disk space when using a time series database.
[0004] In the first aspect, a method for periodically cleaning up expired data in a time series database of the present invention adopts the following technical solution to solve the above technical problem:
[0005] A method for periodically cleaning up expired data in a time series database comprises the following steps:
[0006] S1. When a user creates a time series database or time series table, he / she defines the data retention period by specifying the retention keyword;
[0007] S2. When the database service is started, first check whether there are lifecycle-related tasks in the schedule system table. If so, directly execute the subsequent steps. If not, create a default lifecycle task and then execute the subsequent steps.
[0008] S3. The database service starts a background coroutine to regularly scan the schedule system table to obtain all data cleanup tasks to be executed, and then triggers the operation of cleaning up expired data when the current time exceeds the scheduled execution time of the task;
[0009] S4. When the background coroutine triggers the operation of clearing expired data, a background job is created. The background job obtains the life cycle information of all time series tables and sends it to the storage system. The storage system receives the life cycle information, compares the timestamps, identifies the time series data whose timestamps exceed the life cycle range, and automatically clears them.
[0010] S5. After the automatic clearing of expired data is completed, the next execution time in the schedule system table is updated, and the data clearing task of this cycle is marked as completed.
[0011] Optionally, execute step S1. If the user sets the data retention period of the time series database and also sets the data retention period of the time series table under the time series database, the data retention period of the time series table has a higher priority than the data retention period of the time series database to which the time series table belongs; if the user does not set the data retention period of the time series table when creating the time series table, the time series table inherits the data retention period of the time series database to which it belongs.
[0012] Optionally, execute step S2, the schedule system table involved is used to save all scheduling tasks, including data cleaning tasks related to the life cycle; the schedule system table involved records ten fields: schedule id, name, creator, owner, next execution time, status, cron expression, detail, executor name and executor parameters, among which schedule id is used as the primary key of the schedule system table, the next execution time is used as the index field, and the cron expression is used to define the execution time of the task. The user controls the time period for executing the data cleaning task by modifying the cron expression;
[0013] When there is no lifecycle-related task in the schedule system table involved, the default lifecycle task created contains a predefined data cleanup strategy.
[0014] Further optionally, the Detail of the schedule system table record involved is used to describe the parameters and behavior of the task in detail;
[0015] The user can modify the details of the schedule system table through the alter statement to change the execution mode of the data cleanup task in different situations. The details include the failure handling strategy and the handling mode of the previous task execution:
[0016] (1) For the failure handling strategy, the following three measures are taken: a) immediately try to re-execute the task, b) try to execute the task again at the next scheduling, c) suspend the execution of the task until yoghurt manually intervenes;
[0017] (2) Regarding the handling method of the previous task execution, the following three measures are taken: a) Regardless of whether the previous task is still running, start the current task immediately; b) If the previous task is still running, skip the current execution and wait for the next scheduling; c) If the previous task is still running, wait for it to complete before starting the current task.
[0018] Optionally, after executing step S5, the user can view the historical task execution results of clearing expired data and the execution status of the current task by using the show job command.
[0019] In the first aspect, a device for periodically cleaning up expired data in a time series database of the present invention adopts the following technical solutions to solve the above technical problems:
[0020] A device for regularly cleaning up expired data in a time series database, comprising:
[0021] Create a definition module to assist users in creating a time series database or time series table, and define the data retention period by specifying the retention keyword;
[0022] The check processing module is used to check whether there are life cycle related tasks in the schedule system table when the database service is started. If so, the scan trigger module is directly called. If not, a default life cycle task is created and then the scan trigger module is called.
[0023] The scan trigger module is used to start a background coroutine, which scans the schedule system table to obtain all data cleaning tasks to be executed, and triggers the operation of cleaning expired data when the current time exceeds the scheduled execution time of the task;
[0024] The job creation module is used to create a background job when the background coroutine triggers the operation of clearing expired data. The background job obtains the life cycle information of all time series tables and sends it to the storage system;
[0025] The comparison processing module is used to receive life cycle information, compare timestamps, identify time series data whose timestamps exceed the life cycle range, and automatically clear them;
[0026] The time update module is used to update the next execution time in the schedule system table after the comparison processing module completes the automatic clearing of expired data, and mark the data clearing task of this cycle as completed.
[0027] Optionally, when the user creates a time series database and time series table by creating a definition module:
[0028] If the user sets the data retention period for a time series database and also sets the data retention period for a time series table under the time series database, the data retention period for the time series table takes precedence over the data retention period for the time series database to which the time series table belongs.
[0029] If the user does not set the data retention period for a time series table when creating it, the time series table inherits the data retention period of the time series database to which it belongs.
[0030] Optionally, the schedule system table is used to save all scheduling tasks, including data cleanup tasks related to the life cycle; the schedule system table records ten fields: schedule id, name, creator, owner, next execution time, status, cron expression, detail, executor name, and executor parameters. The schedule id is used as the primary key of the schedule system table, the next execution time is used as the index field, and the cron expression is used to define the execution time of the task. Users can control the time period for executing data cleanup tasks by modifying the cron expression.
[0031] When there is no lifecycle-related task in the schedule system table involved, the default lifecycle task created by the inspection processing module includes a predefined data cleanup strategy.
[0032] Further optionally, the Detail of the schedule system table record involved is used to describe the parameters and behavior of the task in detail;
[0033] The user can modify the details of the schedule system table through the alter statement to change the execution mode of the data cleanup task in different situations. The details include the failure handling strategy and the handling mode of the previous task execution:
[0034] (1) For the failure handling strategy, the following three measures are taken: a) immediately try to re-execute the task, b) try to execute the task again at the next scheduling, c) suspend the execution of the task until yoghurt manually intervenes;
[0035] (2) Regarding the handling method of the previous task execution, the following three measures are taken: a) Regardless of whether the previous task is still running, start the current task immediately; b) If the previous task is still running, skip the current execution and wait for the next scheduling; c) If the previous task is still running, wait for it to complete before starting the current task.
[0036] Optionally, the device for periodically cleaning up expired data in a time series database further includes a task viewing module;
[0037] The user enters the show job command in the task viewing module to view the historical task execution results of clearing expired data and the execution status of the current task.
[0038] Compared with the prior art, the method and device for regularly cleaning up expired data in a time series database of the present invention have the following beneficial effects:
[0039] 1. When the user creates a time series database and a time series table, the present invention can define the retention period of the data by specifying the retention keyword; by starting a background coroutine to regularly scan the schedule system table, the data cleaning task to be executed can be obtained, so as to trigger the operation of cleaning up expired data when the current time exceeds the scheduled execution time of the task, and at the same time, the life cycle information of all time series tables is obtained through the background job, and sent to the storage system to identify the time series data whose timestamp exceeds the life cycle range and automatically clear it; these operations can greatly simplify the user's management and maintenance costs, save disk usage space, and improve user efficiency;
[0040] 2. The present invention can very conveniently help users clean up expired time series data and release disk space without the need for users to manually delete data; it can also facilitate users to modify the interval for deleting data as needed, as well as the parameters when deletion fails or the deletion task is being executed, so that users can use it flexibly; it can facilitate users to view the current execution task status or historical execution records through the show statement, helping users to better manage and control the life cycle of time series data, and when using distributed multi-nodes, it can avoid conflicts caused by direct background execution of each node. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Attached Figure 1 is a flow chart of a method according to Embodiment 1 of the present invention;
[0042] Attached Figure 2 This is a module connection block diagram of the second embodiment of the present invention. DETAILED DESCRIPTION
[0043] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.
[0044] Embodiment 1:
[0045] Combined with Figure 1 This embodiment proposes a method for regularly cleaning up expired data in a time series database, which includes the following steps:
[0046] S1. When a user creates a time series database or time series table, he / she defines the data retention period by specifying the retention keyword.
[0047] If the user sets the data retention period for a time series database and also sets the data retention period for a time series table under the time series database, the data retention period of the time series table takes precedence over the data retention period of the time series database to which the time series table belongs. If the user does not set the data retention period for a time series table when creating it, the time series table inherits the data retention period of the time series database to which it belongs.
[0048] S2. When the database service is started, first check whether there are lifecycle-related tasks in the schedule system table. If so, directly execute the subsequent steps. If not, create a default lifecycle task and then execute the subsequent steps.
[0049] The schedule system table is used to save all scheduling tasks, including lifecycle-related data cleanup tasks. The schedule system table records ten fields: schedule ID, name, creator, owner, next execution time, status, cron expression, detail, executor name, and executor parameters. The Schedule ID field is used as the primary key of the system table to ensure the uniqueness of each task.
[0050] The Name field is used as an identification and reference for the task;
[0051] The "Creator" field refers to the creator of the task, which is usually the database administrator or a user with corresponding permissions;
[0052] The "Owner" field refers to the owner of the current task, which may be an individual or a group;
[0053] The "Next Execution Time" field refers to the time when the task will be executed next time. This field is indexed for quick access and comparison;
[0054] "Status" reflects the current status of the task, such as waiting, executing, or completed;
[0055] The "Cron Expression" field is used to define the time period for task execution. It uses standard cron syntax. Users can modify the cron expression to control the time period for executing data cleanup tasks.
[0056] The "Detail" field is used to describe the parameters and behavior of the task in detail. Users can modify the details of the schedule system table through the alter statement to modify the execution mode of the data cleanup task in different situations. The details include the failure handling strategy and the handling mode of the previous task execution:
[0057] (1) For the failure handling strategy, the following three measures are taken: a) immediately try to re-execute the task, b) try to execute the task again at the next scheduling, c) suspend the execution of the task until yoghurt manually intervenes;
[0058] (2) For the processing method of the previous task execution, the following three measures are taken: a) regardless of whether the previous task is still running, the current task is started immediately; b) if the previous task is still running, skip the current execution and wait for the next scheduling; c) if the previous task is still running, wait for it to complete before starting the current task by default;
[0059] The "Executor Name" field is used to specify the specific service or program that performs the task;
[0060] The "Executor's parameters" field is used for parameters passed to the executor to customize the execution of the task.
[0061] When there are no lifecycle-related tasks in the schedule system table, the default lifecycle task created contains a predefined data cleanup strategy.
[0062] S3. The database service starts a background coroutine to regularly scan the schedule system table to obtain all data cleanup tasks to be executed, and then triggers the operation of cleaning up expired data when the current time exceeds the scheduled execution time of the task;
[0063] S4. When the background coroutine triggers the operation of clearing expired data, a background job is created. The background job obtains the life cycle information of all time series tables and sends it to the storage system. The storage system receives the life cycle information, compares the timestamps, identifies the time series data whose timestamps exceed the life cycle range, and automatically clears them.
[0064] S5. After the automatic clearing of expired data is completed, the next execution time in the schedule system table is updated, and the data clearing task of this cycle is marked as completed.
[0065] It should be added that users can use the show job command to view the historical task execution results and the current task execution status of clearing expired data.
[0066] Embodiment 2:
[0067] Combined with Figure 2 This embodiment provides a device for periodically cleaning up expired data in a time series database, which includes:
[0068] Create a definition module to assist users in creating a time series database or time series table, and define the data retention period by specifying the retention keyword;
[0069] The check processing module is used to check whether there are life cycle related tasks in the schedule system table when the database service is started. If so, the scan trigger module is directly called. If not, a default life cycle task is created and then the scan trigger module is called.
[0070] The scan trigger module is used to start a background coroutine, which scans the schedule system table to obtain all data cleaning tasks to be executed, and triggers the operation of cleaning expired data when the current time exceeds the scheduled execution time of the task;
[0071] The job creation module is used to create a background job when the background coroutine triggers the operation of clearing expired data. The background job obtains the life cycle information of all time series tables and sends it to the storage system;
[0072] The comparison processing module is used to receive life cycle information, compare timestamps, identify time series data whose timestamps exceed the life cycle range, and automatically clear them;
[0073] The time update module is used to update the next execution time in the schedule system table after the comparison processing module completes the automatic clearing of expired data, and mark the data clearing task of this cycle as completed.
[0074] In this embodiment, when the user creates a time series database and a time series table by creating a definition module:
[0075] If the user sets the data retention period for a time series database and also sets the data retention period for a time series table under the time series database, the data retention period for the time series table takes precedence over the data retention period for the time series database to which the time series table belongs.
[0076] If the user does not set the data retention period for a time series table when creating it, the time series table inherits the data retention period of the time series database to which it belongs.
[0077] In this embodiment, the schedule system table is used to save all scheduling tasks, including data cleaning tasks related to the life cycle; the schedule system table involved records ten fields: schedule id, name, creator, owner, next execution time, status, cron expression, detail, executor name and executor parameters, among which schedule id is used as the primary key of the schedule system table, the next execution time is used as the index field, and the cron expression is used to define the execution time of the task. The user controls the time period for executing the data cleaning task by modifying the cron expression.
[0078] When there are no lifecycle-related tasks in the schedule system table, the default lifecycle task created by the inspection processing module contains a predefined data cleanup strategy.
[0079] In this embodiment, the Detail of the schedule system table record is used to describe the parameters and behaviors of the task in detail;
[0080] The user can modify the details of the schedule system table through the alter statement to change the execution mode of the data cleanup task in different situations. The details include the failure handling strategy and the handling mode of the previous task execution:
[0081] (1) For the failure handling strategy, the following three measures are taken: a) immediately try to re-execute the task, b) try to execute the task again at the next scheduling, c) suspend the execution of the task until yoghurt manually intervenes;
[0082] (2) Regarding the handling method of the previous task execution, the following three measures are taken: a) Regardless of whether the previous task is still running, start the current task immediately; b) If the previous task is still running, skip the current execution and wait for the next scheduling; c) If the previous task is still running, wait for it to complete before starting the current task.
[0083] It should be added that the device for periodically clearing expired data in a time series database described in this embodiment further includes a task viewing module. The user enters the show job command through the task viewing module to view the historical task execution results of clearing expired data and the execution status of the current task.
[0084] In summary, the method and device for periodically cleaning up expired data in a time series database of the present invention can simplify the management and maintenance costs of users, help users actively clean up expired time series data, save disk usage space, and improve user efficiency.
[0085] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.
Claims
1. A method for regularly cleaning up expired data in a time series database, characterized in that: It includes the following steps: S1. When a user creates a time series database or time series table, he / she defines the data retention period by specifying the retention keyword; S2. When the database service is started, first check whether there are lifecycle-related tasks in the schedule system table. If so, directly execute the subsequent steps. If not, create a default lifecycle task and then execute the subsequent steps. S3. The database service starts a background coroutine to regularly scan the schedule system table to obtain all data cleanup tasks to be executed, and then triggers the operation of cleaning up expired data when the current time exceeds the scheduled execution time of the task; S4. When the background coroutine triggers the operation of clearing expired data, a background job is created. The background job obtains the life cycle information of all time series tables and sends it to the storage system. The storage system receives the life cycle information, compares the timestamps, identifies the time series data whose timestamps exceed the life cycle range, and automatically clears them. S5. After the automatic clearing of expired data is completed, the next execution time in the schedule system table is updated, and the data clearing task of this cycle is marked as completed.
2. According to claim 1, a method for regularly cleaning up expired data in a time series database is characterized in that: Execute step S1. If the user sets the data retention period of the time series database and also sets the data retention period of the time series table under the time series database, the data retention period of the time series table has a higher priority than the data retention period of the time series database to which the time series table belongs. If the user does not set the data retention period for a time series table when creating it, the time series table inherits the data retention period of the time series database to which it belongs.
3. According to claim 1, a method for regularly cleaning up expired data in a time series database is characterized in that: Execute step S2, the schedule system table is used to save all scheduling tasks, including data cleaning tasks related to the life cycle; the schedule system table records ten fields: schedule id, name, creator, owner, next execution time, status, cron expression, detail, executor name and executor parameters, among which schedule id is used as the primary key of the schedule system table, the next execution time is used as the index field, and the cron expression is used to define the execution time of the task. The user controls the time period for executing the data cleaning task by modifying the cron expression; When there is no lifecycle-related task in the schedule system table, the created default lifecycle task includes a predefined data cleanup strategy.
4. According to claim 3, a method for regularly cleaning up expired data in a time series database is characterized in that: The Detail recorded in the schedule system table is used to describe the parameters and behaviors of the task in detail; The user can modify the details of the schedule system table through the alter statement to change the execution mode of the data cleanup task in different situations. The details include the failure handling strategy and the handling mode of the previous task execution: (1) For the failure handling strategy, the following three measures are taken: a) immediately try to re-execute the task, b) try to execute the task again at the next scheduling, c) suspend the execution of the task until yoghurt manually intervenes; (2) Regarding the handling method of the previous task execution, the following three measures are taken: a) Regardless of whether the previous task is still running, start the current task immediately; b) If the previous task is still running, skip the current execution and wait for the next scheduling; c) If the previous task is still running, wait for it to complete before starting the current task.
5. According to claim 1, a method for regularly cleaning up expired data in a time series database is characterized in that: After executing step S5, the user can view the historical task execution results of clearing expired data and the execution status of the current task by using the show job command.
6. A device for regularly cleaning up expired data in a time series database, characterized in that: It includes: Create a definition module to assist users in creating a time series database or time series table, and define the data retention period by specifying the retention keyword; The check processing module is used to check whether there are life cycle related tasks in the schedule system table when the database service is started. If so, the scan trigger module is directly called. If not, a default life cycle task is created and then the scan trigger module is called. The scan trigger module is used to start a background coroutine, which scans the schedule system table to obtain all data cleaning tasks to be executed, and triggers the operation of cleaning expired data when the current time exceeds the scheduled execution time of the task; The job creation module is used to create a background job when the background coroutine triggers the operation of clearing expired data. The background job obtains the life cycle information of all time series tables and sends it to the storage system; The comparison processing module is used to receive life cycle information, compare timestamps, identify time series data whose timestamps exceed the life cycle range, and automatically clear them; The time update module is used to update the next execution time in the schedule system table after the comparison processing module completes the automatic clearing of expired data, and mark the data clearing task of this cycle as completed.
7. The device for periodically cleaning up expired data in a time series database according to claim 6, characterized in that: When users create a time series database and time series table by creating a definition module: If the user sets the data retention period for a time series database and also sets the data retention period for a time series table under the time series database, the data retention period for the time series table takes precedence over the data retention period for the time series database to which the time series table belongs. If the user does not set the data retention period for a time series table when creating it, the time series table inherits the data retention period of the time series database to which it belongs.
8. The device for periodically cleaning up expired data in a time series database according to claim 6, characterized in that: The schedule system table is used to save all scheduling tasks, including data cleaning tasks related to the life cycle; the schedule system table records ten fields: scheduleid, name, creator, owner, next execution time, status, cron expression, detail, executor name and executor parameters, among which scheduleid is used as the primary key of the schedule system table, next execution time is used as the index field, cron expression is used to define the execution time of the task, and the user controls the time period for executing the data cleaning task by modifying the cron expression; When there is no lifecycle-related task in the schedule system table, the default lifecycle task created by the inspection processing module includes a predefined data cleanup strategy.
9. The device for periodically cleaning up expired data in a time series database according to claim 8, characterized in that: The Detail recorded in the schedule system table is used to describe the parameters and behaviors of the task in detail; The user can modify the details of the schedule system table through the alter statement to change the execution mode of the data cleanup task in different situations. The details include the failure handling strategy and the handling mode of the previous task execution: (1) For the failure handling strategy, the following three measures are taken: a) immediately try to re-execute the task, b) try to execute the task again at the next scheduling, c) suspend the execution of the task until yoghurt manually intervenes; (2) Regarding the handling method of the previous task execution, the following three measures are taken: a) Regardless of whether the previous task is still running, start the current task immediately; b) If the previous task is still running, skip the current execution and wait for the next scheduling; c) If the previous task is still running, wait for it to complete before starting the current task.
10. The device for periodically cleaning up expired data in a time series database according to claim 6, characterized in that: The device also includes a task viewing module; The user enters the show job command in the task viewing module to view the historical task execution results of clearing expired data and the execution status of the current task.