A method for cleaning up expired videos in large-scale video storage

Abstract

The present invention relates to a method for cleaning up expired videos in large-scale video storage, which includes creating a task list in Redis, and putting all storage index table names into the task list; OSS_Scheduler periodically publishes multiple cleaning tasks according to the database table; more than one OSS preempts the cleaning tasks; more than one OSS executes the cleaning tasks at the same time, and each task fixedly queries and cleans up expired videos in a database table. Through the periodic task publishing of OSS_Scheduler and the task preemption mechanism of OSS, each cleaning task is clearly designated to be responsible for cleaning up expired videos in a certain database table, thereby ensuring the uniqueness and accuracy of the cleaning process and eliminating the occurrence of repeated cleaning. Without the need to increase database nodes (i.e., without the need to invest in additional hardware costs), the present invention successfully solves key problems such as insufficient video cleaning speed and repeated cleaning in large-scale video surveillance platforms, and significantly improves the overall performance and operating efficiency of the system.

Description

Technical Field

[0001] The invention belongs to the technical field of video storage services in the video surveillance industry, and in particular relates to a method for clearing expired videos in large-scale video storage. Background Art

[0002] Video surveillance platforms are usually equipped with video storage functions, providing storage services (OSS) for video data recorded by thousands to tens of thousands of cameras on the platform. These videos are stored in storage media in the form of multiple files. Given the large number of videos and the limited storage media capacity, it is necessary to regularly clean up expired videos that were recorded a certain period of time ago.

[0003] The current mainstream cleaning strategy is a timed cycle cleaning mechanism, which performs cleaning tasks through a single or multiple OSSs. This mechanism sets a timer inside the OSS, starts the timer inside the service, and automatically performs the following steps at fixed intervals: first, retrieve the index of expired video files from the database; second, delete the corresponding video files from the storage medium based on the file path in the index; finally, remove the index record of the deleted file from the database.

[0004] However, the prior art has the following obvious deficiencies:

[0005] 1. Because the load of a single OSS is limited, if a single OSS is used for cleaning, the cleaning speed can be maintained when the number of devices connected to the platform is small. However, when facing large-scale devices (such as more than 5,000 channels), the cleaning speed may not be able to keep up with the speed of generating new videos, resulting in a backlog of video data.

[0006] 2. If multiple OSS are used to perform cleanup at the same time, it is easy to cause repeated cleanup problems, resulting in reduced cleanup efficiency and even erroneous deletion. In addition, the cleanup process involves a large number of database operations. The superposition of operations from multiple OSS will put a lot of pressure on the database. Without adding database nodes (adding database nodes means additional hardware cost expenditure), it may cause slow database response and operation timeout (single query response time may be extended to several seconds to tens of seconds), affecting the overall service performance.

[0007] In view of this, an innovative solution is urgently needed in this field to effectively solve the problems of insufficient video cleaning speed and repeated cleaning of large-scale video surveillance platforms without increasing database nodes (i.e. avoiding additional hardware costs). Summary of the invention

[0008] In view of the shortcomings of the current related existing technologies, the present invention provides a method for clearing expired videos in large-scale video storage, which effectively addresses and solves key problems such as insufficient video clearing speed and repeated clearing in large-scale video surveillance platforms without increasing database nodes (i.e. avoiding additional hardware cost investment).

[0009] To achieve the above object, the technical solution adopted by the present invention is: a method for clearing expired videos in large-scale video storage, which at least includes the following steps:

[0010] Create a task list in Redis and put all storage index table names into the task list;

[0011] OSS_Scheduler regularly publishes multiple cleanup tasks according to database tables;

[0012] More than one OSS preempts the cleanup task;

[0013] More than one OSS executes cleanup tasks at the same time. Each task queries and cleans up expired videos in a database table.

[0014] Furthermore, the method further includes at least the following steps: Step S1: OSS_Scheduler is responsible for querying and retrieving all database storage index table names, and then creating and initializing a task list in the Redis database, and adding all retrieved storage index table names to the task list one by one for subsequent processing;

[0015] Step S2: When the OSS starts the cleanup process, it first queries the Redis database for the number of OSSs currently executing cleanup tasks, and compares this number with the system preset concurrent execution limit to determine whether the number of OSSs executing cleanup tasks simultaneously has reached the set value;

[0016] Step S3: To ensure that there is no data competition or conflict during the execution of the cleanup task, the OSS will first lock the task list in Redis. This locking mechanism will prohibit other OSSes from obtaining tasks in the task list, thus preventing conflicts with other OSSes when obtaining tasks.

[0017] Step S4: OSS obtains a storage index table name from the locked task list as the task to be cleaned in this cleanup process, and prepares for subsequent cleanup work;

[0018] Step S5: OSS will lock the acquired storage index table name in Redis. This lock will ensure that only one OSS can clean up this storage index table to prevent multiple OSSs from cleaning up this storage index table at the same time when the cleanup task is executed across days.

[0019] Further, in step S2, it is determined whether the number of OSSs executing the cleaning task simultaneously has reached a set value;

[0020] If it has been reached, the cleaning process will be started again after the scheduled time is set, and this process ends;

[0021] If the set value is not reached, OSS will mark itself as being executed and store it in Redis for query by other OSSs and OSS_Scheduler.

[0022] Furthermore, in step S4, the OSS obtains a storage index table name from the task list as a task to be cleaned up;

[0023] If no results are obtained, it means that all cleanup tasks in this period have been completed. Set the scheduled time and start again. Redis will mark this OSS as no longer executing cleanup tasks. This process ends.

[0024] If OSS obtains a storage index table name, it deletes it from the task list, indicating that it has accepted the cleanup task, and then unlocks the task list in Redis to allow other OSSs to obtain tasks.

[0025] Furthermore, in step S5, OSS will lock the acquired storage index table name in Redis;

[0026] If the lock fails, it means that another OSS is already executing the cleanup task for this index table. Then, the cleanup process will be started again after the scheduled time, and this process will end.

[0027] If the lock is successful, the cleanup task execution process will begin.

[0028] Further, the process of executing the cleanup task is as follows: Step S51, for the specified index table, try to query N expired indexes, and determine the number of queried indexes, where N is a natural number greater than or equal to 1;

[0029] S511, if the number of indexes found is not 0, deleting the corresponding video file according to the address of the video file recorded in the index;

[0030] S512, deleting this index in the database index table;

[0031] S513, punch in the Redis lock of the current cleanup task to indicate that the OSS itself is executing the task; if the deletion action takes too long to execute, the Redis lock times out and the punch in fails, then the current task is locked again and step S51 is executed repeatedly; if the punch in succeeds, step S51 is executed repeatedly;

[0032] S521. If the number of indexes queried is 0, it means that the index table has been cleaned up, and the current cleanup task is unlocked in Redis.

[0033] S522, determining whether the current index table is empty, that is, all indexes have expired and been deleted;

[0034] If the index table is empty, delete the empty table and proceed to the next step;

[0035] If the index table is not empty, proceed directly to the next step;

[0036] Set the scheduled time to execute the next round of cleanup tasks.

[0037] Furthermore, the number of OSSs that execute cleanup tasks simultaneously can be configured to ensure cleanup efficiency while avoiding excessive load pressure on the database system.

[0038] The present invention adopts the above technical solution to build a task list in Redis and include all storage index table names to be processed in this list. Subsequently, OSS_Scheduler (storage scheduling service) will periodically publish multiple cleanup tasks based on the information in the database table. These tasks will be competitively preempted and executed by one or more OSS (object storage service), and each OSS will be independently responsible for querying and cleaning up expired video data in a specific database table.

[0039] This technical solution brings at least the following significant advantages:

[0040] 1. Flexible configuration and pressure control: Users can manually set the number of OSSs that perform cleanup tasks simultaneously according to actual needs, thereby ensuring cleanup efficiency while effectively avoiding unnecessary pressure on the database system.

[0041] 2. Avoid repeated cleaning: Through the regular task release of OSS_Scheduler and the task preemption mechanism of OSS, each cleaning task is clearly designated to be responsible for cleaning up expired videos in a certain database table, thereby ensuring the uniqueness and accuracy of the cleaning process and eliminating the occurrence of repeated cleaning.

[0042] 3. Significant cost-effectiveness: Without the need to increase database nodes (i.e., without the need to invest in additional hardware costs), the present invention successfully solves key problems such as insufficient video cleaning speed and repeated cleaning in large-scale video surveillance platforms, significantly improving the overall performance and operating efficiency of the system.

[0043] In summary, the present invention provides an efficient, reliable and cost-effective method for clearing expired videos in large-scale video storage, which provides strong support for the operation and maintenance management of large-scale video surveillance platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is a flow chart of the implementation method of the method for cleaning up expired videos of the present invention;

[0046] Figure 2 It is a flowchart of the precondition for starting the cleaning task of the present invention;

[0047] Figure 3 It is a flow chart of the present invention performing the cleaning task. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0049] See also Figure 1 This embodiment provides a method for clearing expired videos in large-scale video storage, which at least includes the following steps:

[0050] Create a task list in Redis and put all storage index table names into the task list;

[0051] OSS_Scheduler regularly publishes multiple cleanup tasks according to database tables;

[0052] More than one OSS preempts the cleanup task;

[0053] More than one OSS executes cleanup tasks at the same time. Each task queries and cleans up expired videos in a database table.

[0054] See also Figure 1 and Figure 2As shown, this embodiment also includes at least the following steps: Step S1: OSS_Scheduler is responsible for querying and retrieving all database storage index table names (recordfile_1, recordfile_2, etc.), then creating and initializing a task list in the Redis database, and adding all retrieved storage index table names to the task list one by one for subsequent processing;

[0055] Step S2: When the OSS starts the cleanup process, it first queries the Redis database for the number of OSSs currently executing cleanup tasks, and compares this number with the system preset concurrent execution limit to determine whether the number of OSSs executing cleanup tasks simultaneously has reached the set value;

[0056] Further, in step S2, it is determined whether the number of OSSs executing the cleaning task simultaneously has reached a set value;

[0057] If it has been reached, the cleaning process will be started again after the preset time (20 minutes), and this process will end;

[0058] If the set value is not reached, OSS will mark itself as being executed and store it in Redis for query by other OSSs and OSS_Scheduler.

[0059] Step S3: To ensure that there is no data competition or conflict during the execution of the cleanup task, the OSS will first lock the task list in Redis. This locking mechanism will prohibit other OSSes from obtaining tasks in the task list, thus preventing conflicts with other OSSes when obtaining tasks.

[0060] Step S4: OSS obtains a storage index table name from the locked task list as the task to be cleaned in this cleanup process, and prepares for subsequent cleanup work;

[0061] Furthermore, in step S4, the OSS obtains a storage index table name from the task list as a task to be cleaned up;

[0062] If it is not obtained, it means that all cleanup tasks in this period (today) have been completed. Set the scheduled time (20 minutes) and start again to wait for the next day. Redis marks that this OSS will no longer execute cleanup tasks. This process ends;

[0063] If OSS obtains a storage index table name, it deletes it from the task list, indicating that it has accepted the cleanup task, and then unlocks the task list in Redis to allow other OSSs to obtain tasks.

[0064] Step S5: OSS will lock the acquired storage index table name in Redis. This lock will ensure that only one OSS can clean up this storage index table to prevent multiple OSSs from cleaning up this storage index table at the same time when the cleanup task is executed across days.

[0065] Furthermore, in step S5, OSS will lock the acquired storage index table name in Redis;

[0066] If the lock fails, it means that another OSS is already executing the cleanup task for this index table. Then, the cleanup process will be started again after a preset time (10 seconds) (in order to select another index table for cleanup). This process ends.

[0067] If the locking is successful, the locked task is stored in the database index table being cleaned up in Redis, and then the cleanup task execution process begins.

[0068] As a preferred embodiment, Figure 3 As shown, the process of executing the cleaning task in this embodiment is as follows: Step S51, for the specified index table, try to query N expired indexes, and determine the number of queried indexes, where N is a natural number greater than or equal to 1; in this embodiment, try to query 1000 expired indexes and determine whether the number of queried indexes is 0.

[0069] S511, if the number of indexes found is not 0, deleting the corresponding video file according to the address of the video file recorded in the index;

[0070] S512, deleting this index in the database index table;

[0071] S513, punch in the Redis lock of the current cleanup task to indicate that the OSS itself is executing the task; if the deletion action takes too long to execute, the Redis lock times out and the punch in fails, then the current task is locked again and step S51 is executed repeatedly; if the punch in succeeds, step S51 is executed repeatedly;

[0072] S521. If the number of indexes queried is 0, it means that the index table has been cleaned up, and the current cleanup task is unlocked in Redis.

[0073] S522, determining whether the current index table is empty, that is, all indexes have expired and been deleted;

[0074] If the index table is empty, delete the empty table and proceed to the next step;

[0075] If the index table is not empty, proceed directly to the next step;

[0076] Set the scheduled time to execute the next round of cleanup tasks.

[0077] In this embodiment, the number of OSSs that execute cleaning tasks simultaneously can be configured to ensure cleaning efficiency while avoiding excessive load pressure on the database system.

[0078] The present invention adopts the above technical solution to build a task list in Redis and include all storage index table names to be processed in this list. Subsequently, OSS_Scheduler (storage scheduling service) will periodically issue multiple cleanup tasks based on the information in the database table. These tasks will be competitively preempted and executed by one or more OSS (video storage service), and each OSS will be independently responsible for querying and cleaning up expired video data in a specific database table.

[0079] In this embodiment, the user can manually set the number of OSSs that perform cleaning tasks simultaneously according to actual needs, thereby ensuring cleaning efficiency while effectively avoiding unnecessary pressure on the database system. Through the regular task release of OSS_Scheduler and the task preemption mechanism of OSS, each cleaning task is clearly designated to be responsible for cleaning up expired videos in a certain database table, thereby ensuring the uniqueness and accuracy of the cleaning process and eliminating the occurrence of repeated cleaning. Without the need to increase database nodes (i.e., without the need for additional hardware cost investment), the present invention successfully solves key problems such as insufficient video cleaning speed and repeated cleaning in large-scale video surveillance platforms, and significantly improves the overall performance and operating efficiency of the system. The present invention provides an efficient, reliable and cost-effective method for cleaning up expired videos in large-scale video storage, which provides strong support for the operation and maintenance management of large-scale video surveillance platforms.

[0080] Although the embodiments of the present invention have been shown and described above, it will be appreciated that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, substitute and vary the above embodiments within the scope of the present invention.

Claims

1. A method for cleaning up expired videos in large-scale video storage, characterized by: It at least includes the following steps: Create a task list in Redis and put all storage index table names into the task list; OSS_Scheduler regularly publishes multiple cleanup tasks according to database tables; More than one OSS preempts the cleanup task; More than one OSS executes cleanup tasks at the same time. Each task queries and cleans up expired videos in a database table. It also includes at least the following steps: Step S1: OSS_Scheduler is responsible for querying and retrieving all database storage index table names, then creating and initializing a task list in the Redis database, and adding all retrieved storage index table names to the task list one by one for subsequent processing; Step S2: When the OSS starts the cleanup process, it first queries the Redis database for the number of OSSs currently executing cleanup tasks, and compares this number with the system preset concurrent execution limit to determine whether the number of OSSs executing cleanup tasks simultaneously has reached the set value; Step S3: To ensure that there is no data competition or conflict during the execution of the cleanup task, the OSS will first lock the task list in Redis. This locking mechanism will prohibit other OSSes from obtaining tasks in the task list, thus preventing conflicts with other OSSes when obtaining tasks. Step S4: OSS obtains a storage index table name from the locked task list as the task to be cleaned in this cleanup process, and prepares for subsequent cleanup work; Step S5: OSS will lock the acquired storage index table name in Redis. This lock will ensure that only one OSS can clean up this storage index table to prevent multiple OSSs from cleaning up this storage index table at the same time when the cleanup task is executed across days.

2. The method according to claim 1, characterized in that: In step S2, it is determined whether the number of OSSs executing the cleaning task simultaneously has reached a set value; If it has been reached, the cleaning process will be started again after the scheduled time is set, and this process ends; If the set value is not reached, OSS will mark itself as being executed and store it in Redis for query by other OSSs and OSS_Scheduler.

3. The method according to claim 2, characterized in that: In step S4, OSS obtains a storage index table name from the task list as a task to be cleaned up; If no results are obtained, it means that all cleanup tasks in this period have been completed. Set the scheduled time and start again. Redis will mark this OSS as no longer executing cleanup tasks. This process ends. If OSS obtains a storage index table name, it deletes it from the task list, indicating that it has accepted the cleanup task, and then unlocks the task list in Redis to allow other OSSs to obtain tasks.

4. The method according to claim 3, characterized in that: In step S5, OSS will lock the acquired storage index table name in Redis; If the lock fails, it means that another OSS is already executing the cleanup task for this index table. Then, the cleanup process will be started again after the scheduled time, and this process will end. If the lock is successful, the cleanup task execution process will begin.

5. The method according to any one of claims 1 to 4, characterized in that: The process of executing the cleanup task is as follows: Step S51, for a specified index table, try to query N expired indexes, and determine the number of queried indexes, where N is a natural number greater than or equal to 1; S511, if the number of indexes found is not 0, deleting the corresponding video file according to the address of the video file recorded in the index; S512, deleting this index in the database index table; S513, punch in the Redis lock of the current cleanup task to indicate that the OSS itself is executing the task; if the deletion action takes too long to execute, the Redis lock times out and the punch in fails, then the current task is locked again and step S51 is executed repeatedly; If the clock-in is successful, the step S51 is executed in a loop; S521. If the number of indexes queried is 0, it means that the index table has been cleaned up, and the current cleanup task is unlocked in Redis. S522, determining whether the current index table is empty, that is, all indexes have expired and been deleted; If the index table is empty, delete the empty table and proceed to the next step; If the index table is not empty, proceed directly to the next step; Set the scheduled time to execute the next round of cleanup tasks.

6. The method according to claim 5, characterized in that: The number of OSSs that perform cleanup tasks simultaneously can be configured to ensure cleanup efficiency while avoiding excessive load pressure on the database system.

Images