A video download method based on local index persistence and direct connection to object storage

Abstract

This invention discloses a video recording download method based on local index persistence and direct connection to object storage, belonging to the field of video surveillance and multimedia data processing technology. It obtains a pre-generated local persistent index database and parses the encapsulated object storage service connection configuration information to establish a direct communication link. Download tasks are managed by a restricted download thread pool, which schedules tasks between the task waiting queue and the set of running tasks based on their status. Based on the file list and progress index, video segments are downloaded via the direct connection link, and the progress index is updated in a transactional manner after each segment is successfully downloaded. When preset conditions are met, the downloaded segments undergo format encapsulation processing. This invention, through offline pre-encapsulated local index driving and direct connection mechanism to object storage, decouples the download logic from the business query logic, eliminates the performance bottleneck of the streaming media server, and solves the problems of easy interruption and progress loss requiring retrying during long-term video recording downloads.

Description

Technical Field

[0001] This invention relates to the field of video surveillance and multimedia data processing technology, specifically to a video recording download method based on local index persistence and direct connection to object storage. Background Technology

[0002] In modern large-scale video management systems, the retrieval and download of surveillance footage is one of the core business functions, widely used in critical scenarios such as security evidence collection and event tracing. With the development of cloud computing technology, S3 protocol-compatible object storage systems have gradually become the mainstream storage medium for surveillance video data due to their massive storage capacity, high reliability, and low cost.

[0003] Currently, the mainstream video download method for VMS clients adopts a centralized forwarding architecture, and its complete process is as follows: Users first log in to the VMS client and establish a connection with the server; then, they perform a query operation, selecting the target device, query time range, and video file type; based on the query results, they manually create a download task. At the data transmission level, the video data flow is as follows: the camera records and generates video files, which are stored in the backend storage system, retrieved and forwarded by the streaming media server, and finally transmitted to the client.

[0004] More specifically, the existing download execution steps are as follows: 1. The client first completes identity authentication with the business server and obtains an access token after successful authentication; 2. The client uses the access token to query the address of the streaming media server and establishes a connection with the streaming media server; 3. The user selects the target device from the device tree, specifies the download time range and recording type, and sends a download request to the business server; 4. The business server forwards the download request to the streaming media server, which retrieves the corresponding recording data from the backend storage and sends it in the form of a video stream; 5. The client receives the video stream data, uses multimedia processing tools to encapsulate the received video frames into a recording file of the target format, and saves it locally.

[0005] In the aforementioned existing technical solutions, the stability of video data download is highly dependent on the continuous availability of each node in the intermediate link. Specifically, the current video download method faces the following clear technical shortcomings in practical applications: First, the excessively long download path limits system throughput due to numerous intermediate verification steps, making direct interaction between the client and object storage impossible. Each video download task consumes the streaming media server's forwarding resources, and data and requests must pass through multiple levels of nodes—client, business server, streaming media server, and storage device—each requiring identity verification and data forwarding. When a large number of users concurrently perform download operations, the streaming media server easily becomes the performance bottleneck of the entire system, manifesting as decreased download speed and increased response latency. Furthermore, the multi-level forwarding also increases transmission latency and system failure points, severely impacting the overall system throughput.

[0006] Secondly, the user operation process is cumbersome, with too many prerequisites and low execution efficiency. Users must complete a series of prerequisite interactive steps, including login authentication, device selection, time range setting, video type filtering, and manual creation of download tasks. Furthermore, the entire download process requires maintaining an active connection between the client and server. When batch processing multiple devices or retrieving videos across different time periods is needed, this operation mode significantly increases the user's manual operation costs and lacks the ability to operate independently and flexibly.

[0007] Third, the download status lacks a local persistence mechanism, meaning downloaded data cannot be retained. The status of an ongoing download task is typically only temporarily maintained by client memory or the server's online session. Once the network connection between the client and server is interrupted, or the user actively closes the client, the downloaded temporary file is still stored on the computer, but it is incomplete and fragmented. When the network connection drops or an error occurs, the application deletes this temporary file. Upon restarting the download, the user must re-initiate the download request from the beginning. In scenarios involving long download periods and large video files, this significantly increases time costs and the repeated consumption of network bandwidth.

[0008] Fourth, existing solutions lack effective isolation mechanisms for failures in a single download task, resulting in extremely poor stability during long downloads. During extended video downloads, if the download link is interrupted at any point due to network fluctuations or node anomalies, the entire download task is often deemed a complete failure, and a large amount of previously transmitted data is lost. Even if the anomaly only affects a brief period, users cannot retain successfully transmitted portions, nor can they accurately skip failed segments. Practical testing shows that long-duration video download tasks are highly prone to failure and cannot continue, forcing users to start the download from scratch.

[0009] To address the aforementioned issues, there is an urgent need in this field for a video recording download method that can effectively shorten the download process, simplify user operations, achieve local persistent storage of download progress, and support precise resume recording after abnormal task interruption. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of existing video download technologies, such as lengthy transmission links, high server load, non-persistent download progress, and poor stability of long-term tasks. It provides a video download method based on local index persistence and direct connection to object storage. This method eliminates the performance bottleneck of the streaming media forwarding process by establishing a direct communication link between the client and object storage. By acquiring a pre-generated local persistent index database containing connection configurations and fragment indexes, it achieves deep decoupling between download logic and business query logic, giving the download tool the ability to pre-read local index data offline. Simultaneously, the client still needs to maintain a network connection with the object storage service during actual download execution. Combined with the coordinated operation of a restricted download thread pool scheduling and a fragment-level transactional progress update mechanism, it ensures accurate continuation of download progress and fault isolation after network fluctuations or abnormal program exit. This invention aims to construct a lightweight download solution with a streamlined architecture, persistent task storage, and high fault tolerance.

[0011] To achieve the above objectives, the present invention provides a video recording download method based on local index persistence and direct connection to object storage, comprising: Obtain a pre-generated local persistent index database, wherein the local persistent index database pre-encapsulates the connection configuration information of the object storage service, the file list information of all video segments included in the download task, and the current download progress index; Parse the connection configuration information to establish a direct communication link between the client and the object storage service; The download tasks are managed by a limited download thread pool, and the download tasks are transferred between the task waiting queue and the set of running tasks according to the task status; Based on the file list information and the current download progress index, the storage path of the video segment to be downloaded in the object storage service is determined, and the execution thread is obtained from the running task set through the direct communication link to execute the download of the video segment; After the video segment is successfully downloaded, the current download progress index in the local persistent index database is updated in a transactional manner so that the updated current download progress index points to the next video segment to be downloaded; The system checks whether the download task meets the preset completion conditions. If so, it calls the multimedia processing interface to perform format encapsulation processing on the multiple video segments that have been downloaded to the local cache, and generates a video file in the target format.

[0012] Furthermore, the local persistent index database is pre-generated from the business system by an offline export tool and can be reused by one or more download tools; the local persistent index database includes: The storage configuration table records the access address, port number, access credentials, and bucket name of the object storage service. The task overview table records the task identifier, device identifier, channel identifier, total number of files, and the current download progress index; The file details table records the storage path, timestamp, encryption identifier, and decryption key of each video segment in the object storage service. The name of the file details table is dynamically generated by combining the device identifier, the channel identifier, and the task identifier.

[0013] Furthermore, the download tasks are managed by a restricted download thread pool, and the download tasks are transferred between the task waiting queue and the set of running tasks according to the task status, specifically including: Construct a limited download thread pool with a preset maximum number of threads, and enqueue the download tasks to be executed into the task waiting queue in the order they were acquired; When there is an idle download thread, the idle download thread extracts a download task from the task waiting queue, changes the task status of the download task from the ready state to the running state, and stores the download task in the running task set; When there are no idle download threads and the number of currently running threads has not reached the preset maximum number of threads, a new download thread is created to execute the download task; When there are no idle download threads and the number of currently running threads has reached the preset maximum number of threads, the task status of the download task is kept in the ready state and retained in the task waiting queue.

[0014] Furthermore, it also includes download task pause handling logic in response to pause commands: Receive a pause command for a target running task and mark the task status of the target running task as paused; After the video segment currently being downloaded by the target task is completely downloaded to the local cache and the current download progress index is updated, the execution of the download thread corresponding to the target task is terminated, and the target task is removed from the set of running tasks.

[0015] Furthermore, it also includes download task recovery processing logic in response to recovery commands: Receive the command to resume the paused task for the target; Determine whether the target paused task still exists in the set of running tasks; If it exists, the task status of the target paused task will be changed directly from paused to running, skipping the re-queueing and thread allocation process, and continuing to execute the segmented download; If it does not exist, the target paused task is added back to the end of the task waiting queue, its execution priority is placed after other ready tasks in the task waiting queue, and its task status is changed to ready, so that it can be extracted and executed in sequence when there is an idle download thread.

[0016] Furthermore, the method also includes: Maintain a full task mapping set, storing all started download tasks in the full task mapping set with their respective task identifiers as keys; When a pause or resume command is received for a specific target task, the target task can be directly located in the full task mapping set by the task identifier of the target task, without traversing the task waiting queue or the set of running tasks.

[0017] Furthermore, before downloading the video fragments from the storage path to the local cache, the process also includes: Read the encryption identifier corresponding to the video segment recorded in the file list information; If the encryption identifier indicates that the video segment is encrypted, then the corresponding decryption key is extracted from the file list information; After the video segment is downloaded to the local cache, the decryption key is used to perform a decryption operation on the video segment, so that the local cache stores the decrypted video segment data.

[0018] Furthermore, updating the current download progress index in the local persistent index database in a transactional manner includes: After each video segment is successfully downloaded, a database transaction is initiated to write the updated current download progress index into the local persistent index database. This ensures that if the client exits abnormally and restarts, the download can continue from the next video segment indicated by the persistent current download progress index.

[0019] Furthermore, it also includes fault tolerance handling for download failure events: When a download failure occurs during the download of the video segment, the reason for the download failure is determined. If the failure is determined to be due to a network anomaly, a network failure message is sent to the user, and a periodic network connectivity detection process for the object storage service is initiated; once the network is detected to have recovered, the download task is automatically resumed. If the failure is determined to be due to file corruption or path invalidation that is not caused by network factors, the failure information of the current video segment is recorded in the local log, the current download progress index is incremented to skip the current video segment, and the download of subsequent video segments in the download task continues.

[0020] Furthermore, the method employs a dual-path storage mechanism, including: The original video segments downloaded from the object storage service are stored in a temporary download path; The target format video file generated after decryption and format encapsulation is stored in the video storage path, which is different from the temporary download path.

[0021] Furthermore, it checks whether the download task meets preset completion conditions. If so, it calls the multimedia processing interface to perform format encapsulation processing on the multiple video segments already downloaded to the local cache, generating a recording file in the target format, including: When the number of video segments downloaded to the local cache reaches a preset merging threshold, the preset number of video segments are decrypted and format-encapsulated to generate a video file in the target format. When all video segments in the download task have been downloaded and the number of remaining unmerged video segments has not reached the merging threshold, the decryption and format encapsulation processes of the remaining unmerged video segments are forcibly triggered to generate the last target format recording file.

[0022] Compared with the prior art, the present invention has the following significant advantages: This invention completely bypasses the multi-level forwarding process of traditional architectures, involving clients, business servers, streaming media servers, and storage devices, by establishing a direct communication link between the client and object storage service. By pre-generating authentication credentials, fragmentation indexes, and storage configurations in a local persistent index database and embedding them on the client, the data transmission link evolves from multi-level forwarding to a point-to-point direct connection, eliminating the performance bottlenecks of streaming media servers in terms of bandwidth and concurrent processing. In large-scale concurrent download scenarios, the overall system throughput is no longer limited by the processing capacity of a single forwarding node, exhibiting near-linear scalability.

[0023] This invention constructs an offline scheduling engine with task awareness by acquiring a pre-generated local persistent index database. This database pre-encapsulates complete object storage connection configuration and file manifest information, enabling the download execution process to completely decouple from the online signaling control of the business management system. Users can initiate download tasks directly without performing pre-interaction operations such as login authentication, device tree retrieval, and time setting. This design endows the download tool with independent operation capabilities in weak network or isolated environments, achieving deep decoupling between download tasks and business query logic, and significantly reducing the dependence of long-term tasks on the stability of the main system connection.

[0024] To address the industry challenges of frequent interruptions and high retry costs associated with long-duration video downloads, this invention employs a database transaction-based progress management mechanism. By updating the current download progress index in the local persistent index database in a transactional manner after each video segment is successfully downloaded, the real-time integrity and consistency of progress data at the local physical layer are ensured. Even in the event of abnormal client exit, network interruption, or system crash and restart, completed download progress is not lost. Download can resume from the next video segment indicated by the persistent current download progress index, achieving precise file-level breakpoint continuation. This avoids the time and network bandwidth consumption caused by re-downloading from the starting position due to progress loss in existing technologies.

[0025] This invention addresses the industry pain point of single-point failures causing overall download task failure in existing technologies by classifying failure causes and implementing differentiated fault-tolerance processing for download failure events. In case of network anomalies, a periodic network connectivity detection process is initiated, and the download task is automatically resumed after network recovery. In cases of file corruption or path invalidation due to non-network factors, the current download progress index is incremented to skip the currently failed segment and continue downloading subsequent segments. This segment-level fault isolation design ensures that successfully downloaded segment data is not erased due to the failure of individual segments, guaranteeing that even in extremely unstable network environments, long-duration video download tasks can be delivered with maximum availability, significantly improving the download success rate. Simultaneously, the non-preemptive pause processing logic of the restricted download thread pool waits for the current segment to be fully downloaded and the progress index to be updated before terminating the thread. The priority downgrading scheduling in the recovery processing logic, which places paused tasks at the end of the ready queue, combined with a rapid location mechanism based on the full task mapping set using task identifiers, collectively ensures data consistency, scheduling fairness, and response efficiency of the task scheduling system in abnormal scenarios.

[0026] This invention supports asynchronous local decryption during the download process. The decryption key is provided by a local persistent index database, ensuring both cloud-based encrypted storage security and efficient local decryption on the client side. By setting a preset merging threshold, batch format encapsulation is triggered when the number of video segments downloaded to the local cache reaches the threshold, generating a playable target format video file. Simultaneously, a dual-path storage mechanism separating the temporary download path from the final video storage path avoids interference from temporary files, improving data management security. This flexible processing mechanism of downloading, decrypting, and encapsulating simultaneously allows users to obtain playable video results in stages without waiting for hours of long-term tasks to complete, significantly reducing the waiting time for users to obtain valid video data. Attached Figure Description

[0027] Figure 1 This is an overall flowchart of the video download method of the present invention; Figure 2 This is a flowchart illustrating the overall download process of this invention. Detailed Implementation

[0028] Example 1

[0029] This embodiment provides a video download method based on local index persistence and direct connection to object storage. This method is entirely based on a pre-exported local persistent index database driver, transforming operations such as video querying, device authentication, and file location—which must be performed online in traditional architectures—into two independent stages: offline pre-packaging and direct client-side download. This achieves a complete separation between the download execution logic and the business query logic. The following detailed explanation of the method follows.

[0030] Before a user initiates a download task, this method first obtains a local persistent index database pre-generated by the offline export tool. This database pre-encapsulates the connection configuration information for the object storage service, a file list of all video segments included in the download task, and the current download progress index. This database is extracted and solidified once by the export tool while maintaining a normal connection with the business system, rather than being retrieved from the server in real time during the download process. Once generated, this database file can be independently loaded and used by one or more download tools without reconnecting to the business system. This "export once, reuse multiple times" mechanism allows download tools to independently initiate and execute download tasks in weak network environments, isolated networks, and even completely offline scenarios, fundamentally eliminating dependence on online signaling from the business management system.

[0031] As a preferred implementation, the local persistent index database is implemented using a lightweight relational database file (such as SQLite), and it contains at least the following three core data tables: The storage configuration table records the access address, port number, access credentials (such as AK / SK key pairs), and bucket name of the object storage service. This table securely offloads the object storage access credentials, which were originally managed centrally by the business server, to the client's local machine, allowing the client to establish a trusted connection directly with the object storage service without needing authentication and forwarding from the business server.

[0032] The task overview table records the task identifier, device identifier, channel identifier, total number of files, and the current download progress index. This table is the top-level index for the entire download task, establishing a globally unique task identifier for each download task, facilitating rapid task retrieval and status management in multi-task concurrent scenarios. The current download progress index field records the sequence number of the downloaded segment as a single integer value. Its update operation is designed as an atomic operation at the database transaction level, ensuring the consistency of progress data.

[0033] The file details table records the storage path, timestamp, encryption identifier, and decryption key of each video segment in the object storage service. The table name is dynamically generated by concatenating the device identifier, channel identifier, and task identifier. This dynamic naming design allows the recording file indexes for different devices, channels, and tasks to be organized into their respective independent database tables. This ensures data structure clarity and facilitates quick location of the correct file details table for querying during the download process based on the current task context, avoiding the performance overhead of performing a full table traversal on a large-scale segment list.

[0034] After loading the local persistent index database, this method parses the connection configuration information recorded in the storage configuration table, extracts the access address, port number, and access credentials of the object storage service, and then establishes a direct communication link between the client and the object storage service. This link establishment process completely bypasses the business server and streaming media server; the client uses the authentication credentials encapsulated in the local persistent index database to directly initiate a connection request to the object storage service. Compared with the multi-level forwarding architecture of the client, business server, streaming media server, and storage device in existing technologies, this step achieves a "de-streaming media-based" point-to-point direct connection on the data transmission link. Since each download task no longer needs to go through the relay and encapsulation processing of the streaming media server, this link architecture has a natural horizontal scalability advantage when facing large-scale concurrent downloads. For each additional download client, only one direct link needs to be added between the client and the object storage service, without causing additional bandwidth and computational pressure on the centralized server.

[0035] After establishing a direct communication link, this method entrusts the download tasks to a restricted download thread pool for unified management, and redirects the download tasks between the task waiting queue and the set of running tasks according to the task status. The restricted download thread pool is the core component for task scheduling and control in this method. It constrains the upper limit of concurrent downloads by setting a maximum number of threads, avoiding the exhaustion of client system resources due to the unrestrained creation of threads when a large number of tasks start simultaneously. Unlike the simple thread reuse of general thread pools, the restricted download thread pool in this method is deeply coupled with the task state machine, dividing the lifecycle of each download task into four states: ready, running, paused, and completed. The task waiting queue and the set of running tasks are managed by two data structures, respectively, to manage tasks waiting to be executed and tasks currently being executed.

[0036] As a preferred implementation method, the specific execution logic of this step is as follows: A limited download thread pool with a preset maximum number of threads is constructed, and download tasks to be executed are enqueued into a task waiting queue in the order they are acquired. This preset maximum number of threads can be flexibly adjusted according to client hardware configuration or user settings, with a preferred value of 5 threads to achieve a balance between concurrency efficiency and system resource consumption. Enqueuing tasks in the order they are acquired ensures that tasks started earlier have priority for execution, avoiding task starvation.

[0037] When an idle download thread exists, it retrieves a download task from the task waiting queue, changes the task status from ready to running, and stores the task in the running task set. When no idle download thread exists and the current number of running threads has not reached the preset maximum number of threads, a new download thread is created to execute the download task. When no idle download thread exists and the current number of running threads has reached the preset maximum number of threads, the task status of the download task remains ready and is retained in the task waiting queue. This three-level judgment mechanism—allocating execution when idle, creating a new thread when the limit is not reached, and queuing when the limit is reached—makes the concurrent scheduling behavior of the thread pool have clear and predictable logic, facilitating precise state control in subsequent pause, resume, and fault-tolerant processing.

[0038] After a task is scheduled to run, this method determines the storage path of the video segment to be downloaded in the object storage service based on the file list information and the current download progress index. It then uses the direct communication link and obtains an execution thread from the set of running tasks to download the video segment. In this step, the introduction of the current download progress index allows for precise location of the specific segment to be downloaded each time a download is executed, rather than traversing from the task's starting position. For tasks that have been interrupted and then resumed, since the progress index was persistently updated via transactions before the interruption, the initial location after resumption can directly jump to the segment to be downloaded. Furthermore, the operation of "obtaining an execution thread from the set of running tasks" binds the allocation of download threads to the task state. Only when the task state has legally transitioned to the running state and been included in the set of running tasks can its corresponding thread execute the download operation. This design ensures strict synchronization between the state changes of pause and resume operations and the allocation of thread resources at the architectural level.

[0039] After each video segment is successfully downloaded, this method updates the current download progress index in the local persistent index database using a transaction, so that the updated current download progress index points to the next video segment to be downloaded. "Updated using a transaction" means initiating an atomic transaction at the database level, committing the increment operation of the progress index as an indivisible whole to the database file. This mechanism leverages the transactional characteristics inherent in lightweight databases (such as SQLite), ensuring that even if a program crash or power outage occurs during the progress index writing process, the database will not remain in an intermediate state where only part of the data has been written, but will instead be restored to the complete state before the write or the latest state after the commit through a transaction rollback mechanism. Compared to solutions that store progress data in memory or rely on server-side online sessions for maintenance, the "segment download completion and persistence" mechanism implemented in this step allows the download to continue from the next video segment indicated by the already persisted current download progress index when the client restarts after an abnormal exit, without having to start from the beginning. This can save significant time and bandwidth resources in scenarios involving large-volume, multi-segment long-duration video downloads.

[0040] During the download process, users can issue a pause command for any task at any time. As a preferred implementation, this method includes a complete set of non-preemptive pause processing logic: receiving a pause command for the target running task, marking the task status of the target running task as paused; after the video segment currently being downloaded by the target running task is fully downloaded to the local cache and the current download progress index is updated, terminating the execution of the download thread corresponding to the target running task, and removing the target running task from the set of running tasks. The reason this method adopts a non-preemptive strategy of "waiting for the current segment to complete before exiting," rather than directly interrupting the ongoing network transmission, is a targeted design based on the technical characteristics of object storage downloads. File downloads in object storage services (such as S3 protocol-compatible storage) are typically implemented based on HTTP long connections. If the connection is abruptly closed during transmission, not only will all received data be rendered useless, but it may also cause orphan connections on the server, resulting in resource waste. This method's non-preemptive strategy, by delaying the pause action until the current segment is fully downloaded and the progress is persisted, maximizes the protection of the validity of transmitted data and avoids resource waste while ensuring timely response to user operations.

[0041] For tasks in a paused state, users can issue a resumption command. As a preferred implementation, this method includes a set of resumption processing logic based on queue status: receiving a resumption command for a target paused task; determining whether the target paused task still exists in the set of running tasks; if it exists, directly changing the task status of the target paused task from paused to running, skipping the re-queueing and thread allocation process, and continuing to execute the segmented download; if it does not exist, adding the target paused task back to the end of the task waiting queue, placing its execution priority after other ready tasks in the task waiting queue, and changing its task status to ready, so that it can be extracted and executed in sequence when an idle download thread becomes available. This step effectively solves the efficiency problem of short-term misoperation in S3 download scenarios by differentiating whether the task is still in the running queue and adopting different resumption strategies. Because the non-preemptive pause strategy requires waiting for the current segment to be downloaded before exiting the thread, when the user performs consecutive "pause, quick resume" operations, the download of the current segment may not yet be completed, and the task may still remain in the set of running tasks. Mechanically re-queuing the task at this time would cause unnecessary scheduling overhead. This method determines the task's remaining status in the run queue. For such scenarios, execution is directly resumed. For regular recovery scenarios where the task has completely exited the thread, priority is downgraded and the task is enqueued at the end of the queue, thus balancing scheduling efficiency and fairness.

[0042] As a preferred implementation, this method also maintains a full task mapping set, storing all initiated download tasks with their respective task identifiers as keys. When a pause or resume command for a specific target task is received, the target task is directly located in the full task mapping set using its task identifier, without traversing the task waiting queue or the set of running tasks. This full task mapping set can be implemented using a key-value pair mapping (such as a hash table or dictionary), directly indexing the corresponding task object using the task identifier as the key, achieving constant-time task lookup. In scenarios with dozens or more download tasks executing concurrently, without this mapping set, each response to a user's pause or resume operation for a specific task would require sequentially traversing the task waiting queue and the set of running tasks to find the target task, causing lookup efficiency to decrease linearly with the number of tasks. This mechanism, by introducing an independent mapping structure, achieves a significant reduction in UI response latency with minimal memory overhead, ensuring a smooth user experience.

[0043] As a preferred implementation, before downloading the video segment from the storage path to the local cache, this method further includes the following security processing steps: reading the encryption identifier corresponding to the video segment recorded in the file manifest information; if the encryption identifier indicates that the video segment is encrypted, extracting the corresponding decryption key from the file manifest information; after the video segment is downloaded to the local cache, using the decryption key to perform a decryption operation on the video segment, so that the local cache stores decrypted video segment data. This step pushes the decryption operation from the server to the client's local execution. The key required for decryption is pre-encapsulated in the file details table of the local persistent index database during the offline export stage, so the entire decryption process does not rely on real-time key distribution or online authentication on the server. For unencrypted video segments, the decryption step is skipped directly. This on-demand decryption processing method, while ensuring the security of encrypted storage of data in the cloud, forms a consistent technical closed loop with the offline operation capability of the overall solution. Both downloading and decryption can be completed independently in an environment detached from the business system.

[0044] As a preferred implementation, this method also includes a classification and fault-tolerant processing for download failure events: when a download failure occurs during the download of the video segments, the cause of the download failure is first determined, and the cause of failure is classified into at least two categories: network anomaly and file corruption or path failure due to non-network factors.

[0045] If the failure is determined to be due to a network anomaly, a network failure message is sent to the user, and a periodic network connectivity check process for the object storage service is initiated. Once network recovery is detected, the download task is automatically resumed. Network anomalies typically manifest as TCP connection timeouts, DNS resolution failures, or unresponsive HTTP requests; these types of failures are temporary and recoverable. This method does not directly classify network anomalies as task failures. Instead, it continuously monitors the network status through a background heartbeat mechanism and automatically resumes the download once the network recovers, avoiding the inconvenience of manual monitoring and recovery by the user.

[0046] If the failure is determined to be due to file corruption or path invalidation (not network-related), the failure information for the current video segment is recorded in the local log, the current download progress index is incremented to skip the current video segment, and the download of subsequent video segments in the download task continues. Failures due to non-network factors (such as accidental deletion of a segment file in object storage, path changes, or file data corruption) typically cannot be resolved by retries. For such failures, this method adopts a fast-failure strategy of recording, skipping, and continuing, abandoning only the individual segment that cannot be downloaded at the moment, without blocking the entire task. Although there may be missing footage at individual points in the final generated recording file, compared to the result of a single point of failure causing the entire long-duration task to be unusable, this strategy achieves data delivery under the principle of "maximum availability," significantly improving the success rate of long-duration recording download tasks in real-world network environments.

[0047] As a preferred implementation, this method employs a dual-path storage mechanism to organize local cached data and final output data: raw video segments downloaded from the object storage service are stored in the download temporary path; the target format video files generated after decryption and format encapsulation are stored in the video storage path, which is distinct from the download temporary path. This storage design, which physically isolates intermediate temporary data from the final product, brings convenience in data management and space maintenance. When the client exits or the user clears the download cache, the entire download temporary path can be cleared directly without affecting the encapsulated video files; simultaneously, the files in the video storage path are all in standard formats that users can directly play, facilitating viewing, copying, or archiving operations, and avoiding accidental deletion or omissions caused by the mixed storage of intermediate temporary files and the final product.

[0048] As a preferred implementation, this method refines the steps for detecting whether the download task meets preset completion conditions and performing format encapsulation: when the number of video segments downloaded to the local cache reaches a preset merging threshold, decryption and format encapsulation are performed on the preset number of video segments to generate a recording file in the target format; when all video segments in the download task have been downloaded and the number of remaining unmerged video segments has not reached the merging threshold, decryption and format encapsulation are forcibly triggered on the remaining unmerged video segments to generate a final recording file in the target format. The preset merging threshold can be flexibly set according to configuration, for example, set to trigger a merging and encapsulation operation every 10 segments. The significance of this batch merging strategy is that for a long-duration recording download task containing hundreds of video segments and a total duration of several hours, if it is necessary to wait for all segments to be downloaded before unified encapsulation, users would have to wait for several hours to view any playable recording content. By merging in batches, the system can output individual, playable video files in the target format as the download progresses. Users can start viewing downloaded video segments without waiting for the entire task to complete, significantly reducing the waiting time from starting the download to obtaining usable video results. Furthermore, for cases where the last batch in the task might not meet the merging threshold, a final encapsulation operation is forcibly triggered, ensuring that all downloaded segments are converted into playable video files, without generating unusable residual data.

[0049] Example 2 This embodiment provides supplementary explanations of the specific execution logic and thread scheduling mechanism of the method described in Embodiment 1 from an engineering implementation perspective. This embodiment is based on the same inventive concept as Embodiment 1, and its technical features can be combined with those of Embodiment 1.

[0050] The overall architecture of this embodiment is divided into two stages: the first stage is the pre-preparation stage, which generates a local persistent index database file named vod.db through an offline export tool; the second stage is the download and execution stage, which loads the vod.db file, parses the configuration information and file list encapsulated therein, and performs video segment download, decryption, merging and format encapsulation operations.

[0051] Phase 1: Offline export to generate vod.db file; During the pre-processing phase, the export tool first identifies the database storing video recording metadata in the business system and retrieves the object storage service connection information from this database, including the object storage service access address, port number, and authentication credentials, thereby establishing a connection with the object storage service. Based on the target device information input by the user (including device identifier PUID, channel identifier IDX, device name, etc.) and the query time period (the start and end times of recording), the export tool retrieves a list of all video segment objects stored by the object storage service within that time period from the business system's video recording metadata database and organizes the retrieval results into the following three-layer data structure.

[0052] The first layer generates a task overview table (vodtaskinfo), which records the basic information of the exported tasks. The format of each field is as follows: id is the task sequence number, puid is the device identifier, idx is the channel identifier, puname is the device name, begintime is the recording start time, endtime is the recording end time, totalfilecnt is the total number of files to be downloaded, and currentidx is the index of the currently downloaded files, with an initial value of zero.

[0053] The second layer generates a corresponding file details table for each record in the task overview table, recording detailed information about each video segment included in the task. The name of the file details table is dynamically concatenated based on the `puid`, `idx`, and `id` fields in the task overview table, specifically in the format "puid_idx_id". Each file details table contains the following fields: `id` (the index number of the recording file in the table), `name` (the name of the recording file), `path` (the storage path of the recording file in the object storage service), `secretflag` (an identifier indicating whether the recording file is encrypted), `secretkey` (the decryption key for encrypted files), `begintime` (the start timestamp of the recording file), `endtime` (the end timestamp of the recording file), and `size` (the data size of the recording file).

[0054] The third layer generates a storage configuration table (storageinfo), which specifically records all connection information for the object storage service, including the object storage service access address (obs_url), port number (obs_port), authentication credentials (AK / SK), and bucket name (Bucket).

[0055] The above three data tables are stored in the vod.db file. vod.db, as a complete local persistent index database, encapsulates all the metadata required for subsequent download execution. This vod.db file can be generated once by an export tool and can be independently loaded and reused by one or more download tools. Users only need to obtain this vod.db file to initiate a download task in any client environment.

[0056] Phase Two: Video Download Execution Process; During the download execution phase, the video download tool operates according to the following steps.

[0057] First, the configuration file information is read to obtain the merging parameters, which specify how many video segments should be merged and packaged into a single target format video file (such as an MP4 file). These merging parameters determine the final number of target video files generated by a download task—a download task may contain multiple target format video files, and each target format video file is composed of one or more video segments merged and packaged.

[0058] Next, load the vod.db file. First, read the information from the storageinfo table to obtain key parameters such as the access address, port number, authentication credentials (AK / SK), and bucket name of the object storage service. Use these parameters to initialize and establish the S3 client, completing the connection preparation work with the object storage service. Then, read the information from the vodtaskinfo table and extract the following from each record: id (as the task identifier taskId for the download task, and to establish a binding relationship with the task list item in the user interface), puname (as the task name in the user interface), begintime and endtime (as the recording time period displayed in the user interface; the database storage format needs to be converted to a user-readable format), totalfilecnt (as the total number of files displayed in the user interface), and currentidx (the index of the currently downloaded file, used to indicate which video segment the task starts downloading from in the file details table).

[0059] If the storageinfo or vodtaskinfo table fails to be read, an error message will pop up, informing the user that the configuration file loading failed and prompting them to check the loaded configuration.

[0060] After all tasks are successfully loaded, the user interface will render and display each download task. When a user clicks on a download task, the system executes the following specific logic: It determines the target task based on the clicked task item, retrieves the corresponding id, puid, and idx from the task overview table; it combines the id, puid, and idx according to the concatenation rule "task_puid_idx_id" to obtain the file details table name corresponding to the task; it locates the corresponding file details table in the vod.db database based on this table name, reads the information from each field in the table, and stores all records in a container of a VideoFileInfo structure as a complete list of files to be downloaded later.

[0061] After a download task is created, it is allocated and managed by a download thread pool. The maximum number of download threads in the thread pool is preset to five. The specific scheduling logic is as follows, combined with... Figure 2 The overall download process is explained using a flowchart.

[0062] Start the download process (see) Figure 2 (Download Startup Flowchart) The thread pool management class provides a download interface. Tasks are added to the task queue `m_JobQueue`, following the FIFO principle. All task information is stored using `taskid` as the key, facilitating subsequent pauses and resumptions of download operations. The class checks for idle threads: if there are no idle threads and the thread count has not reached the maximum, a new download thread is created, added to the thread pool, and begins execution; otherwise, if there are no idle threads and the maximum thread count has been reached, the class waits for a thread to execute the task.

[0063] After the worker thread starts, it executes `CWorkerThread::Run()`. It checks if there are any tasks in the `m_JobQueue` queue. If so, it executes a FIFO (First-In, First-Out) process and begins downloading the task. The task status changes from `JOB_STATUS_READY` to `JOB_STATUS_RUNNING`, and it is removed from `m_JobQueue` and moved to `m_RunningQueue`. After downloading, the task is removed from `m_RunningQueue`, and an attempt is made to find the next task in `m_JobQueue`. If there are no tasks in `m_JobQueue`, the thread is suspended, waiting to be awakened by a new download task.

[0064] Let's take a look at the specific execution process of the download task: First, create a temporary download path and a video storage path. The temporary download path is used to store the raw video segments downloaded from the S3 service (such as .x1w format files). This path also serves as the processing path for subsequent decryption operations and merging a specified number of video segments into an intermediate file. The video storage path is used to store the target format video file (such as an MP4 format file) generated after the intermediate file has been format-encapsulated, for users to watch directly using a media player.

[0065] Then, the synchronous download operation is performed using the created S3 client. Based on the file order recorded in the VideoFileInfo container and the current download progress index currentidx, the storage paths of the video segments to be downloaded in the object storage service are determined sequentially, and the download is performed through the established direct communication link.

[0066] Upon successful download, the index of the currently downloaded file is recorded, and the file information is stored in the merge queue (mergeList). Then, the VideoFileInfo container is traversed to find the next recording file to be downloaded. When the number of successfully downloaded video segments reaches a preset merge threshold, decryption is initiated—each video segment has an encrypted or unencrypted status indicator, which can be obtained through the encryption identifier field and the corresponding decryption key field in the VideoFileInfo structure. After decryption, a multimedia processing interface (such as ffmpeg) is used to merge and encapsulate the multiple decrypted video segments into a single recording file of the target format. The process_ variable records the current download task's progress (download, decryption, and merging processes) to accurately restore the execution progress when the task resumes after a pause. If the currently downloaded successfully downloaded video segment is the last one in the task, the subsequent decryption and merging / encapsulation processes are forcibly executed regardless of whether the number of downloaded segments has reached the preset merge threshold. After merging and encapsulation, the number of currently downloaded files is updated to the user interface, and the updated current download progress index (currentidx) is written to the vod.db database. The purpose of this design is to enable the download of recording files from the position indicated by the persistent currentidx index when the client is closed and reopened, thus achieving resume recording.

[0067] In the event of a download failure, differentiate the reasons for the failure and handle them accordingly.

[0068] If the failure is due to a network error, the user is notified that the download failed because of a network issue, and a network connectivity check for the S3 service is initiated. Once the network is restored, the download process is restarted based on the currently running download tasks in the user interface, and execution is resumed using the saved data records in the task (including the previous download index and download process) to avoid starting from scratch.

[0069] If the failure is due to non-network factors (such as file corruption or invalid path), the successfully downloaded files currently stored in mergeList will be decrypted and merged, the user interface will be updated, and the current progress will be written to the currentidx field of the vod.db database. Simultaneously, the file information of the currently failed video segment will be output as a log record, the currently failed segment will be skipped, and the download of the next video segment will continue.

[0070] Once all records in the VideoFileInfo container have been traversed, it indicates that the download task has been completed. The task status is then set to completed (JOB_STATUS_FINISHED), and the task is removed from the RunningJobVector collection.

[0071] After a single task download is complete, the system checks the task waiting queue (m_JobQueue). If there are still download tasks waiting to be executed in the queue, the task at the head of the queue is started, its status is set to running, it is removed from m_JobQueue and added to the running task set (m_RunningQueue), and the download begins according to the same process described above. If there are no ready tasks in m_JobQueue, the download thread containing that task is suspended until it is awakened when a new download task arrives.

[0072] Pause the download process (see Figure 2 (Download pause process line) The thread pool management class provides a download interface. It locates the current job by searching the task ID and sets the current job's status to JOB_STATUS_PAUSED. If the current target task's status is already ready (JOB_STATUS_READY), it is directly removed from JobVector.

[0073] If the current target task is in a running state (JOB_STATUS_RUNNING), the DownloadJob will monitor the current state when it reaches this task. If it is in a JOB_STATUS_PAUSED state, it will exit directly, saving the index of the current download task. The task will be removed from the RunningJobVector, and the corresponding thread will search for the next task in the JobVector to continue downloading. If there are no tasks in the JobVector, the thread will be suspended.

[0074] Resume the download process (see) Figure 2 (Download recovery process line in the middle) The thread pool management class provides a download interface. It locates the current job by searching the task ID and checks if the task can be found in the RunningJobVector.

[0075] If the task is found in RunningJobVector, the current job status is set to JOB_STATUS_RUNNING, and the current download task continues. There's no need to wait for an idle thread, as this download thread is already occupied by this task. This is because S3 file downloads cannot be truly interrupted; they can only be interrupted once the download is complete. Pausing and resuming operations while waiting for the download to finish are considered as if no pause was performed.

[0076] If the task is not found in RunningJobVector, the current job status is set to JOB_STATUS_READY, the task is added back to the ready queue, its status is changed to ready, and it waits for an idle thread to call the task for download. This means that after a file download is paused, the priority of the download task will drop to the lowest level, and the download will begin after the previous tasks that needed to be downloaded are completed. When this recovery task is executed, the system reads the previously saved data to find which process (download, decryption, or merging) the file belongs to, and then performs the corresponding action in the corresponding process. If it is in the download process, it will read the index of the already downloaded files and start downloading from the next file.

[0077] It's important to note that the paused and saved task status is stored in a cache, and this record is cleared and deleted when the client exits. However, upon restarting the client, because the download progress was written to the vod.db file via a transaction before the pause, this progress record is saved, allowing the download to resume from the currentidx index position recorded in vod.db. Recording and downloading tools typically store the task's download progress in the vod.db file.

[0078] In this embodiment, the lifecycle of each download task is defined by the following four states: JOB_STATUS_READY indicates the ready state, that is, the task has been created and is queued in the task waiting queue; JOB_STATUS_RUNNING indicates the running state, that is, the task has been allocated a thread and is executing the download; JOB_STATUS_PAUSED indicates the paused state, that is, the task has been paused by the user; JOB_STATUS_FINISHED indicates the completed state, that is, all video segments of the task have been downloaded and packaged.

[0079] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for recording video download based on local index persistence and direct connection with object storage, characterized in that, include: Obtain a pre-generated local persistent index database, wherein the local persistent index database pre-encapsulates the connection configuration information of the object storage service, the file list information of all video segments included in the download task, and the current download progress index; Parse the connection configuration information to establish a direct communication link between the client and the object storage service; The download tasks are managed by a limited download thread pool, and the download tasks are transferred between the task waiting queue and the set of running tasks according to the task status; Based on the file list information and the current download progress index, the storage path of the video segment to be downloaded in the object storage service is determined, and the execution thread is obtained from the running task set through the direct communication link to execute the download of the video segment; After the video segment is successfully downloaded, the current download progress index in the local persistent index database is updated in a transactional manner so that the updated current download progress index points to the next video segment to be downloaded; The system checks whether the download task meets the preset completion conditions. If so, it calls the multimedia processing interface to perform format encapsulation processing on the multiple video segments that have been downloaded to the local cache, and generates a video file in the target format.

2. The method of claim 1, wherein, The local persistent index database is pre-generated from the business system by an offline export tool and can be reused by one or more download tools; the local persistent index database includes: The storage configuration table records the access address, port number, access credentials, and bucket name of the object storage service. The task overview table records the task identifier, device identifier, channel identifier, total number of files, and the current download progress index; The file details table records the storage path, timestamp, encryption identifier, and decryption key of each video segment in the object storage service. The name of the file details table is dynamically generated by combining the device identifier, the channel identifier, and the task identifier.

3. The method of claim 1, wherein, The download tasks are managed by a limited download thread pool, and the download tasks are transferred between the task waiting queue and the set of running tasks according to the task status, specifically including: Construct a limited download thread pool with a preset maximum number of threads, and enqueue the download tasks to be executed into the task waiting queue in the order they were acquired; When there is an idle download thread, the idle download thread extracts a download task from the task waiting queue, changes the task status of the download task from the ready state to the running state, and stores the download task in the running task set; When there are no idle download threads and the number of currently running threads has not reached the preset maximum number of threads, a new download thread is created to execute the download task; When there are no idle download threads and the number of currently running threads has reached the preset maximum number of threads, the task status of the download task is kept in the ready state and retained in the task waiting queue.

4. The method according to claim 3, characterized in that, It also includes download task pause handling logic in response to pause commands: Receive a pause command for a target running task and mark the task status of the target running task as paused; After the video segment currently being downloaded by the target task is completely downloaded to the local cache and the current download progress index is updated, the execution of the download thread corresponding to the target task is terminated, and the target task is removed from the set of running tasks.

5. The method according to claim 4, characterized in that, It also includes download task recovery processing logic in response to recovery commands: Receive the command to resume the paused task for the target; Determine whether the target paused task still exists in the set of running tasks; If it exists, the task status of the target paused task will be changed directly from paused to running, skipping the re-queueing and thread allocation process, and continuing to execute the segmented download; If it does not exist, the target paused task is added back to the end of the task waiting queue, its execution priority is placed after other ready tasks in the task waiting queue, and its task status is changed to ready, so that it can be extracted and executed in sequence when there is an idle download thread.

6. The method according to claim 4 or 5, characterized in that, The method further includes: Maintain a full task mapping set, storing all started download tasks in the full task mapping set with their respective task identifiers as keys; When a pause or resume command is received for a specific target task, the target task can be directly located in the full task mapping set by the task identifier of the target task, without traversing the task waiting queue or the set of running tasks.

7. The method according to claim 1, characterized in that, Before downloading the video fragments from the storage path to the local cache, the process also includes: Read the encryption identifier corresponding to the video segment recorded in the file list information; If the encryption identifier indicates that the video segment is encrypted, then the corresponding decryption key is extracted from the file list information; After the video segment is downloaded to the local cache, the decryption key is used to perform a decryption operation on the video segment, so that the local cache stores the decrypted video segment data.

8. The method according to claim 1, characterized in that, Updating the current download progress index in the local persistent index database in a transactional manner includes: After each video segment is successfully downloaded, a database transaction is initiated to write the updated current download progress index into the local persistent index database. This ensures that if the client exits abnormally and restarts, the download can continue from the next video segment indicated by the persistent current download progress index.

9. The method according to claim 1, characterized in that, It also includes fault tolerance handling for download failure events: When a download failure occurs during the download of the video segment, the reason for the download failure is determined. If the failure is determined to be due to a network anomaly, a network failure message is sent to the user, and a periodic network connectivity detection process for the object storage service is initiated; once the network is detected to have recovered, the download task is automatically resumed. If the failure is determined to be due to file corruption or path invalidation that is not caused by network factors, the failure information of the current video segment is recorded in the local log, the current download progress index is incremented to skip the current video segment, and the download of subsequent video segments in the download task continues.

10. The method according to any one of claims 1 to 9, characterized in that, The system checks whether the download task meets preset completion conditions. If so, it calls the multimedia processing interface to perform format encapsulation processing on the multiple video segments already downloaded to the local cache, generating a video file in the target format, including: When the number of video segments downloaded to the local cache reaches a preset merging threshold, the preset number of video segments are decrypted and format-encapsulated to generate a video file in the target format. When all video segments in the download task have been downloaded and the number of remaining unmerged video segments has not reached the merging threshold, the decryption and format encapsulation processes of the remaining unmerged video segments are forcibly triggered to generate the last target format recording file.

Images