A method, device and storage medium for preventing repeated downloading of a media processing system

Abstract

The application discloses a kind of methods, devices and storage medium for preventing media processing system from repeated downloading, and relates to the technical field of media processing.The method for preventing media processing system from repeated downloading uses file Etag to uniquely identify file, and Etag value of media file is obtained each time when downloading, then it is judged whether corresponding Etag file exists on shared disk, if not, it is downloaded, otherwise it is not necessary to download;Fusion multi-factor design competition elimination strategy is realized using bidirectional file management queue;Active cleaning, passive cleaning in combination of multiple ways clean disk space, achieve excellent effect of preventing media file from repeated downloading, solve the problem that existing technology based on NFS protocol for sharing, direct timing deletion exists difficult to effectively prevent media file from repeated downloading, disk space cleaning is not efficient, not timely, realize the effect of effectively preventing media file from repeated downloading for conveniently and efficiently cleaning shared disk space.

Description

Technical Field

[0001] This invention relates to the field of media processing technology, and in particular to a method, apparatus, and storage medium for preventing repeated downloads in a media processing system. Background Technology

[0002] With the proliferation of video platforms worldwide and the explosive growth of various video formats, the demand for media processing such as transcoding, screenshotting, watermarking, and AI review is booming. However, before performing any specific media processing, the media processing system must first download the source video completely from S3 storage to its local machine, and then perform certain processing based on specific media processing parameters. Because various media processing tasks are submitted to the media processing system asynchronously as independent tasks, this leads to frequent and repeated downloads of the source video or watermarks when users perform different media processing on the same video at different times. If the source video is large and there are many users performing similar time-segmented processing, frequent and repeated downloads not only waste significant bandwidth resources and increase the S3 storage I / O load, but also severely impact the overall efficiency of media processing.

[0003] The industry practice for addressing the issue of duplicate media file downloads is as follows: Machines within the service cluster use the NFS (Network Fibre System) protocol to mount a shared disk within the cluster. Downloaded media files are directly saved to this shared disk using their s3 keys. When the shared disk space is insufficient, files are cleaned up sequentially by s3 key, download time, or size. This method has the following significant problems: 1. When two media files have identical content but different s3 keys, they will still be downloaded repeatedly; 2. Media files with identical s3 keys but different content may be mistakenly processed; 3. The cleanup is too forceful and cannot effectively prevent duplicate downloads; 4. Disk space cleanup is inefficient and untimely. Summary of the Invention

[0004] This application provides a method, apparatus, and storage medium for preventing duplicate downloads of media files in a media processing system. This solves the problems of ineffective prevention of duplicate downloads of media files and inefficient and untimely disk space cleanup in the prior art based on NFS protocol sharing and direct timed deletion. It achieves the effect of facilitating efficient cleanup of shared disk space and effectively preventing duplicate downloads of media files.

[0005] This application provides a method for preventing repeated downloads in a media processing system, including the following steps:

[0006] S1. Determine whether the current media file needs to be downloaded to the shared disk based on the Etag value; if so, download it.

[0007] S2. Based on a multi-factor competitive elimination strategy, a two-way queue is used to manage files, and active and passive cleanup are combined to realize the utilization and cleanup of shared disk space.

[0008] Furthermore, the specific process of S1 determining whether the current media file needs to be downloaded includes:

[0009] S11. Store the newly downloaded media file on the shared disk with the name {Etag}.{format};

[0010] S12. Obtain the Etag value of the corresponding media file before downloading;

[0011] S13. Determine if the obtained Etag value exists on the shared disk. If it does not exist, download it.

[0012] Furthermore, the multiple factors in S2 include file size, access count n, and the time of each access. These factors determine the local file cleanup priority, and the equivalent number of bytes accessed per minute is defined as Fvbpm.

[0013]

[0014] Where, Δt expire-days This represents the total expiration time in minutes for local media files, as specified in accessTim. n The last access time is specified, and fileSize is the file size.

[0015] Furthermore, when using a bidirectional queue to manage files in S2, the nodes are arranged in descending order based on the last access time and the equivalent number of bytes accessed per minute.

[0016] Queue operations use head pointers, current pointers, and tail pointers to implement file addition, deletion, updating, and viewing;

[0017] The head node is a system-reserved node that records the total size of the current local media files.

[0018] Furthermore, the elimination strategy in S2 includes a background resident queue management thread and a background resident queue cleanup thread. The background resident queue management thread is used for adding and updating queue nodes, and the background resident queue cleanup thread is used for actively cleaning up expired nodes.

[0019] Furthermore, the methods for cleaning up shared disk space in S2 include active cleaning and passive cleaning.

[0020] Furthermore, the active cleaning process specifically includes:

[0021] Step 1: The background resident queue cleanup thread traverses the queue in reverse order from the tail node every hour.

[0022] The second step is to determine whether a node has expired based on the last access time, and then clean up expired nodes and their corresponding media files in the queue in turn.

[0023] Step 3: Update the total number of bytes in the head node.

[0024] Furthermore, the passive cleanup process specifically includes:

[0025] The first step is to obtain the metadata length of a new media file before downloading it.

[0026] The second step is to determine whether there is enough remaining space on the shared disk. If not, start from the tail node of the queue and traverse backwards to forcibly clean up the nodes that have been traversed until there is enough disk space.

[0027] This application provides an apparatus for preventing repeated downloads in a media processing system, comprising a download module and a cleanup module, wherein:

[0028] The download module is used to determine whether the current media file needs to be downloaded to the shared disk based on the Etag value; if so, it is downloaded.

[0029] The cleanup module is used to utilize and clean up shared disk space based on a multi-factor competitive elimination strategy, using a two-way queue to manage files, and combining active and passive cleanup.

[0030] This application provides a storage medium storing a stored program, which, when executed by a processor, implements the method for preventing repeated downloads by a media processing system.

[0031] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0032] Files are uniquely identified using ETag. Each time a media file is downloaded, its ETag value is obtained first, and then it is determined whether the corresponding ETag file exists on the shared disk. If it does not exist, it is downloaded; otherwise, it is not downloaded. A competitive elimination strategy is designed by incorporating multiple factors and implemented using a two-way file management queue. Disk space is cleaned up by combining active and passive cleanup methods, achieving excellent results in preventing duplicate downloads of media files. Attached Figure Description

[0033] Figure 1 This is a flowchart of a method for preventing repeated downloads in a media processing system provided in Embodiment 1 of this application;

[0034] Figure 2 This is a schematic diagram of a system for determining whether a new media file needs to be downloaded, provided in Embodiment 1 of this application.

[0035] Figure 3 This is a schematic diagram of the specific queues and system threads for bidirectional queue management provided in Embodiment 1 of this application;

[0036] Figure 4 This is a flowchart illustrating the active cleanup process provided in Embodiment 1 of this application.

[0037] Figure 5 This is a flowchart illustrating the specific process of passive cleanup provided in Embodiment 1 of this application;

[0038] Figure 6 This is a structural diagram of a device for preventing repeated downloads of a media processing system provided in Embodiment 2 of this application. Detailed Implementation

[0039] This application provides a method, apparatus, and storage medium for preventing duplicate downloads in a media processing system. It solves the problems of existing technologies that rely on NFS protocol-based sharing and direct timed deletion, which are difficult to effectively prevent duplicate downloads of media files and have inefficient and untimely disk space cleanup. The media cleanup system uses file ETag unique identifiers and integrates a multi-factor competitive elimination strategy to achieve efficient cleanup of shared disk space and effectively prevent duplicate downloads of media files.

[0040] The technical solution in this application embodiment aims to address the problem of improving retrieval speed while ensuring recognition accuracy in the existing deep learning-based retrieval methods. The overall approach is as follows:

[0041] During massive media processing, frequent and repeated downloads of media files not only waste significant bandwidth resources and increase the S3 storage I / O load, but also severely impact the overall efficiency of media processing. To address these issues, a file ETag is used to uniquely identify files. Each time a media file is downloaded, its ETag value is obtained first, and then it is determined whether the corresponding ETag file exists on the shared disk. If it does not exist, it is downloaded; otherwise, it is not downloaded. A multi-factor competitive elimination strategy is designed and implemented using a bidirectional file management queue. A combination of active and passive cleanup methods is used to clean up disk space, achieving excellent results in preventing repeated downloads of media files.

[0042] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0043] Example 1

[0044] like Figure 1 The diagram shown is a flowchart of a method for preventing duplicate downloads in a media processing system according to an embodiment of this application. The method includes:

[0045] S1. Determine whether the current media file needs to be downloaded to the shared disk based on the Etag value; if so, download it.

[0046] S2. Based on a multi-factor competitive elimination strategy, a two-way queue is used to manage files, and active and passive cleanup are combined to realize the utilization and cleanup of shared disk space.

[0047] Furthermore, the method provided in this application embodiment can facilitate efficient cleanup of shared disk space and effectively prevent duplicate downloads of media files. The specific steps are as follows:

[0048] like Figure 2 As shown, the specific process of S1 determining whether the current media file needs to be downloaded includes:

[0049] S11. Store the newly downloaded media file on the shared disk with the name {Etag}.{format};

[0050] S12. Obtain the Etag value of the corresponding media file before downloading;

[0051] S13. Determine if the obtained Etag value exists on the shared disk. If it does not exist, download it.

[0052] In this embodiment, a file ETag is used to uniquely identify the file. The ETag is a file hash value that accurately reflects changes in file content and is read-only. Newly downloaded media files are named {ETag}.{format} and stored on the shared disk. Whenever a media file is downloaded, the ETag value of the media file is obtained first. Based on the obtained ETag value, it is determined whether the corresponding file exists on the shared disk. If it exists, there is no need to download the media file again, thus avoiding duplicate downloads.

[0053] Furthermore, the multiple factors in S2 include file size, number of accesses, and the time of each access. These factors determine the local file cleanup priority, and the equivalent number of bytes accessed per minute is defined as Fvbpm.

[0054]

[0055] Where, Δt expire-day This represents the total expiration time in minutes for local media files, as specified in accessTim. n The last access time is specified, and fileSize is the file size.

[0056] Furthermore, when using a bidirectional queue to manage files in S2, the nodes are arranged in descending order based on the last access time and the equivalent number of bytes accessed per minute.

[0057] Queue operations use head pointers, current pointers, and tail pointers to implement file addition, deletion, updating, and viewing;

[0058] The head node is a system-reserved node that records the total size of the current local media files;

[0059] Furthermore, the elimination strategy in S2 includes a background resident queue management thread and a background resident queue cleanup thread. The background resident queue management thread is used for adding and updating queue nodes, and the background resident queue cleanup thread is used for actively cleaning up expired nodes.

[0060] In this embodiment, as Figure 3 As shown, an ordered bidirectional queue is used to manage files, with nodes ordered by accessTime. n The queue is sorted in descending order in conjunction with Fvbpm. The head node is a pre-reserved node of the system, which records the total size of the current local media files. In order to balance the overall efficiency of CRUD (Create, Read, Update, Delete), the queue operation uses three pointers: head pointer, current pointer, and tail pointer.

[0061] Fvbpm is obtained through the formula

[0062] We obtain Δt expire-days The total number of minutes until the local media files expire, accessTime n The last access time is specified, fileSize is the file size, and the expiration time is a system environment variable that can be flexibly set; the default is 14 days. accessTime is the access time. n Accurate to the minute.

[0063] Furthermore, the methods for cleaning up shared disk space in S2 include active cleaning and passive cleaning.

[0064] Furthermore, the active cleaning process specifically includes:

[0065] Step 1: The background resident queue cleanup thread traverses the queue in reverse order from the tail node every hour.

[0066] The second step is to determine whether a node has expired based on the last access time, and then clean up expired nodes and their corresponding media files in the queue in turn.

[0067] Step 3: Update the total number of bytes in the head node.

[0068] In this embodiment, as Figure 4 As shown, the resident cleanup thread traverses the queue backwards from the tail node every hour. First, it checks if a previous data node exists. If not, the backward traversal ends immediately. If a previous data node exists, it checks the accessTime... nDetermine if the node has expired, i.e. how many days it has not been accessed, remove the expired node from the queue, clean up the corresponding media files, and update the total number of bytes in the header node.

[0069] Furthermore, the passive cleanup process specifically includes:

[0070] The first step is to obtain the metadata length of a new media file before downloading it.

[0071] The second step is to determine whether there is enough remaining space on the shared disk. If not, start from the tail node of the queue and traverse backwards to forcibly clean up the nodes that have been traversed until there is enough disk space.

[0072] In this embodiment, as Figure 5 As shown, when a new media file needs to be downloaded, its metadata Content-Length is obtained first. If the remaining space on the shared disk is insufficient, the queue is traversed in reverse order from the tail node until there is enough disk space. While deleting nodes and their corresponding files, the total number of bytes in the head node is also updated.

[0073] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages:

[0074] It can avoid duplicate downloads of media files with the same content but different s3Keys or mishandling of media files with different content but the same s3Key. After using the designed competitive elimination strategy, the deletion and retention of shared disk files is more accurate, improving the effective utilization of disk space. The combination of active and passive cleaning can achieve both timeliness and efficiency in cleaning.

[0075] Example 2

[0076] like Figure 6 The diagram shown is a structural diagram of a device for preventing duplicate downloads in a media processing system according to an embodiment of this application. The device includes a download module and a cleanup module, wherein:

[0077] The download module is used to determine whether the current media file needs to be downloaded to the shared disk based on the Etag value; if so, it is downloaded.

[0078] The cleanup module is used to utilize and clean up shared disk space based on a multi-factor competitive elimination strategy, using a two-way queue to manage files, and combining active and passive cleanup.

[0079] In this embodiment, before downloading a new media file, the download module will first determine whether the current media file needs to be downloaded to the shared disk based on the Etag value. If the media file does not exist on the shared disk, it will be downloaded if needed.

[0080] The cleanup module is based on a multi-factor competitive elimination strategy. The multi-factor factors mainly include file size, number of accesses, and the time of each access. It uses a two-way queue to manage files and combines active and passive cleanup to realize the utilization and cleanup of shared disk space.

[0081] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages:

[0082] It can avoid duplicate downloads of media files with the same content but different s3Keys or mishandling of media files with different content but the same s3Key. After using the designed competitive elimination strategy, the deletion and retention of shared disk files is more accurate, improving the effective utilization of disk space. The combination of active and passive cleaning can achieve both timeliness and efficiency in cleaning.

[0083] This application also provides a storage medium storing a program that, when executed by a processor, implements a method for preventing repeated downloads by a media processing system.

[0084] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0089] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for preventing duplicate downloads in a media processing system, characterized in that, Includes the following steps: S1. Determine whether the current media file needs to be downloaded to the shared disk based on the Etag value; if so, download it. S2. Based on a multi-factor competitive elimination strategy, a two-way queue is used to manage files, and active and passive cleanup are combined to realize the utilization and cleanup of shared disk space; The specific process of S1 determining whether the current media file needs to be downloaded includes: S11. Store the newly downloaded media file on the shared disk with the name {Etag}.{format}; S12. Obtain the Etag value of the corresponding media file before downloading; S13. Determine if the obtained Etag value exists on the shared disk; if not, download it. The multiple factors in S2 include file size, access count n, and the time of each access. These factors determine the local file cleanup priority, and the equivalent number of bytes accessed per minute for a file is defined as... : ， in, This represents the total number of minutes until the local media files expire. The last access time, File size; When using a bidirectional queue to manage files in S2, nodes are sorted in descending order based on the last access time and the equivalent number of bytes accessed per minute; Queue operations use head pointers, current pointers, and tail pointers to implement file addition, deletion, updating, and viewing; The head node is a system-reserved node that records the total size of the current local media files; The elimination strategy in S2 includes a background resident queue management thread and a background resident queue cleanup thread. The background resident queue management thread is used for adding and updating queue nodes, and the background resident queue cleanup thread is used for actively cleaning up expired nodes.

2. The method for preventing duplicate downloads in a media processing system as described in claim 1, characterized in that: The methods for cleaning up shared disk space in S2 include active cleaning and passive cleaning.

3. The method for preventing duplicate downloads in a media processing system as described in claim 2, characterized in that: The active cleaning process is as follows: Step 1: The background resident queue cleanup thread traverses the queue in reverse order from the tail node every hour. The second step is to determine whether a node has expired based on the last access time, and then clean up expired nodes and their corresponding media files in the queue in turn. Step 3: Update the total number of bytes in the head node.

4. The method for preventing duplicate downloads in a media processing system as described in claim 2, characterized in that: The passive cleanup process is as follows: The first step is to obtain the length of the metadata of a new media file when it needs to be downloaded. The second step is to determine whether there is enough remaining space on the shared disk. If not, start from the tail node of the queue and traverse backwards to forcibly clean up the nodes that have been traversed until there is enough disk space.

5. A device for preventing repeated downloads in a media processing system, characterized in that, Includes a download module and a cleanup module, among which: The download module is used to determine whether the current media file needs to be downloaded to the shared disk based on the Etag value; if so, it is downloaded. The cleanup module is used to utilize and clean up shared disk space based on a multi-factor competitive elimination strategy, using a two-way queue to manage files, and combining active and passive cleanup. The specific process by which the download module determines whether the current media file needs to be downloaded includes: Store the newly downloaded media files on the shared disk with the name {Etag}.{format}; Obtain the ETag value of the corresponding media file before downloading; Check if the obtained ETag value exists on the shared disk; if not, download it. The cleaning module incorporates multiple factors, including file size, access count (n), and the time of each access. These factors determine the local file cleaning priority, and the equivalent number of bytes accessed per minute for a file is defined as... : ， in, This represents the total number of minutes until the local media files expire. The last access time, File size; When the cleanup module uses a bidirectional queue to manage files, the nodes are sorted in descending order based on the last access time and the equivalent number of bytes accessed per minute. Queue operations use head pointers, current pointers, and tail pointers to implement file addition, deletion, updating, and viewing; The head node is a system-reserved node that records the total size of the current local media files; The eviction strategy in the cleanup module includes a background resident queue management thread and a background resident queue cleanup thread. The background resident queue management thread is used for adding and updating queue nodes, and the background resident queue cleanup thread is used for actively cleaning up expired nodes.

6. A computer-readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the method for preventing repeated downloads of a media processing system as described in any one of claims 1 to 4.

Images