Methods, devices, electronic devices and computer-readable media for downloading media assets
By employing multi-threaded parallel downloading and a distributed lock mechanism, the problems of duplicate downloading and storage of media asset files were solved, improving download speed and quality inspection efficiency while saving resources.
Patent Information
- Application Number
- CN202311023277.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-14
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-08-14
AI Technical Summary
In existing technologies, the repeated downloading and storage of the same media asset files leads to a waste of network bandwidth, CPU computing power, and device storage resources, affecting system processing time and increasing the workload of quality inspection.
Session records are retrieved from the session server using a multi-threaded parallel approach, and a distributed lock mechanism is used to ensure that each media file is downloaded and stored only once by one thread. Redis is used to cache the storage address to avoid duplicate operations.
It reduces the use of network bandwidth and device storage resources, improves the download speed of media asset files, and saves quality inspection manpower and processing time.
Smart Images

Figure CN117082050B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data processing technology, and in particular to a method, apparatus, electronic device, and computer-readable medium for downloading media asset files. Background Technology
[0002] By retrieving chat logs from instant messaging sessions, the content of chats between operations staff and customers can be stored. The media asset files involved in the chat content are usually downloaded one by one using a media asset file download SDK.
[0003] In the process of realizing this invention, the inventors discovered at least the following problems in the prior art:
[0004] When operations personnel send the same images, videos, and other media asset files to all customers, each identical file needs to be downloaded and stored again. This results in a large number of duplicate network requests and a significant amount of physical storage space, consuming application service network bandwidth, CPU computing power, and device storage resources. Since downloading and processing a large number of identical media asset files takes time, it impacts the overall system processing efficiency. Furthermore, the repeated downloading of identical media asset files also leads to duplicate submissions for quality inspection, causing significant repetitive work for the content quality inspection system and personnel. Summary of the Invention
[0005] In view of this, embodiments of the present invention provide a method, apparatus, electronic device, and computer-readable medium for downloading media asset files, in order to solve the technical problem of consuming application service network bandwidth, CPU computing, device storage, and other resources.
[0006] To achieve the above objectives, according to one aspect of the present invention, a method for downloading media asset files is provided, comprising:
[0007] A multi-threaded parallel approach is used to retrieve session records from the session server. Each session record includes the signature value of the media asset file to be downloaded.
[0008] The media asset files to be downloaded from the session records are downloaded using a multi-threaded parallel approach. Specifically, for each media asset file to be downloaded in a session record, the thread that acquires the distributed lock of the media asset file to be downloaded downloads the media asset file from the session server, uploads the media asset file to be downloaded to the file system for storage, obtains the storage address of the media asset file to be downloaded, stores the storage address in the cache, and submits the storage address of the media asset file to be downloaded to the quality inspection platform.
[0009] Optionally, it also includes:
[0010] Other threads that fail to acquire the distributed lock of the media asset file to be downloaded obtain the storage address of the media asset file to be downloaded from the cache;
[0011] In response to the other threads failing to obtain the storage address of the media asset file to be downloaded from the cache, the other threads continue to wait;
[0012] In response to the other thread being able to obtain the storage address of the media asset file to be downloaded from the cache, the other thread submits the storage address of the media asset file to be downloaded to the quality inspection platform or terminates the process.
[0013] Optionally, the thread that acquires the distributed lock of the media asset file to be downloaded downloads the media asset file from the session server, including:
[0014] Based on the signature value of the media asset file to be downloaded, a distributed lock for the media asset file to be downloaded is generated and the distributed lock is stored in the cache. Each thread competes to acquire the distributed lock, so that the distributed lock is acquired by only one thread.
[0015] The thread that acquires the distributed lock downloads the media file to be downloaded from the session server, while other threads that have not acquired the distributed lock for the media file to be downloaded wait.
[0016] Optionally, each session record may also include the file format name suffix of the media asset file to be downloaded;
[0017] Based on the signature value of the media asset file to be downloaded, a distributed lock for the media asset file to be downloaded is generated, including:
[0018] A distributed lock for the media asset file to be downloaded is generated based on the signature value and file format name suffix of the file to be downloaded.
[0019] Optionally, each session record may also include the sender identifier of the media asset file to be downloaded;
[0020] Based on the signature value and file format name suffix of the media asset file to be downloaded, a distributed lock is generated for the media asset file to be downloaded, including:
[0021] The sender identifier, signature value, and file format name suffix of the media asset file to be downloaded are concatenated to generate a distributed lock for the media asset file to be downloaded.
[0022] Optionally, storing the distributed lock in a cache includes:
[0023] The distributed lock of the media asset file to be downloaded is stored in the cache using a key-value pair.
[0024] Wherein, key is the distributed lock of the media asset file to be downloaded, and value is a preset default value or a business parameter of the media asset file to be downloaded.
[0025] Optionally, the media asset file to be downloaded is uploaded to the file system for storage to obtain the storage address of the media asset file to be downloaded, and the storage address is stored in the cache, including:
[0026] The media asset file to be downloaded is uploaded to the file system for storage, thereby obtaining the storage address of the media asset file to be downloaded;
[0027] The storage address of the media asset file to be downloaded is stored in the cache using a key-value pair.
[0028] Wherein, key is the distributed lock of the media asset file to be downloaded, and value is the storage address of the media asset file to be downloaded.
[0029] In addition, according to another aspect of the present invention, a media asset file downloading device is provided, comprising:
[0030] The pull module is used to pull session records from the session server in a multi-threaded parallel manner. Each session record includes the signature value of the media asset file to be downloaded.
[0031] The download module is used to download the media asset files to be downloaded from the session records in a multi-threaded parallel manner. Specifically, for each media asset file to be downloaded in each session record, the thread that acquires the distributed lock of the media asset file to be downloaded downloads the media asset file to be downloaded from the session server, uploads the media asset file to be downloaded to the file system for storage, obtains the storage address of the media asset file to be downloaded, stores the storage address in the cache, and submits the storage address of the media asset file to be downloaded to the quality inspection platform.
[0032] Optionally, the download module is further configured to:
[0033] Other threads that fail to acquire the distributed lock of the media asset file to be downloaded obtain the storage address of the media asset file to be downloaded from the cache;
[0034] In response to the other threads failing to obtain the storage address of the media asset file to be downloaded from the cache, the other threads continue to wait;
[0035] In response to the other thread being able to obtain the storage address of the media asset file to be downloaded from the cache, the other thread submits the storage address of the media asset file to be downloaded to the quality inspection platform or terminates the process.
[0036] Optionally, the download module is further configured to:
[0037] Based on the signature value of the media asset file to be downloaded, a distributed lock for the media asset file to be downloaded is generated and the distributed lock is stored in the cache. Each thread competes to acquire the distributed lock, so that the distributed lock is acquired by only one thread.
[0038] The thread that acquires the distributed lock downloads the media file to be downloaded from the session server, while other threads that have not acquired the distributed lock for the media file to be downloaded wait.
[0039] Optionally, each session record may also include the file format name suffix of the media asset file to be downloaded;
[0040] The download module is also used for:
[0041] A distributed lock for the media asset file to be downloaded is generated based on the signature value and file format name suffix of the file to be downloaded.
[0042] Optionally, each session record may also include the sender identifier of the media asset file to be downloaded;
[0043] The download module is also used for:
[0044] The sender identifier, signature value, and file format name suffix of the media asset file to be downloaded are concatenated to generate a distributed lock for the media asset file to be downloaded.
[0045] Optionally, the download module is further configured to:
[0046] The distributed lock of the media asset file to be downloaded is stored in the cache using a key-value pair.
[0047] Wherein, key is the distributed lock of the media asset file to be downloaded, and value is a preset default value or a business parameter of the media asset file to be downloaded.
[0048] Optionally, the download module is further configured to:
[0049] The media asset file to be downloaded is uploaded to the file system for storage, thereby obtaining the storage address of the media asset file to be downloaded;
[0050] The storage address of the media asset file to be downloaded is stored in the cache using a key-value pair.
[0051] Wherein, key is the distributed lock of the media asset file to be downloaded, and value is the storage address of the media asset file to be downloaded.
[0052] According to another aspect of the present invention, an electronic device is also provided, comprising:
[0053] One or more processors;
[0054] Storage device for storing one or more programs.
[0055] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.
[0056] According to another aspect of the present invention, a computer-readable medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the methods described in any of the above embodiments.
[0057] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods described in any of the above embodiments.
[0058] One embodiment of the above invention has the following advantages or beneficial effects: Because it employs a distributed lock on the media asset file to be downloaded, which is then downloaded from the session server by a thread that acquires the distributed lock, uploaded to the file system for storage, its storage address is obtained, stored in a cache, and submitted to the quality inspection platform, it overcomes the technical problems of consuming application service network bandwidth, CPU computing, and device storage resources in the prior art. This embodiment of the invention uses a distributed lock to download and store the same media asset file only once, avoiding repeated processing of the same media asset file. This reduces the consumption of network resources such as application service network bandwidth and saves machine resources such as CPU computing and device storage, thereby improving the download speed of media asset files. Furthermore, since the storage address of the same media asset file is submitted to quality inspection only once, it not only saves quality inspection manpower but also improves the quality inspection processing speed.
[0059] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description
[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:
[0061] Figure 1 This is a flowchart of a media asset file download method according to an embodiment of the present invention;
[0062] Figure 2This is a schematic diagram of the system architecture of a media asset file download method according to an embodiment of the present invention;
[0063] Figure 3 This is a flowchart of a media asset file download method according to a possible embodiment of the present invention;
[0064] Figure 4 This is a flowchart of a media asset file download method according to another applicable embodiment of the present invention;
[0065] Figure 5 This is a schematic diagram of a media asset file downloading device according to an embodiment of the present invention;
[0066] Figure 6 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;
[0067] Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation
[0068] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0069] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.
[0070] Figure 1 This is a flowchart of a media asset file download method according to an embodiment of the present invention. As one embodiment of the present invention, such as... Figure 1 As shown, the media asset file download method is applied to the operating platform and may include:
[0071] Step 101: Use a multi-threaded parallel approach to retrieve session records from the session server. Each session record includes the signature value of the media asset file to be downloaded.
[0072] After the operations staff on the platform send images, videos, and other session content to end-users via the session server, the platform requests the session records from the session server. The session server then generates session records between the operations staff and the customer group. The operations platform uses a multi-threaded parallel approach to retrieve session records from the session server. Each session record includes a field indicating whether it is media asset-related content. The operations platform can use this field to parse whether the content of the session record is media asset-related. If the content of the session record is media asset-related, the session record also includes the signature value (e.g., MD5 value) of the file to be downloaded.
[0073] Optionally, to generate a distributed lock in subsequent steps, each session record also includes the file format name suffix of the media asset file to be downloaded. In this embodiment, each session record includes a field indicating whether the session content is media asset type, the MD5 value of the media asset file to be downloaded, and the file format name suffix of the media asset file to be downloaded. Optionally, each session record further includes the sender identifier and receiver identifier of the media asset file to be downloaded to improve data robustness. Optionally, each session record further includes business parameters of the media asset file to be downloaded, such as the timestamp of the sender sending the media asset file.
[0074] In embodiments of the present invention, the operation platform is a user operation platform integrating socialized management and rapid conversion capabilities. Driven by a digital engine and combined with AI and automation capabilities, this platform achieves user management, coupled with telemarketing to strengthen conversion, realizing a closed-loop operation throughout the user's entire lifecycle. Media asset files refer to media and data files such as images, videos, documents (documents, spreadsheets, PPTs, etc.), audio, and compressed files transmitted during the session retrieval process.
[0075] Step 102: The media asset files to be downloaded in the session records are downloaded in a multi-threaded parallel manner. Specifically, for each media asset file to be downloaded in each session record, the thread that acquires the distributed lock of the media asset file to be downloaded downloads the media asset file to be downloaded from the session server, uploads the media asset file to be downloaded to the file system for storage, obtains the storage address of the media asset file to be downloaded, stores the storage address in the cache, and submits the storage address of the media asset file to be downloaded to the quality inspection platform.
[0076] In this step, the operating platform uses a multi-threaded parallel approach to download the media asset files to be downloaded from the session records. Specifically, for each media asset file to be downloaded in a session record, the thread that acquires the distributed lock for that media asset file downloads it from the session server. Then, the thread uploads the media asset file to the file system for storage, obtaining its storage address. Next, the thread stores this storage address in a cache (e.g., Redis) and submits the storage address of the media asset file to the quality inspection platform. Quality inspection refers to checking the content, copywriting, and voice to ensure they meet marketing script standards, are safe, and compliant; it is one of the measures to ensure marketing security.
[0077] Optionally, the method further includes: other threads that have not acquired the distributed lock of the media asset file to be downloaded obtain the storage address of the media asset file to be downloaded from the cache; in response to the other threads failing to obtain the storage address of the media asset file to be downloaded from the cache, the other threads continue to wait; in response to the other threads being able to obtain the storage address of the media asset file to be downloaded from the cache, the other threads submit the storage address of the media asset file to be downloaded to the quality inspection platform or terminate the process. Since a distributed lock can only be acquired by one thread at a time, when the thread that acquired the distributed lock is downloading a media asset file, other threads that also need to download the same media asset file cannot acquire the distributed lock. These other threads continuously attempt to retrieve the storage address of the media asset file to be downloaded from the cache. If a other thread can retrieve the storage address from the cache, it submits the storage address to the quality inspection platform (which will perform deduplication) or terminates the process (submitting only once). If a other thread cannot retrieve the storage address from the cache, it continues to wait until it can retrieve the storage address from the cache.
[0078] Optionally, the thread that acquires the distributed lock of the media asset file to be downloaded downloads the media asset file from the session server, including: generating a distributed lock for the media asset file to be downloaded based on the signature value of the media asset file to be downloaded and storing the distributed lock in a cache; each thread competing to acquire the distributed lock, so that the distributed lock is acquired by only one thread; the thread that acquires the distributed lock downloads the media asset file to be downloaded from the session server, while other threads that do not acquire the distributed lock for the media asset file to be downloaded wait.
[0079] During the session retrieval process, this embodiment of the invention ensures, by appropriately configuring the distributed lock in Redis (lock key-1), that even under concurrent conditions, there is only one download request for media asset files with the same signature value, and only one download operation is performed. After the download is completed and successfully stored, the storage address (URL) of the downloaded media asset file is written to Redis (cache key-2), and the storage address of the media asset file is submitted to the quality inspection platform. For media asset files with the same signature value, the storage address URL is retrieved from Redis first; if it exists, there is no need to re-download or re-submit to the quality inspection platform.
[0080] It should be noted that in this embodiment of the invention, a distributed lock is generated through Redis. Each media asset file generates a distributed lock, which can only be acquired by one thread. This thread downloads the media asset file, while other threads wait. Therefore, each media asset file has only one download request and performs a download operation only once, avoiding duplicate downloads.
[0081] Optionally, generating a distributed lock for the media asset file to be downloaded based on its signature value includes: generating a distributed lock for the media asset file to be downloaded based on its signature value and file format name suffix. If media asset files of the same type have the same signature value, it can be guaranteed that they are the same media asset file with identical content. Therefore, in this embodiment of the invention, generating a distributed lock based on the signature value and file format name suffix of the media asset file to be downloaded ensures that if the same media asset file has identical content, the distributed lock is acquired by only one thread.
[0082] Optionally, generating a distributed lock for the media asset file to be downloaded based on its signature value and file format name suffix includes concatenating the sender identifier, signature value, and file format name suffix of the media asset file to be downloaded to generate the distributed lock for the media asset file to be downloaded.
[0083] Since session retrieval is a parallel retrieval process, to ensure that the downloaded content is not duplicated during parallel processing, this embodiment of the invention uses a Redis distributed lock. The distributed lock consists of: the sender identifier of the media asset file to be downloaded, the file format name suffix of the media asset file to be downloaded, and the MD5 value of the media asset file to be downloaded. Specifically, if the sender identifier of the media asset file to be downloaded is empty, it means that only one thread can perform the download operation when the same media asset file under all senders is retrieved in parallel; if the sender identifier of the media asset file to be downloaded is not empty, it ensures that only one thread can perform the download operation when the same media asset file under the same sender is retrieved in parallel. Therefore, this embodiment of the invention guarantees unique downloads across senders or under a specified sender.
[0084] Optionally, storing the distributed lock in a cache includes: storing the distributed lock of the media asset file to be downloaded in the cache using a key-value pair; wherein the key is the distributed lock of the media asset file to be downloaded, and the value is a preset default value or a business parameter of the media asset file to be downloaded. To minimize Redis storage space, the distributed lock of the media asset file to be downloaded is stored in the cache using a key-value pair, where the key is the distributed lock of the media asset file to be downloaded, and the value is a preset default value (e.g., value = 1) or a business parameter of the media asset file to be downloaded (e.g., the timestamp of the sender sending the media asset file).
[0085] Optionally, the media asset file to be downloaded is uploaded to a file system for storage to obtain its storage address. This storage address is then stored in a cache. This process includes: uploading the media asset file to be downloaded to a file system for storage to obtain its storage address; and storing the storage address in the cache using a key-value pair. The key is the distributed lock of the media asset file to be downloaded, and the value is its storage address. The thread that acquires the distributed lock uploads the downloaded media asset file to a file system (e.g., HDFS distributed file storage system) to obtain its storage address. This storage address is then stored in the cache using a key-value pair, where the key is the distributed lock and the value is the storage address. Other waiting threads retrieve the storage address from the cache and submit it to the quality inspection platform (which performs deduplication). Alternatively, no deduplication is required, thus avoiding multiple downloads and repeated submissions for quality inspection.
[0086] Based on the various embodiments above, it can be seen that the embodiments of the present invention solve the technical problems of occupying application service network bandwidth, CPU computing, device storage, and other resources in the prior art by using a thread that acquires a distributed lock to download the media asset file from the session server, uploading the media asset file to be downloaded to the file system for storage, obtaining the storage address of the media asset file to be downloaded, storing the storage address in the cache, and submitting the storage address of the media asset file to the quality inspection platform. The embodiments of the present invention, by using a distributed lock to download and store the same media asset file only once, avoid repeated processing of the same media asset file, which reduces the network resources occupied such as application service network bandwidth and saves machine resources such as CPU computing and device storage, thereby improving the download speed of media asset files. Moreover, since the storage address of the same media asset file is only submitted for quality inspection once, it not only saves quality inspection manpower but also improves the quality inspection processing speed.
[0087] Figure 3 This is a flowchart of a media asset file download method according to a possible embodiment of the present invention. As another embodiment of the present invention, such as... Figure 3 As shown, the media asset file download method may include:
[0088] After the operations staff on the operations platform send the same images, videos, and other conversation content to the C-end customer group through the conversation server, the operations platform requests the conversation server to retrieve the conversation records, and the conversation server will generate the conversation records between the operations staff and the customer group.
[0089] The operation platform uses a multi-threaded parallel approach to retrieve session records from the session server. Each session record includes a field indicating whether it is media asset-type session content. If it is media asset-type session content, the session record also includes the signature value (such as MD5 value) of the file to be downloaded, the file format name suffix, the sender identifier and the receiver identifier, and the timestamp of the media asset file sent.
[0090] The media asset file download tool on the operation platform uses multi-threading to parse each session record. If the session record contains media asset-related content, Redis generates a distributed lock based on the signature value, file format name extension, and sender identifier of the file to be downloaded, and stores this distributed lock in Redis. Threads downloading the same media asset file compete to acquire this distributed lock, ensuring that only one thread acquires it. The thread that acquires the distributed lock downloads the media asset file from the session server, while other threads that have not acquired the distributed lock wait.
[0091] The thread that acquires the distributed lock uploads the successfully downloaded media asset file to the file system for storage, obtains the storage address of the media asset file, stores the storage address in Redis, and submits the storage address of the media asset file to the quality inspection platform.
[0092] Other threads that fail to acquire the distributed lock for the media asset file retrieve the storage address of the media asset file from Redis. If they can retrieve the storage address from Redis, they submit the storage address to the quality inspection platform or do nothing. If they fail to retrieve the storage address from Redis, they continue to wait until they can retrieve the storage address from Redis.
[0093] This invention utilizes Redis's distributed caching and distributed locking technologies to solve the technical problems of repeated downloading, uploading, and quality inspection during the media asset file download process, significantly improving system processing efficiency and saving system storage costs, hardware costs, and overall network costs.
[0094] Furthermore, the specific implementation details of the media asset file download method in one of the reference embodiments of the present invention have been described in detail in the above-described media asset file download method, so the details will not be repeated here.
[0095] Figure 4 This is a flowchart of a media asset file download method according to another possible embodiment of the present invention. As another embodiment of the present invention, such as... Figure 4 As shown, the media asset file download method may include:
[0096] The operations platform requests session records from the session server, which then generates session records between operations personnel and customer groups.
[0097] The operation platform uses a multi-threaded parallel approach to retrieve session records from the session server. Each session record includes a field indicating whether it is media asset-type session content. If it is media asset-type session content, the session record also includes the MD5 value of the file to be downloaded, the file format name suffix, the sender identifier (corpID) and the receiver identifier, and the timestamp of the media asset file sent.
[0098] The media asset file download tool on the operation platform uses multi-threading to parse each session record. If the session record contains media asset-related content, the thread generates a key (C + corpID + file format name extension + MD5 value) based on the signature value, file format name extension, and sender identifier of the file to be downloaded. It then iterates through each key in Redis (where the value is the URL of the media asset file) to determine if the key exists. If it does, the process ends; otherwise, Redis generates a distributed lock key (S + corpID + file format name extension + MD5 value) based on the signature value, file format name extension, and sender identifier of the file to be downloaded. This distributed lock is stored in Redis (where the value is a preset default value or a business parameter of the media asset file). Threads downloading the same media asset file compete to acquire this distributed lock, ensuring that only one thread acquires it. The thread that acquires the distributed lock downloads the media asset file from the session server, while other threads that have not acquired the distributed lock wait.
[0099] The thread that acquires the distributed lock uploads the successfully downloaded media asset file to the file system for storage, obtains the storage address URL of the media asset file, stores the URL of the media asset file in Redis, and submits the URL of the media asset file to the quality inspection platform.
[0100] Other threads that have not acquired the distributed lock for the media asset file retrieve the URL of the media asset file from the cache. If they can retrieve the URL of the media asset file from the cache, they submit the URL of the media asset file to the quality inspection platform or do not process it. If they cannot retrieve the URL of the media asset file from Redis, they continue to wait until they can retrieve the URL of the media asset file from Redis.
[0101] In addition, the specific implementation details of the media asset file download method in another reference embodiment of the present invention have been described in detail in the above-described media asset file download method, so the details will not be repeated here.
[0102] Figure 5 This is a schematic diagram of a media asset file downloading device according to an embodiment of the present invention. Figure 5 As shown, the media asset file download device 500 includes a pull module 501 and a download module 502. The pull module 501 is used to pull session records from the session server using a multi-threaded parallel approach, with each session record including the signature value of the media asset file to be downloaded. The download module 502 is used to download the media asset file to be downloaded from the session records using a multi-threaded parallel approach. Specifically, for each media asset file to be downloaded in a session record, the thread that acquires the distributed lock of the media asset file to be downloaded downloads the media asset file from the session server, uploads the media asset file to be downloaded to the file system for storage, obtains the storage address of the media asset file to be downloaded, stores the storage address in the cache, and submits the storage address of the media asset file to the quality inspection platform.
[0103] Optionally, the download module 502 is further configured to:
[0104] Other threads that fail to acquire the distributed lock of the media asset file to be downloaded obtain the storage address of the media asset file to be downloaded from the cache;
[0105] In response to the other threads failing to obtain the storage address of the media asset file to be downloaded from the cache, the other threads continue to wait;
[0106] In response to the other thread being able to obtain the storage address of the media asset file to be downloaded from the cache, the other thread submits the storage address of the media asset file to be downloaded to the quality inspection platform or terminates the process.
[0107] Optionally, the download module 502 is further configured to:
[0108] Based on the signature value of the media asset file to be downloaded, a distributed lock for the media asset file to be downloaded is generated and the distributed lock is stored in the cache. Each thread competes to acquire the distributed lock, so that the distributed lock is acquired by only one thread.
[0109] The thread that acquires the distributed lock downloads the media file to be downloaded from the session server, while other threads that have not acquired the distributed lock for the media file to be downloaded wait.
[0110] Optionally, each session record may also include the file format name suffix of the media asset file to be downloaded;
[0111] The download module 502 is also used for:
[0112] A distributed lock for the media asset file to be downloaded is generated based on the signature value and file format name suffix of the file to be downloaded.
[0113] Optionally, each session record may also include the sender identifier of the media asset file to be downloaded;
[0114] The download module 502 is also used for:
[0115] The sender identifier, signature value, and file format name suffix of the media asset file to be downloaded are concatenated to generate a distributed lock for the media asset file to be downloaded.
[0116] Optionally, the download module 502 is further configured to:
[0117] The distributed lock of the media asset file to be downloaded is stored in the cache using a key-value pair.
[0118] Wherein, key is the distributed lock of the media asset file to be downloaded, and value is a preset default value or a business parameter of the media asset file to be downloaded.
[0119] Optionally, the download module 502 is further configured to:
[0120] The media asset file to be downloaded is uploaded to the file system for storage, thereby obtaining the storage address of the media asset file to be downloaded;
[0121] The storage address of the media asset file to be downloaded is stored in the cache using a key-value pair.
[0122] Wherein, key is the distributed lock of the media asset file to be downloaded, and value is the storage address of the media asset file to be downloaded.
[0123] It should be noted that the specific implementation details of the media asset file download device described in this invention have been described in detail in the media asset file download method described above, so the details will not be repeated here.
[0124] Figure 6An exemplary system architecture 600 is shown that can be applied to the media asset file download method or media asset file download apparatus of the present invention.
[0125] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, and 603, a network 604, and a server 605. Network 604 serves as the medium for providing communication links between terminal devices 601, 602, and 603 and server 605. Network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.
[0126] Users can use terminal devices 601, 602, and 603 to interact with server 605 via network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).
[0127] Terminal devices 601, 602, and 603 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.
[0128] Server 605 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 601, 602, and 603 (this is just an example). The backend management server can analyze and process data such as received item information query requests, and then feed the processing results back to the terminal devices.
[0129] It should be noted that the media asset file download method provided in this embodiment of the invention is generally executed by server 605, and correspondingly, the media asset file download device is generally installed in server 605. The media asset file download method provided in this embodiment of the invention can also be executed by terminal devices 601, 602, and 603, and correspondingly, the media asset file download device can be installed in terminal devices 601, 602, and 603.
[0130] It should be understood that Figure 6 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0131] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing a terminal device of the present invention. Figure 7 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.
[0132] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0133] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.
[0134] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined above in the system of this invention.
[0135] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer programs according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0137] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a fetch module and a download module, wherein the names of these modules do not necessarily limit the module itself.
[0138] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device implements the following method: retrieving session records from a session server using a multi-threaded parallel approach, each session record including a signature value of a media asset file to be downloaded; downloading the media asset file to be downloaded from the session records using a multi-threaded parallel approach; wherein, for each media asset file to be downloaded in a session record, a thread that acquires the distributed lock of the media asset file to be downloaded downloads the media asset file from the session server, uploads the media asset file to be downloaded to a file system for storage, obtains the storage address of the media asset file to be downloaded, stores the storage address in a cache, and submits the storage address of the media asset file to be downloaded to a quality inspection platform.
[0139] In another aspect, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the above embodiments.
[0140] According to the technical solution of this invention, by employing a distributed lock on the media asset file to be downloaded by a thread that acquires the distributed lock, downloading the media asset file from the session server, uploading it to the file system for storage, obtaining the storage address of the media asset file, storing the storage address in the cache, and submitting the storage address of the media asset file to the quality inspection platform, this invention overcomes the technical problems of consuming application service network bandwidth, CPU computing, and device storage resources in the prior art. This invention, through distributed locks, downloads and stores the same media asset file only once, avoiding repeated processing of the same media asset file. This reduces the consumption of network resources such as application service network bandwidth and saves machine resources such as CPU computing and device storage, thereby improving the download speed of media asset files. Moreover, since the storage address of the same media asset file is submitted to quality inspection only once, it not only saves quality inspection manpower but also improves the quality inspection processing speed.
[0141] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A method of downloading a media file, characterized by, include: A multi-threaded parallel approach is used to retrieve session records from the session server. Each session record includes the signature value of the media asset file to be downloaded. The media asset files to be downloaded from the session records are downloaded using a multi-threaded parallel approach. Specifically, for each media asset file in a session record, the thread that acquires the distributed lock for the media asset file downloads the file from the session server, uploads it to the file system for storage, obtains the storage address of the media asset file, stores the storage address in the cache, and submits the storage address of the media asset file to the quality inspection platform. A distributed lock is generated for each media asset file, and this distributed lock can only be acquired by one thread. This thread downloads the media asset file, while other threads wait. The distributed lock for each media asset file is generated based on the signature value of the media asset file to be downloaded.
2. The method of claim 1, wherein, Also includes: Other threads that fail to acquire the distributed lock of the media asset file to be downloaded obtain the storage address of the media asset file to be downloaded from the cache; In response to the other threads failing to obtain the storage address of the media asset file to be downloaded from the cache, the other threads continue to wait; In response to the other thread being able to obtain the storage address of the media asset file to be downloaded from the cache, the other thread submits the storage address of the media asset file to be downloaded to the quality inspection platform or terminates the process.
3. The method according to claim 1, characterized in that, The thread that acquires the distributed lock of the media asset file to be downloaded downloads the media asset file from the session server, including: Based on the signature value of the media asset file to be downloaded, a distributed lock for the media asset file to be downloaded is generated and the distributed lock is stored in the cache. Each thread competes to acquire the distributed lock, so that the distributed lock is acquired by only one thread. The thread that acquires the distributed lock downloads the media file to be downloaded from the session server, while other threads that have not acquired the distributed lock for the media file to be downloaded wait.
4. The method according to claim 3, characterized in that, Each session record also includes the file format name suffix of the media asset file to be downloaded; Based on the signature value of the media asset file to be downloaded, a distributed lock for the media asset file to be downloaded is generated, including: A distributed lock for the media asset file to be downloaded is generated based on the signature value and file format name suffix of the file to be downloaded.
5. The method according to claim 4, characterized in that, Each session record also includes the sender identifier of the media asset file to be downloaded; Based on the signature value and file format name suffix of the media asset file to be downloaded, a distributed lock is generated for the media asset file to be downloaded, including: The sender identifier, signature value, and file format name suffix of the media asset file to be downloaded are concatenated to generate a distributed lock for the media asset file to be downloaded.
6. The method according to claim 3, characterized in that, Storing the distributed lock in a cache includes: The distributed lock of the media asset file to be downloaded is stored in the cache using a key-value pair. Wherein, key is the distributed lock of the media asset file to be downloaded, and value is a preset default value or a business parameter of the media asset file to be downloaded.
7. The method according to claim 1, characterized in that, Upload the media asset file to be downloaded to the file system for storage, obtain the storage address of the media asset file to be downloaded, and store the storage address in the cache, including: The media asset file to be downloaded is uploaded to the file system for storage, thereby obtaining the storage address of the media asset file to be downloaded; The storage address of the media asset file to be downloaded is stored in the cache using a key-value pair. Wherein, key is the distributed lock of the media asset file to be downloaded, and value is the storage address of the media asset file to be downloaded.
8. A media asset file downloading device, characterized in that, include: The pull module is used to pull session records from the session server in a multi-threaded parallel manner. Each session record includes the signature value of the media asset file to be downloaded. The download module is used to download the media asset files to be downloaded from the session records in a multi-threaded parallel manner. Specifically, for each media asset file to be downloaded in a session record, the thread that acquires the distributed lock of the media asset file to be downloaded downloads the media asset file from the session server, uploads the media asset file to the file system for storage, obtains the storage address of the media asset file to be downloaded, stores the storage address in the cache, and submits the storage address of the media asset file to the quality inspection platform. A distributed lock is generated for each media asset file, and this distributed lock can only be acquired by one thread. This thread downloads the media asset file, while other threads wait. The distributed lock for the media asset file to be downloaded is generated based on the signature value of the media asset file to be downloaded.
9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.
10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.
11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.
Citation Information
Patent Citations
Static resource uploading method, device, integrated server and system
CN111049889A
Multi-source distributed downloading system based on FTP (File Transfer Protocol)
CN115086304A
