A material downloading method and a material downloader

Abstract

The present application relates to the field of web crawler, especially to a material downloading method and a material downloader, the method first acquires the search field input by a user, then crawls the material resource links related to the search field and pushes them into a message queue, then parses the message queue to obtain parsing information, and pushes the parsing information into a task queue to execute the material batch downloading task in parallel. Compared with the prior art, the method of the present application realizes efficient crawling of materials, supports users to smoothly edit videos while downloading new materials in batches, and solves the technical problem of affecting video production efficiency caused by the existing material crawling method.

Description

Technical Field

[0001] This invention relates to the field of web crawling technology. More specifically, this invention relates to a method and tool for downloading web content. Background Technology

[0002] OBS (Open Broadcaster Software) is a free, open-source real-time video recording and live streaming tool widely used in game streaming, online education, video conferencing, and content creation. When creating videos, video creators typically need to download video or image materials one by one from web pages. This method incurs excessive time costs for web browsing, registration / authorization, etc., hindering efficient video editing.

[0003] To achieve rapid material acquisition, existing technology includes a data crawling method, apparatus, storage medium, and system disclosed in Chinese patent application CN119739919A. This method primarily utilizes web crawler functionality embedded in browser automation tools to automatically crawl materials from web pages. While this approach eliminates the need for video producers to individually access and download materials from various video material websites, it still suffers from the following problems:

[0004] (1) The web page needs to be opened to enable the material crawling function, which increases the computer's memory load, reduces the overall performance of the computer, and slows down the computer's interactive response speed. This is not conducive to video producers performing video production (including editing, video rendering and dubbing, etc.) during the material crawling process (editing based on existing materials while downloading tasks).

[0005] (2) The crawled material information will be displayed in a list on the webpage, and the video creator still needs to click to confirm the download one by one.

[0006] Therefore, existing methods of scraping materials have the problem of affecting video production efficiency. Summary of the Invention

[0007] To address the technical problem of how the aforementioned material scraping methods affect video production efficiency, this invention discloses a material downloading method and a material downloader.

[0008] In a first aspect, this invention discloses a method for downloading materials, which automatically loads a downloader based on yt-dlp in response to the startup of OBS software. The method of this invention includes:

[0009] Get the search fields entered by the user;

[0010] Crawl links to relevant source materials related to the search fields and push them into the message queue;

[0011] The message queue is parsed to obtain the parsed information, which is then pushed into the task queue to execute batch download tasks of materials in parallel.

[0012] Beneficial effects: The method of this invention automatically loads the yt-dlp-based downloader when the OBS software starts, eliminating the need for users to enable plugins through a browser. This method consumes less computer memory resources and helps improve the interactive response speed in video editing. Furthermore, this invention improves the way the downloader crawls and downloads resources. When a user enters a search field in the OBS software, the downloader first crawls the relevant material resource links and pushes them into a message queue. Then, it parses the message queue and automatically executes batch download tasks, achieving efficient material crawling and allowing users to smoothly download new materials in batches while editing videos.

[0013] Preferably, in obtaining the search fields input by the user, the method of the present invention further includes:

[0014] Get the user's input source link;

[0015] Determine the type of the input source link;

[0016] If the type of the input source link is a business-initiated call, then the corresponding API module will be driven to execute the corresponding download task;

[0017] If the type of the input source link is a timed push source, then extract the material resource link from the input source link and push it into the message queue.

[0018] Beneficial effects: The method of this invention supports remote active request for direct download (i.e., business active call) and periodic automatic batch download for timed push sources. This approach is beneficial for material collectors to obtain materials in batches by remote request or remote link provision, which helps improve the video production efficiency of video producers.

[0019] Preferably, the yt-dlp secondary encapsulation module is used to parse the message queue and execute batch download tasks.

[0020] Preferably, the yt-dlp secondary encapsulation module is used to parse the message queue and execute batch download tasks, including:

[0021] During the parsing phase, a low-traffic dynamic IP proxy is used to simulate the user's real network behavior and obtain the target material information in the material resource link as parsing information;

[0022] During the download phase, a high-traffic dynamic IP proxy is used to perform batch download tasks of parsed information in parallel.

[0023] Beneficial effects: During the parsing phase, low-traffic dynamic IP proxies are used to simulate real user network behavior. This method avoids triggering the anti-scraping mechanisms of the resource website servers, preventing user accounts from being banned. During the download phase, high-traffic dynamic IP proxies are used to achieve greater bandwidth, thereby accelerating the download and reducing the time required to download the materials.

[0024] Preferably, before obtaining the search fields input by the user, the method of the present invention further includes:

[0025] Define an OBS software startup script so that when the OBS software startup event is triggered, the downloader startup event is triggered synchronously.

[0026] Define the communication interface between the OBS software and the downloader;

[0027] Set the download archive address of the downloader to the OBS software's media library storage address.

[0028] Beneficial effects: The above technical solution achieves the effects of binding the OBS software and the downloader to start, as well as information association and unified storage address by defining the association between the startup scripts of the OBS software and the downloader.

[0029] Preferably, after performing the batch download task of materials in parallel, the method of the present invention further includes:

[0030] The downloaded results are sent to the local media repository and cloud storage in the OBS software.

[0031] Preferably, the cloud storage provider is OSS or S3.

[0032] Preferably, the message queue is a Kafka message queue.

[0033] Preferably, the search types corresponding to the search fields include video materials, photo materials, artistic font materials, and / or audio sound effects materials.

[0034] Secondly, the present invention also discloses a material downloader for executing the material downloading method described in the first aspect. The downloader includes a field acquisition module, a crawler scheduling module, and a parsing-downloading module; wherein,

[0035] The field retrieval module is used to retrieve the search fields input by the user;

[0036] The crawler scheduling module is used to crawl material resource links related to the search fields and push them into the message queue;

[0037] The parsing-download module is used to parse the message queue, obtain parsing information, and push the parsing information into the task queue to execute batch download tasks of materials in parallel.

[0038] Beneficial effects: The downloader of this invention has the ability to directly interact with OBS software and supports batch download of materials. It occupies less memory resources and has a fast download speed, which can meet the daily needs of video producers for material crawling.

[0039] The beneficial effects of this invention are as follows:

[0040] (1) Compared with the prior art, the method of the present invention achieves efficient material crawling, supports users to smoothly edit videos while downloading new materials in batches, and solves the technical problem that the existing material crawling method affects the efficiency of video production.

[0041] (2) Compared with the prior art, the method of the present invention supports remote active request for direct download and periodic automatic batch download for timed push sources. This method is conducive to material collectors to obtain materials in batches by remote request or remote link provision, which is conducive to improving the video production efficiency of video producers.

[0042] (3) Compared with the prior art, the downloader of the present invention has the ability to directly interact with OBS software and supports batch download of materials. It occupies less memory resources and has a faster download speed, which can meet the daily needs of video producers for material crawling. Attached Figure Description

[0043] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0044] Figure 1 This is a flowchart of the material downloading method in Embodiment 1 of the present invention;

[0045] Figure 2 This is a schematic diagram of the material downloader in Embodiment 2 of the present invention;

[0046] Figure 3 This is a schematic diagram of the computer system in Embodiment 3 of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] This embodiment discloses a material downloading method and a material downloader, which is used to solve the technical problem that existing material crawling technology affects the efficiency of video production.

[0049] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0050] Example 1

[0051] like Figure 1 As shown, this embodiment discloses a method for downloading materials. Before actually executing a specific download task, the method automatically loads a preset downloader when the OBS software starts, and then performs the following steps through the downloader:

[0052] S10: Obtain the search field input by the user.

[0053] In this embodiment, the search fields can be entered into the OBS software's material search bar. The search fields are then forwarded to the downloader's information input interface via an information interface. The types of search fields include video materials, photo materials, artistic font materials, and audio / sound effect materials. The specific content of the search fields can be a general description of the material content, such as "action game," "Premier League match," or "weapon collision sound effects," or it can be the name of the material provider, or the name of the source URL (requiring account registration and login), such as "Light Factory," "Behind the Scenes Network," or "Vidlery."

[0054] S20: Crawl the material resource links related to the search field and push them into the message queue.

[0055] In this embodiment, the message queue is a Kafka queue. By integrating the material resource links into a queue using a Kafka queue, task buffering can be achieved, thus balancing the load of the information flow and ensuring that data overload does not occur.

[0056] S30: Parse the message queue, obtain the parsed information, push the parsed information into the task queue, and execute the batch download task of materials in parallel.

[0057] In this embodiment, step S30 above uses the yt-dlp secondary encapsulation module to parse the message queue and execute batch download tasks.

[0058] It should be explained that the traditional yt-dlp is an open-source download tool, while the method in this embodiment is a secondary encapsulation of yt-dlp, which specifically combines functions such as dynamic resolution, IP proxy change, and batch download task scheduling.

[0059] The aforementioned secondary encapsulation of yt-dlp to form a yt-dlp secondary encapsulation module mainly includes:

[0060] S300: Optimize yt-dlp to support remote aria2 download tasks, resulting in the first packaged script.

[0061] It should be explained that the traditional yt-dlp can only be used for on-premises downloads, while the method in this embodiment replaces the external.py file in yt-dlp to enable remote aria2 downloads. Specifically, the installation path of yt-dlp is first obtained, and shutil.copyfile overwrites the original file with the custom external.py file, thus extending its functionality. After the replacement, yt-dlp's download logic will call the remote aria2 service instead of downloading locally. During this process, exception handling logic also needs to be defined to catch exceptions in file operations (such as insufficient permissions or incorrect paths) to prevent program crashes.

[0062] More specifically, the optimized code for step S300 above is as follows:

[0063] def init_aria2(self):

[0064] # Get the installation path of yt-dlp

[0065] yt_dlp_path = os.path.dirname(yt_dlp.__file__)

[0066] # Build the replacement file path (remote aria2 adaptation script)

[0067] original_file_path = os.path.join(yt_dlp_path, "downloader", "external.py")

[0068] replacement_file_path = os.path.join(

[0069] os.path.dirname(os.path.dirname(os.path.abspath(__file__))),

[0070] "static",

[0071] "external.py" )

[0073] # Perform file replacement (more complex processing may be required in real-world projects)

[0074] try:

[0075] shutil.copyfile(replacement_file_path, original_file_path)

[0076] print(f"Successfully replaced {original_file_path} with {replacement_file_path}")

[0077] except Exception as e:

[0078] print(f"Error occurred while replacing files: {e}")

[0079] S301: Defines the communication protocol when calling the remote aria2 service to perform download tasks, resulting in the second encapsulated script.

[0080] It should be noted that improvements to step S300 alone are insufficient to achieve batch download task scheduling and dynamic IP functionality; a JSON-RPC protocol for communication with aria2 also needs to be established. Specifically, step S301 first constructs the JSON-RPC protocol in the script, then defines the HTTP request sending script, and sets up a corresponding response handling mechanism. This mechanism primarily verifies whether the ID in the response matches the request to prevent request confusion. If the IDs match, the result field is returned.

[0081] More specifically, the core code for step S302 above is:

[0082] def aria2c_rpc(self, rpc_url, rpc_secret, method, params=()):

[0083] # Generate a unique request ID (sanitycheck)

[0084] sanitycheck = str(uuid.uuid4())

[0085] # Construct JSON-RPC request data

[0086] d = json.dumps({

[0087] "jsonrpc": "2.0",

[0088] "id": sanitycheck,

[0089] "method": method,

[0090] "params": [f"token:{rpc_secret}", *params],

[0091] }).encode()

[0092] # Send an HTTP request to the aria2 service

[0093] request = Request(

[0094] url=rpc_url,

[0095] data=d,

[0096] headers={

[0097] "Content-Type": "application / json",

[0098] "Content-Length": f"{len(d)}",

[0099] },

[0100] proxies={"all": None}, # Disable proxy )

[0102] # Use yt-dlp's urlopen to send requests and get responses

[0103] with self.ydl.urlopen(request) as r:

[0104] resp = json.load(r)

[0105] assert resp.get("id") == sanitycheck, "RPC request ID does not match"

[0106] return resp["result"]

[0107] S302: Integrate the first encapsulation script and the second encapsulation script to obtain the yt-dlp secondary encapsulation module.

[0108] It should be explained that the integration of the first and second encapsulation scripts can be achieved by configuring the initialization conditions / parameters and calling relationships of the first and second encapsulation scripts. The resulting yt-dlp secondary encapsulation module can also be used independently as a lightweight application (similar to a downloader like Xunlei or Baidu Cloud), supporting PC, Android, and OS platforms.

[0109] Through the above steps S301-S302, the secondary encapsulation of yt-dlp can be achieved, enabling it to not only have functions such as dynamic resolution, IP proxy change and batch download task scheduling, but also error self-checking function.

[0110] Furthermore, the above-mentioned use of the yt-dlp secondary encapsulation module to parse the message queue and execute batch download tasks is as follows:

[0111] S31: During the parsing phase, a low-traffic dynamic IP proxy is used to simulate the user's real network behavior to obtain the target material information in the material resource link as parsing information.

[0112] It's important to further explain that the aforementioned low-traffic dynamic IP is a specific type of network proxy service that combines two core attributes: low-traffic and dynamic IP. It's primarily used to simulate the network behavior of real users to circumvent the anti-scraping or anti-abuse mechanisms of target websites. The low-traffic characteristic refers to the downloader reducing the frequency (intermittently) of interactive requests to the target crawling URL, simulating the reasonable delay of human keystrokes. The dynamic IP characteristic means that the IP address changes periodically or automatically as needed (e.g., every few minutes to a few hours), avoiding the long-term use of the same IP to initiate requests, thereby reducing the risk of being identified as a crawler or malicious traffic.

[0113] S32: During the download phase, a high-traffic dynamic IP proxy is used to perform batch download tasks of parsing information in parallel.

[0114] It needs further explanation that the aforementioned high-traffic dynamic IP proxy is a proxy technology that supports high-concurrency requests by frequently switching IP addresses. Its core lies in the combination of traffic-carrying capacity and a dynamic IP switching mechanism, making it suitable for scenarios requiring large-scale data transmission or high-frequency access. It integrates two core attributes: high traffic and dynamic IP. Its high traffic characteristic allows a single IP or proxy pool to handle higher concurrent requests and can distribute traffic through load balancing technology, avoiding single-point overload and ensuring the stability of download task execution. Its dynamic IP characteristic is similar to that described in step S31 above and will not be repeated here.

[0115] Through the above steps S31-S32, the method of this embodiment, based on the secondary encapsulation of yt-dlp, enables it to flexibly switch network proxy service types during the parsing and downloading stages. This approach can reduce the risk of being identified as a crawler or malicious traffic, and can also achieve a larger bandwidth proxy, making batch download tasks possible.

[0116] Compared to existing technologies, steps S10-S30 automatically load the downloader upon OBS software startup, eliminating the need for users to open a browser to enable plugin functionality. This reduces the computer's memory usage and improves the interactive response rate in video editing. Furthermore, this embodiment improves the way the downloader crawls and downloads resources. When a user enters a search field in OBS software, the downloader first crawls the relevant material resource links and pushes them into a message queue. Then, the yt-dlp secondary encapsulation module parses the message queue and downloads the parsed information in batches. This achieves efficient and low-memory-usage material crawling, thus solving the technical problem of existing material crawling technologies affecting video production efficiency.

[0117] Furthermore, in order to achieve startup binding and information interaction / sharing between the OBS software and the aforementioned downloader, prior to step S10, the method of this embodiment further includes:

[0118] S100: Defines the OBS software startup script so that when the OBS software startup event is triggered, the downloader startup event is triggered synchronously.

[0119] Specifically, when OBS starts, the Aria2 downloader is triggered to start via the startup script to achieve the above step S100.

[0120] S101: Defines the communication interface between the OBS software and the downloader.

[0121] Specifically, based on the TCP / UDP Socket communication protocol, the downloader is used as the server listening port, and the OBS software is used as the client, thereby defining the communication interface between the OBS software and the downloader.

[0122] S102: Set the download archive address of the downloader to the OBS software's media library storage address.

[0123] Specifically, configure the aria2.conf script of the downloader to set the default save path of the downloader to the OBS software's media library storage address.

[0124] Through the above steps S100-S102, the effects of binding and starting OBS and the downloader, as well as information association and unified storage address, are achieved.

[0125] Furthermore, in step S10 above, the method of this embodiment supports remote assistance and periodic material downloads. The specific steps to achieve the above technical effects are as follows:

[0126] S200: Obtain the user's input source link.

[0127] S201: Determine the type of the input source link.

[0128] S202: If the type of the input source link is a business-initiated call, then drive the corresponding API module to execute the corresponding download task.

[0129] It should be noted that "business-initiated invocation" refers to the request link for invocation operations from the remote material supplier / assistance end.

[0130] Specifically, the remote material supplier / assistant provides the material they are shooting / recording to the downloader's API module by sharing the input source link to execute the corresponding download task.

[0131] S203: If the type of the input source link is a timed push source, then extract the material resource link from the input source link and push it into the message queue.

[0132] It should be explained that the timed push source is a type of other call source. Its input source link is a periodically automatically pushed request link. For example, when a video blogger updates a recorded video, the link is automatically pushed to followers (video creators). At this time, the downloader will automatically receive the link and push it into the message queue. Step S203 allows the behind-the-scenes video production team of celebrities, bloggers, or live streamers to receive relevant materials immediately after the celebrity, blogger, or live streamer completes the on-site recording, saving the material handover process and improving video editing efficiency.

[0133] Furthermore, after step S50 above, the method of this embodiment further includes:

[0134] S60: Delivers the downloaded results to the local media repository and cloud storage of the OBS software.

[0135] The above steps S60 facilitate the reliable storage of download results. The cloud storage terminal serves as an information backup storage terminal, enabling information retrieval through data copying when information in the local material repository is lost.

[0136] Preferably, the cloud storage terminal is OSS or S3. These two cloud storage terminals have the advantages of high reliability, low cost and long-term storage, and can be highly integrated with OBS software.

[0137] Example 2

[0138] like Figure 2 As shown, this embodiment discloses a downloader for executing the material downloading method described in Embodiment 1. The downloader includes a field acquisition module, a crawler scheduling module, and a parsing-downloading module.

[0139] Specifically, the field acquisition module is used to acquire the search fields input by the user. The crawler scheduling module is used to crawl the material resource links related to the search fields and push them into the message queue. The parsing-downloading module (i.e., the yt-dlp secondary encapsulation module) is used to parse the message queue, obtain the parsing information, and push the parsing information into the task queue to execute batch material download tasks in parallel.

[0140] Through the aforementioned technical improvements, the downloader in this embodiment possesses the ability to directly interact with OBS, supports batch downloading of materials, consumes relatively little memory, and boasts a fast download speed, thus meeting the daily material scraping needs of video producers. The concrete characteristics of this downloader are:

[0141] Firstly, it combines the open-source framework dy-dlp and performs secondary encapsulation of dy-dlp, adapting to the download of material types from 75 websites.

[0142] Secondly, it has a built-in download cluster function, supporting batch download and return with 10T bandwidth.

[0143] Third, it can adapt to multiple access methods.

[0144] Example 3

[0145] like Figure 3 As shown, this embodiment discloses a computer system, including a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the material downloading method described in Embodiment 1 is implemented.

[0146] The system also includes other components well known to those skilled in the art, such as communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

[0147] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions that can be stored or otherwise maintained by such a computer-readable medium.

[0148] In the description of this specification, "multiple" means at least two, such as two, three or more, etc., unless otherwise expressly and specifically defined.

[0149] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A method for downloading materials, characterized in that, In response to the startup of the OBS software, an automatic loading of a yt-dlp-based downloader is performed, the method comprising: Get the search fields entered by the user; Crawl the material resource links related to the search fields and push them into the message queue; The message queue is parsed to obtain parsed information, and the parsed information is pushed into the task queue to execute batch download tasks of materials in parallel. The source materials were obtained through account registration and login; Specifically, the yt-dlp secondary encapsulation module is used to parse the message queue and execute batch download tasks; Specifically, YT-DLP was repackaged to form a YT-DLP repackaged module, which includes: Optimize yt-dlp to support remote aria2 download tasks, resulting in the first packaged script; Define the communication protocol when calling the remote aria2 service to perform download tasks, and obtain the second encapsulated script; The first and second encapsulation scripts are integrated to obtain the yt-dlp secondary encapsulation module.

2. The material download method according to claim 1, characterized in that, The method further includes, in addition to, obtaining the search fields input by the user: Get the user's input source link; Determine the type of the input source link; If the type of the input source link is a business-initiated call, then the corresponding API module will be driven to execute the corresponding download task; If the type of the input source link is a timed push source, then the material resource link is extracted from the input source link and pushed into the message queue.

3. The material download method according to claim 1, characterized in that, Before obtaining the search fields input by the user, the method further includes: Define an OBS software startup script so that when the OBS software startup event is triggered, the downloader startup event is triggered synchronously. Define the communication interface between the OBS software and the downloader; Set the download archive address of the downloader to the OBS software's media library storage address.

4. The material download method according to claim 1, characterized in that, After performing the batch download task of materials in parallel, the method further includes: The downloaded results are sent to the local media repository and cloud storage in the OBS software.

5. The material download method according to claim 4, characterized in that, The cloud storage terminal is OSS or S3.

6. The material download method according to claim 1, characterized in that, The message queue used is a Kafka message queue.

7. The material download method according to claim 1, characterized in that, The search types corresponding to the search fields include video materials, photo materials, artistic font materials, and / or audio sound effects materials.

8. A material downloader, characterized in that, The downloader, used to perform the material downloading method according to any one of claims 1-7, comprises a field acquisition module, a crawler scheduling module, and a parsing-downloading module; wherein... The field acquisition module is used to acquire the search fields input by the user; The crawler scheduling module is used to crawl material resource links related to the search field and push them into the message queue; The parsing-download module is used to parse the message queue, obtain parsing information, and push the parsing information into the task queue to execute batch download tasks of materials in parallel. Specifically, the yt-dlp secondary encapsulation module is used to parse the message queue and execute batch download tasks; Specifically, YT-DLP was repackaged to form a YT-DLP repackaged module, which includes: Optimize yt-dlp to support remote aria2 download tasks, resulting in the first packaged script; Define the communication protocol when calling the remote aria2 service to perform download tasks, and obtain the second encapsulated script; The first and second encapsulation scripts are integrated to obtain the yt-dlp secondary encapsulation module.

Images