Data capturing method and device, medium and product

Abstract

The embodiment of the invention discloses a data capturing method and device, a medium and a product. The method comprises the steps that an interface combination and exclusive data blocks of all interfaces in the interface combination are determined, the interface combination comprises at least two interfaces, the at least two interfaces are complementary on the data block downloading speed level, and an exclusive data block combination corresponding to the interface combination comprises all data blocks of a target file; corresponding data blocks in a target file are downloaded through the interfaces in the interface combination, and the identifier of each data block carries a file identifier; and for each file identifier, when it is detected that all the data blocks under the file identifier are downloaded, determining a target file for the file identifier according to all the data blocks under the file identifier. According to the technical scheme provided by the embodiment of the invention, the file downloading speed can be improved under the condition that the interface transmission bandwidth is not changed.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a data capture method, device, medium and product. Background Technology

[0002] In the field of file downloading, slow download speeds are a common problem. Existing technologies typically improve download speeds by increasing interface bandwidth. However, this method is usually costly and limited by hardware resources. Summary of the Invention

[0003] This invention provides a data capture method, device, medium, and product that can improve file download speed without changing the interface transmission bandwidth.

[0004] According to one aspect of the present invention, a data scraping method is provided, the method comprising: Determine the interface combination and the dedicated data block of each interface in the interface combination. The interface combination includes at least two interfaces, and the at least two interfaces are complementary in terms of data block download speed. The dedicated data block combination corresponding to the interface combination includes all data blocks of the target file. Each of the interfaces in the interface combination downloads the corresponding data block in the target file, and the identifier of each data block carries a file identifier; For each of the aforementioned file identifiers, if it is detected that all data blocks under the file identifier have been downloaded, the target file for the file identifier is determined based on all the data blocks under the file identifier.

[0005] According to another aspect of the present invention, a data capture device is provided, the device comprising: An interface combination module is used to determine an interface combination and the dedicated data block of each interface in the interface combination. The interface combination includes at least two interfaces, and the at least two interfaces are complementary in terms of data block download speed. The dedicated data block combination corresponding to the interface combination includes all data blocks of the target file. The download module is used to download corresponding data blocks from the target file through each of the interfaces in the interface combination, wherein the identifier of each data block carries a file identifier; The result module is used to determine the target file for each file identifier based on all the data blocks under the file identifier, provided that all data blocks under the file identifier have been downloaded.

[0006] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: One or more processors; Storage device for storing one or more programs. When one or more programs are executed by one or more processors, the one or more processors implement the data fetching method as described in any embodiment of the present invention.

[0007] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the data capture method described in any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer program product is provided, which, when executed by a processor, implements the data capture method as described in any embodiment of the present invention.

[0009] The technical solution provided by this invention, since at least two interfaces in the interface combination are matched with dedicated data blocks, and all dedicated data blocks corresponding to the at least two interfaces include all data blocks of the target file, and the at least two interfaces are complementary in terms of data block download speed, the dedicated data blocks of each interface in the interface combination do not overlap and can be pieced together to form a complete "high-speed download map". Thus, the overall performance can be maximized through collaborative division of labor. Therefore, by downloading each data block of the file based on each interface in the interface combination, the overall download time of the file can be minimized, thereby significantly improving the download speed of batch files and the user experience.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating the data capture method provided in an embodiment of the present invention; Figure 2 This is another flowchart illustrating the data scraping method provided in this embodiment of the invention; Figure 3 This is another flowchart illustrating the data scraping method provided in this embodiment of the invention; Figure 4 This is a schematic diagram of the structure of the data capture device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] Figure 1 This is a flowchart illustrating the data capture method provided in an embodiment of the present invention. This embodiment is applicable to situations where file downloads are completed through interface cooperation. The method can be executed by a data capture device, which can be implemented in hardware and / or software and can be configured in electronic devices such as computers or servers. Figure 1 As shown, the method in this embodiment includes: S110. Determine the interface combination and the dedicated data block of each interface in the interface combination. The interface combination includes at least two interfaces, and the at least two interfaces are complementary in terms of data block download speed. The dedicated data block combination corresponding to the interface combination includes all data blocks of the target file.

[0016] Periodically calculate the download speed of each interface for each data block, determine the average download speed of each interface for each data block based on the download speed of each interface for each data block, determine the average download speed deviation rate of each interface based on the average download speed of each interface for each data block, identify all candidate interfaces whose average download speed deviation rate exceeds a predetermined threshold, and determine the interface combination and the dedicated data block of each interface in the interface combination based on the average download speed of each candidate interface for each data block.

[0017] Since the average download speed deviation rate of an interface can accurately measure the consistency of the download speed of each data block, it can accurately pinpoint the interface with a bottleneck. Because the bottlenecks of multiple interfaces are usually different, interfaces with complementary bottlenecks are grouped together.

[0018] The statistical period is a configurable setting, such as 30 minutes or other time intervals. Users can set it according to their actual needs.

[0019] By periodically determining the interface combination and the dedicated data block corresponding to each interface in the interface combination, the interface combination and the dedicated data block corresponding to each interface in the interface combination are not static, but dynamically change periodically according to the actual performance of each interface. In this way, based on the interface combination and the dedicated data block corresponding to each interface in the interface combination, the download time of the target file can be shortened to the maximum extent.

[0020] Regarding dedicated data blocks for each interface: Once the interface combination is determined, the dedicated data blocks for each interface are determined based on the average download speed of each data block for each interface. Specifically, for each interface, the data block with the highest average download speed is designated as its dedicated data block.

[0021] The at least two interfaces are complementary in terms of data block download speed, and the dedicated data block combination corresponding to the interface combination includes all data blocks of the target file. This can be understood as the dedicated data blocks of each interface not overlapping and being able to be pieced together to form a complete "high-speed download map", thereby maximizing overall performance through collaborative division of labor.

[0022] For example, the target file includes a first data block and a second data block, and the interface combination includes a first interface and a second interface. The first interface downloads the first data block at a speed of S11 and downloads the second data block at a speed of S12, where S11 is greater than S12; the second interface downloads the first data block at a speed of S21 and downloads the second data block at a speed of S22, where S22 is greater than both S12 and S21. Therefore, the first data block is designated as the exclusive data block for the first interface, meaning the first interface is only used to download the first data block; and the second data block is designated as the exclusive data block for the second interface, meaning the second interface is only used to download the second data block.

[0023] For example, the target file includes a first data block, a second data block, and a third data block, and the interface combination includes a first interface, a second interface, and a third interface. The download speeds of the first interface, the second interface, and the third interface for the first data block are S11, S21, and S31, respectively; the download speeds of the first interface, the second interface, and the third interface for the second data block are S12, S22, and S32, respectively; and the download speeds of the first interface, the second interface, and the third interface for the third data block are S13, S23, and S33, respectively. Wherein, S11 is greater than S21, and S21 is greater than S31; S22 is greater than S32, and S32 is greater than S12; S33 is greater than S13, and S13 is greater than S23. Therefore, the first data block is designated as the dedicated data block for the first interface, the second data block as the dedicated data block for the second interface, and the third data block as the dedicated data block for the third interface.

[0024] It should be noted that the principle for determining the interface combination and the dedicated data blocks of each interface in the interface combination is to maximize the overall speed, so as to avoid data transmission bottlenecks caused by uneven download speeds of interfaces, especially in batch file download scenarios where the amount of data to be downloaded at one time is limited.

[0025] S120. Download the corresponding data blocks in the target file through each interface in the interface combination. The identifier of each data block carries the file identifier.

[0026] Obtain the identifier of the file to be downloaded, and then each interface in the driver interface combination downloads the exclusive data block in the file corresponding to the file identifier, so that each data block in the file can be downloaded at high speed.

[0027] Since each data block's identifier carries a file identifier, the processor can determine whether all data blocks of the file corresponding to each file identifier have been downloaded based on the data block identifiers in the storage area or cache area.

[0028] S130. For each file identifier, if it is detected that all data blocks under the file identifier have been downloaded, determine the target file for the file identifier based on all data blocks under the file identifier.

[0029] For each file identifier, if it is detected that all data blocks of its corresponding file have been downloaded, all its data blocks are concatenated into a target file for that file identifier.

[0030] The technical solution provided by this invention, since at least two interfaces in the interface combination are matched with dedicated data blocks, and all dedicated data blocks corresponding to the at least two interfaces include all data blocks of the target file, and the at least two interfaces are complementary in terms of data block download speed, the dedicated data blocks of each interface in the interface combination do not overlap and can be pieced together to form a complete "high-speed download map". Thus, the overall performance can be maximized through collaborative division of labor. Therefore, by downloading each data block of the file based on each interface in the interface combination, the overall download time of the file can be minimized, thereby significantly improving the download speed of batch files and the user experience.

[0031] Figure 2 This is another flowchart illustrating the data capture method provided in this embodiment of the invention. This embodiment is used to refine the data block download steps in the aforementioned embodiments. Figure 2 As shown, the method in this embodiment may specifically include: S210. Determine the interface combination and the dedicated data block of each interface in the interface combination. The interface combination includes at least two interfaces, and the at least two interfaces are complementary in terms of data block download speed. The dedicated data block combination corresponding to the interface combination includes all data blocks of the target file.

[0032] S220. For each interface in the interface combination, if it is detected that the private taskbar corresponding to the interface is missing the first file identifier compared to the combined taskbar, the corresponding data block in the file corresponding to the first file identifier is downloaded through the interface, and the first file identifier is added to the private taskbar of the interface. The combined taskbar is the taskbar for all interfaces in the interface combination.

[0033] This embodiment sets up a private taskbar for each interface in the interface combination, and a combined taskbar for the interface combination. The private taskbar is used to record the tasks that the corresponding interface has completed, and the combined taskbar is used to record the tasks initiated by the interface combination.

[0034] The interfaces in the interface group have the same function and are all configured to prioritize the execution of supplementary download tasks. Specifically, for each interface in the interface group, when the processor detects that the current download task of the interface has ended, it first determines whether the group task bar includes a first file identifier that is not in the private task bar of that interface. If so, it is determined that the interface needs to prioritize the execution of supplementary download tasks. Therefore, the first file identifier is used as the file identifier to be downloaded for that interface, and the corresponding data block of the file corresponding to the first file identifier is downloaded through the interface. Then, the first file identifier is added to the private task bar of that interface.

[0035] S230. For each file identifier, if it is detected that all data blocks under the file identifier have been downloaded, determine the target file for the file identifier based on all data blocks under the file identifier.

[0036] The technical solution provided in this invention configures each interface in the interface combination to prioritize the execution of supplementary download tasks. This ensures that once a file corresponding to a file identifier is started for download by any interface in the interface combination, other interfaces will complete the supplementary download in the shortest possible time, thereby completing the download of all its data blocks. This guarantees a short overall download time for the target file and maximizes throughput in scenarios where the amount of download in a single session is limited, making full use of the parallel transmission capabilities of multiple interfaces.

[0037] Based on the aforementioned embodiments, after S210, the method further includes: for each interface in the interface combination, if it is detected that the private taskbar corresponding to the interface does not lack a first file identifier compared to the combined taskbar, obtaining a second file identifier to be downloaded, downloading the corresponding data block in the file corresponding to the second file identifier through the interface, and adding the second file identifier to the main taskbar, the combined taskbar for the interface combination, and the private taskbar for the interface.

[0038] Specifically, for each interface in the interface combination, if the interface is detected to be currently idle, it checks whether the private taskbar corresponding to the interface is missing a first file identifier compared to the combined taskbar. If not, it obtains the second file identifier to be downloaded, then downloads the corresponding data block from the file corresponding to the second file identifier through the interface, and adds the second file identifier to the main taskbar, the combined taskbar, and the interface's private taskbar. This embodiment further illustrates that each interface in the interface combination must prioritize completing the supplementary download task. If there is no supplementary download task, it initiates the download of the dedicated data block of the new file.

[0039] Furthermore, for each interface in the interface combination, if the identifier of the second file to be downloaded is obtained, it is confirmed whether the identifier of the second file exists in the main taskbar. If it exists, it means that the identifier of the second file has been downloaded through other interfaces. Therefore, the identifier of the second file to be downloaded is obtained again, and the process of confirming whether the identifier of the second file exists in the main taskbar is repeated until the identifier of the second file does not exist in the main taskbar. Then, the file corresponding to the identifier of the second file is determined to be the file to be downloaded.

[0040] Understandably, for each interface in the interface combination, if its corresponding download task is a supplementary download task, the processor does not need to add a file identifier to the main taskbar after the supplementary download task is completed; if its corresponding download task is a non-supplementary download task, the processor needs to add a file identifier to the main taskbar after the download task is completed.

[0041] Based on the aforementioned embodiments, for each interface in the interface combination, multiple crawling parameter combinations are randomly generated based on a predetermined time interval, and the average crawling speed corresponding to each crawling parameter combination is determined; based on the average crawling speed corresponding to each crawling parameter combination, an optimal crawling parameter combination is determined, wherein the crawling parameter combination includes single request volume, number of concurrent threads, and request interval; correspondingly, for each interface in the interface combination, based on the optimal crawling parameter combination of the interface, the interface is driven to download the corresponding data blocks of each target file in the target file combination.

[0042] Specifically, during the data crawling process based on interface combinations, the parameter data in each crawling parameter combination is randomly changed at regular intervals, such as the single request volume, the number of concurrent threads, and the request interval. For each crawling parameter combination, multiple crawling speeds of each interface under that crawling parameter combination are recorded at predetermined time intervals, and the average crawling speed corresponding to these multiple crawling speeds is determined. Based on the maximum crawling speed of each interface and the average crawling speed under different crawling parameter combinations, a parameter combination and predicted speed model is established through a fitting algorithm. Based on this model, the optimal crawling parameter combination that makes the download speed closest to the maximum speed of the interface can be predicted. The fitting algorithm can be a linear regression algorithm or a machine learning model. For example, the crawling parameter combination includes the single request volume, the number of concurrent threads, and the request interval. The average crawling speed of interface A under [M11, M21, M31] is N1; the average crawling speed under [M12, M22, M32] is N2; and the average crawling speed under [M13, M23, M33] is N3. The maximum speed N of interface A can typically be obtained based on publicly available information about interface A, for example, N. The optimal combination of fetching parameters is determined based on N, N1, N2, and N3. Once the optimal combination of fetching parameters for each interface is determined, the corresponding interface can be driven to perform data download based on this optimal combination. Since the terminal network environment and server network status are not static, the optimal combination of fetching parameters in this embodiment is dynamically changed to allow each interface to adapt to changes in the terminal network environment and server network status in a timely manner.

[0043] This embodiment is suitable for batch download scenarios. It drives the corresponding interface to capture data based on the optimal combination of capture parameters, which can further improve the download speed of batch files.

[0044] Figure 3 This is another flowchart illustrating the data capture method provided in this embodiment of the invention. This embodiment adds a step of downloading the complete file based on a load balancer interface, building upon the previous examples. For example... Figure 3 As shown, the method in this embodiment may specifically include: S310. Determine the load balancing interface, the interface combination, and the dedicated data blocks of each interface in the interface combination. The interface combination includes at least two interfaces, and the at least two interfaces are complementary in terms of data block download speed. The dedicated data block combination corresponding to the interface combination includes all data blocks of the target file.

[0045] A balanced interface, as the name suggests, downloads each data block of a target file at a similar speed, meaning its average download speed deviation is within an acceptable range.

[0046] S320. For each interface in the interface combination, if it is detected that the private taskbar corresponding to the interface is missing the first file identifier compared to the combined taskbar, the corresponding data block in the file corresponding to the first file identifier is downloaded through the interface, and the first file identifier is added to the private taskbar of the interface. The combined taskbar is the taskbar for all interfaces in the interface combination.

[0047] S330. For each file identifier, if it is detected that all data blocks under the file identifier have been downloaded, determine the target file for the file identifier based on all data blocks under the file identifier.

[0048] S340. Obtain the identifier of the third file to be downloaded, download the full text of the file corresponding to the third file identifier through the load balancer interface, and add the third file identifier to the main taskbar and the private taskbar for the load balancer interface respectively.

[0049] The load balancer interface is used to download all data blocks of a file. When the processor detects that all data blocks of a file have been downloaded through the load balancer interface, it adds a third file identifier status to both the main taskbar and the load balancer interface's private taskbar.

[0050] Furthermore, the identifier of the third file to be downloaded is obtained. If the identifier of the third file to be downloaded is not detected in the main taskbar, the full text of the file corresponding to the third file identifier is downloaded through the load balancer interface.

[0051] Specifically, if the third file identifier is in the main taskbar, it means that the third file identifier has been downloaded through other interfaces. Therefore, the third file identifier to be downloaded is retrieved again, and the process returns to the step of confirming whether the third file identifier exists in the main taskbar. This continues until the third file identifier is no longer in the main taskbar. Then, the file corresponding to the third file identifier is determined to be the file to be downloaded, and all data blocks in the file corresponding to the third file identifier are downloaded through the load balancing interface.

[0052] In this embodiment, the load balancing interface and the interface combination do not interact. The load balancing interface only needs to determine whether the current third file identifier is the real file identifier to be downloaded through the main task bar, which simplifies the data download process of the load balancing interface. The simplification of the load balancing interface data download process can improve the data capture speed of the load balancing interface.

[0053] The technical solution provided in this embodiment of the invention, based on the aforementioned examples, also includes a load balancing interface for downloading the full text of a file. This combination of interfaces, used in conjunction with the load balancing interface, can further improve the download speed of batch files.

[0054] Based on the aforementioned embodiments, for the load balancing interface, multiple crawling parameter combinations are randomly generated based on a predetermined time interval, and the average crawling speed corresponding to each crawling parameter combination is determined; based on the average crawling speed corresponding to each crawling parameter combination, the optimal crawling parameter combination is determined, and the crawling parameter combination includes the single request volume, the number of concurrent threads, and the request interval; accordingly, based on the optimal crawling parameter combination, the load balancing interface is driven to download all data blocks of each target file in the target file combination.

[0055] Taking a patent document as an example, this invention details the data retrieval process of an embodiment. The patent document includes a bibliographic entry page, a main text page, and drawing pages. For ease of description, the bibliographic entry page is designated as the first data block, the main text page as the second data block, and the drawing pages as the third data block. The interface is a publicly available patent download interface, such as the download interface on the State Intellectual Property Office website or the European Patent Office website. It should be noted that the data blocks included in the target document can be determined based on the user's actual needs; for example, the target document can also be a combination of a bibliographic entry page, a specification abstract page, and an abstract drawing page.

[0056] Record the download speed of each interface for each data block, including the download speed of each interface for the first data block, the second data block, and the third data block. After a period of time (which can be customized, such as 30 minutes), calculate the average download speed of each interface for each data block. Filter the average download speed deviation rate of each interface, and identify interfaces whose average download speed deviation rate is greater than a predetermined threshold as interface combinations. This interface combination is designed as a complementary interface combination. Specifically, this interface combination includes at least a first interface and a second interface. (For example, the first interface has a fast download speed for the first and third data blocks, but a slower download speed for the second data block, resulting in a large average download speed deviation rate for the first interface; conversely, the second interface has a slower download speed for the first and third data blocks, but a faster download speed for the second data block, resulting in a large deviation rate for the second interface. Thus, the first and second interfaces can form a complementary interface combination.)

[0057] Each interface within an interface group has its own private taskbar, and the interface group corresponds to a group taskbar. If there is an interface whose average download speed deviation rate is less than a predetermined deviation rate threshold, then that interface is designated as the balancing interface. The balancing interface also has its own private taskbar, while all interfaces share a common taskbar.

[0058] For the load balancer interface, after capturing the full text of the target file, the processor records the file identifier of the target file in the private taskbar of the corresponding load balancer interface and also uploads the file identifier to the main taskbar. For each interface in the interface combination, after starting the data capture operation of the target file through each interface and capturing some data blocks of the target file, the processor records the file identifier of the target file in the private taskbar of the corresponding interface, uploads the file identifier to the main taskbar, and also uploads it to the combined taskbar corresponding to the interface combination. For each interface in the interface combination, after performing supplementary download of the target file through each interface and completing the supplementary download, the processor records the file identifier of the target file in the private taskbar of the corresponding interface.

[0059] Specifically, regardless of whether the first data block and the third data block of the file are captured first through the first interface, or the second data block of the file is captured first through the second interface, when the capture of the corresponding data block is finished, the file identifier of the corresponding file is uploaded to the combined task bar and the main task bar, and added to the private task bar of the corresponding interface; when the other interface is idle, the processor captures the remaining data blocks of the file through the other interface, and after the capture is finished, the file identifier of the file is added to the private task bar of the other interface.

[0060] After a target file has been completely downloaded through the load balancer interface, when starting to download the next file, the file identifier of the next file should be obtained first to determine whether the file identifier of the file is included in the main task bar. Only if the file identifier of the file is not included in the main task bar should the file corresponding to the file identifier be determined as the file to be downloaded; otherwise, the file identifier of the file to be downloaded should be obtained again.

[0061] During the data fetching process based on balanced interfaces and interface combinations, the parameters in the fetching parameter combinations are randomly changed at regular intervals, such as the number of requests per request, the number of concurrent threads, and the request interval. The fetching speed corresponding to each fetching parameter combination is recorded to fit and obtain the optimal fetching parameter combination. For example, the fetching speed of interface A is N1 when the fetching parameter combination is [M11, M21, M31]; N2 when the fetching parameter combination is [M12, M22, M32]; and N3 when the fetching parameter combination is [M13, M23, M33]. The maximum speed of interface A can usually be obtained based on publicly available information about interface A, such as N. The optimal fetching parameter combination can then be determined based on the above data fitting, and the corresponding interface can be driven to perform data fetching according to this combination. This embodiment is suitable for batch download scenarios. By driving the corresponding interface to perform data fetching based on the optimal fetching parameter combination, the download speed of batch files can be further improved.

[0062] Figure 4 This is a schematic diagram of the data capture device provided in an embodiment of the present invention. Figure 4 As shown, the data capture device includes: The interface combination module 110 is used to determine the interface combination and the dedicated data block of each interface in the interface combination. The interface combination includes at least two interfaces, and the at least two interfaces are complementary in terms of data block download speed. The dedicated data block combination corresponding to the interface combination includes all data blocks of the target file. The download module 120 is used to download corresponding data blocks from the target file through each of the interfaces in the interface combination, wherein the identifier of each data block carries a file identifier; The result module 130 is used to determine the target file for each file identifier based on all the data blocks under the file identifier when it is detected that all data blocks under the file identifier have been downloaded.

[0063] The technical solution provided by this invention, since at least two interfaces in the interface combination are matched with dedicated data blocks, and all dedicated data blocks corresponding to the at least two interfaces include all data blocks of the target file, and the at least two interfaces are complementary in terms of data block download speed, the dedicated data blocks of each interface in the interface combination do not overlap and can be pieced together to form a complete "high-speed download map". Thus, the overall performance can be maximized through collaborative division of labor. Therefore, by downloading each data block of the file based on each interface in the interface combination, the overall download time of the file can be minimized, thereby significantly improving the download speed of batch files and the user experience.

[0064] In one embodiment, the download module 120 is specifically used for: For each interface in the interface combination, if it is detected that the private taskbar corresponding to the interface lacks a first file identifier compared to the combined taskbar, the corresponding data block in the file corresponding to the first file identifier is downloaded through the interface, and the first file identifier is added to the private taskbar of the interface. The combined taskbar is the taskbar for all interfaces in the interface combination.

[0065] In one embodiment, the download module 120 is further configured to: For each interface in the interface combination, if it is detected that the private taskbar corresponding to the interface is not missing the first file identifier compared to the combined taskbar, the second file identifier to be downloaded is obtained, and the corresponding data block in the file corresponding to the second file identifier is downloaded through the interface, and the second file identifier is added to the main taskbar, the combined taskbar, and the private taskbar for the interface.

[0066] In one embodiment, the interface combination module 110 is further configured to: Determine the load balancer interface; The device also includes a load balancing module, which is used for: Obtain the identifier of the third file to be downloaded, download the full text of the file corresponding to the third file identifier through the load balancer interface, and add the third file identifier to the main taskbar and the private taskbar for the load balancer interface respectively.

[0067] In one embodiment, the load balancing module is used for: Obtain the identifier of the third file to be downloaded. If the identifier of the third file to be downloaded is not detected in the main task bar, download the full text of the file corresponding to the third file identifier through the load balancer interface.

[0068] In one embodiment, the interface combination module 110 is specifically used for: The download speed of each interface for each data block is periodically calculated, and the average download speed of each interface for each data block is determined based on the download speed of each interface for each data block. Based on the average download speed of each interface for each data block, determine the average download speed deviation rate of each interface; Identify all candidate interfaces whose average download speed deviation rate exceeds a predetermined threshold; Based on the average download speed of each candidate interface for each data block, the interface combination and the dedicated data block for each interface in the interface combination are determined.

[0069] In one embodiment, the device further includes a parameter tuning module, which can be used for: For each interface, multiple crawling parameter combinations are randomly generated based on a predetermined time interval, and the average crawling speed of the interface under each crawling parameter combination is determined; based on the maximum crawling speed of the interface and the average crawling speed corresponding to each crawling parameter combination, the optimal crawling parameter combination of the interface is determined, wherein the crawling parameter combination includes single request volume, number of concurrent threads and request interval; Accordingly, the download module 120 is used for: For each of the interfaces in the interface combination, the interface is driven to download the corresponding data blocks of each target file in the target file combination according to the optimal capture parameter combination of the interface.

[0070] The data capture device provided in the embodiments of the present invention can execute the data capture method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0071] It is worth noting that the various units and modules included in the above-mentioned data capture device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of the present invention.

[0072] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0073] like Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0074] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0075] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data fetching methods.

[0076] In some embodiments, the data grabbing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via read-only memory (ROM) 12 and / or communication unit 19. When the computer program is loaded into random access memory (RAM) 13 and executed by processor 11, one or more steps of the data grabbing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the data grabbing method by any other suitable means (e.g., by means of firmware).

[0077] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0078] Computer programs used to implement the data capture method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0079] This invention provides a computer-readable storage medium storing computer instructions for causing a processor to execute a data fetching method, including: Determine the interface combination and the dedicated data block of each interface in the interface combination. The interface combination includes at least two interfaces, and the at least two interfaces are complementary in terms of data block download speed. The dedicated data block combination corresponding to the interface combination includes all data blocks of the target file. Each of the interfaces in the interface combination downloads the corresponding data block in the target file, and the identifier of each data block carries a file identifier; For each of the aforementioned file identifiers, if it is detected that all data blocks under the file identifier have been downloaded, the target file for the file identifier is determined based on all the data blocks under the file identifier.

[0080] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0081] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0082] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0083] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0084] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0085] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements a data capture method according to any embodiment of the invention.

[0086] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0087] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0088] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data scraping method, characterized by, The method includes: Determine the interface combination and the dedicated data block of each interface in the interface combination. The interface combination includes at least two interfaces, and the at least two interfaces are complementary in terms of data block download speed. The dedicated data block combination corresponding to the interface combination includes all data blocks of the target file. Each of the interfaces in the interface combination downloads the corresponding data block in the target file, and the identifier of each data block carries a file identifier; For each of the aforementioned file identifiers, if it is detected that all data blocks under the file identifier have been downloaded, the target file for the file identifier is determined based on all the data blocks under the file identifier.

2. The method of claim 1, wherein, The step of downloading the corresponding data block from the target file through each of the interfaces in the interface combination includes: For each interface in the interface combination, if it is detected that the private taskbar corresponding to the interface lacks a first file identifier compared to the combined taskbar, the corresponding data block in the file corresponding to the first file identifier is downloaded through the interface, and the first file identifier is added to the private taskbar of the interface. The combined taskbar is the taskbar for all interfaces in the interface combination.

3. The method of claim 2, wherein, Also includes: For each interface in the interface combination, if it is detected that the private taskbar corresponding to the interface is not missing the first file identifier compared to the combined taskbar, the second file identifier to be downloaded is obtained, and the corresponding data block in the file corresponding to the second file identifier is downloaded through the interface, and the second file identifier is added to the main taskbar, the combined taskbar, and the private taskbar for the interface.

4. The method of claim 2, wherein, Determining the interface combination also includes: Determine the load balancer interface; The method also includes: Obtain the identifier of the third file to be downloaded, download the full text of the file corresponding to the third file identifier through the load balancer interface, and add the third file identifier to the main taskbar and the private taskbar for the load balancer interface respectively.

5. The method of claim 4, wherein, The step of obtaining the identifier of the third file to be downloaded and downloading the full text of the file corresponding to the third file identifier through the load balancing interface includes: Obtain the identifier of the third file to be downloaded. If the identifier of the third file to be downloaded is not detected in the main task bar, download the full text of the file corresponding to the third file identifier through the load balancer interface.

6. The method of claim 1, wherein, The determination of the interface combination and the dedicated data block for each interface in the interface combination includes: The download speed of each interface for each data block is periodically calculated, and the average download speed of each interface for each data block is determined based on the download speed of each interface for each data block. Based on the average download speed of each interface for each data block, determine the average download speed deviation rate of each interface; Identify all candidate interfaces whose average download speed deviation rate exceeds a predetermined threshold; Based on the average download speed of each candidate interface for each data block, the interface combination and the dedicated data block for each interface in the interface combination are determined.

7. The method of claim 1, wherein, Also includes: For each of the interfaces in the interface combination, multiple crawling parameter combinations are randomly generated based on a predetermined time interval, and the average crawling speed of the interface under each crawling parameter combination is determined; based on the maximum crawling speed of the interface and the average crawling speed corresponding to each crawling parameter combination, the optimal crawling parameter combination of the interface is determined, wherein the crawling parameter combination includes single request quantity, number of concurrent threads and request interval; The step of downloading the corresponding data block from the target file through each of the interfaces in the interface combination includes: For each of the interfaces in the interface combination, the interface is driven to download the corresponding data blocks of each target file in the target file combination according to the optimal capture parameter combination of the interface.

8. An electronic device, comprising: The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the data capture method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data capture method according to any one of claims 1-7.

10. A computer program product, characterised in that, The computer program product includes a computer program that, when executed by a processor, implements the data capture method according to any one of claims 1-7.

Images