Data acquisition method, device, electronic device and storage medium

Abstract

The present disclosure provides a data acquisition method, device, electronic device and storage medium, wherein the method comprises: in response to a data acquisition request, determining request queue description information based on a resource location identifier indicated by the request; storing the data acquisition request in a request queue that matches the request; in response to a queue acquisition instruction sent by a downloader, sending request queue description information corresponding to a currently stored request queue to the downloader, so that the downloader extracts target request queue description information from the request queue description information of each request queue based on the request queue description information of the currently responded data acquisition request; the target request queue description information is different from the request queue description information corresponding to the data acquisition request in the current response of the downloader; in response to a request acquisition instruction sent by the downloader, extracting a target data acquisition request from a target request queue corresponding to the target request queue description information indicated by the instruction, and sending the target data acquisition request to the downloader.

Description

Technical Field

[0001] The present disclosure relates to the field of data acquisition, and in particular, to a data acquisition method, device, electronic device and storage medium. Background Art

[0002] Currently, data is showing an explosive growth trend. How to quickly and efficiently collect and utilize this data has become a new challenge. In actual operation, data information can be obtained from the website through web crawlers or website public APIs.

[0003] The most common web crawler is the distributed crawler, which can collect large-scale data. In order to improve efficiency, distributed crawlers can have multiple downloaders and run in a distributed manner. In order to avoid waste of resources caused by different downloaders responding to the same data acquisition request, a single request queue is usually used, and different downloaders cannot process crawler requests for the same site in the same request queue at the same time. Therefore, when there are adjacent crawler requests for the same site in the same queue, the requests in the queue may not be dequeued in time, resulting in waste of download resources. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a data acquisition method, device, electronic device and storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a data acquisition method, including:

[0006] In response to the data acquisition request, determining request queue description information based on the resource location identifier indicated by the data acquisition request; the request queue description information includes domain name indication information;

[0007] storing the data acquisition request in a request queue that matches the request queue description information;

[0008] In response to the queue acquisition instruction sent by the downloader, the request queue description information corresponding to each of the currently stored request queues is sent to the downloader, so that the downloader extracts the target request queue description information from the request queue description information corresponding to each of the request queues based on the request queue description information corresponding to the data acquisition request currently responded to; wherein the target request queue description information is different from the request queue description information corresponding to the data acquisition request in the current response of the downloader;

[0009] In response to the request acquisition instruction sent by the downloader, a target data acquisition request is extracted from the target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, and the target data acquisition request is sent to the downloader.

[0010] In an optional implementation manner, the request queue description information is determined based on the resource location identifier indicated by the data acquisition request; the request queue description information includes domain name indication information, including:

[0011] Extracting the domain name information, transmission protocol information and port information corresponding to the data acquisition request from the resource location identifier;

[0012] Determine request queue description information based on the domain name information, the transmission protocol information, and the port information.

[0013] In an optional implementation manner, storing the data acquisition request in a request queue that matches the request queue description information includes:

[0014] Querying a request queue that matches the request queue description information, and storing the data acquisition request in the queried request queue;

[0015] When a request queue matching the request queue description information cannot be found, a request queue matching the request queue description information is generated, and the data acquisition request is stored in the generated request queue.

[0016] In an optional implementation manner, extracting the target data acquisition request from the target request queue corresponding to the target request queue description information indicated by the request acquisition instruction includes:

[0017] Determine a queue type of the target request queue; the queue type includes at least one of a first-in-first-out queue and a priority queue;

[0018] Based on the dequeue mode corresponding to the queue type, a target data acquisition request is determined from the target request queue.

[0019] In an optional implementation manner, the method further includes:

[0020] Obtaining request queue parameters corresponding to multiple users through a request proxy module; the request queue parameters include request queue description information of multiple request queues corresponding to the users, and queue lengths of the request queues of the users;

[0021] Based on the acquired request queue parameters, the request information of the request proxy module is updated; the request information includes a plurality of information groups, each of which includes the request queue description information corresponding to the information group, and the queue length of the request queue corresponding to the request queue description information;

[0022] The sending the request queue description information corresponding to each currently stored request queue to the downloader includes:

[0023] In the case where the downloader is the downloader corresponding to the request proxy module, sending the request queue description information corresponding to each currently stored information group to the downloader;

[0024] The step of extracting a target data acquisition request from a target request queue corresponding to the target request queue description information indicated by the request acquisition instruction comprises:

[0025] In the case that there are multiple target request queues, determining the target request queue with the highest queue length among the target request queues;

[0026] The target data acquisition request is extracted from the target request queue with the highest queue length.

[0027] In an optional implementation manner, the method further includes:

[0028] Obtaining, through the request proxy module, request information of other request proxy modules corresponding to the request proxy module;

[0029] The updating of the request information of the request proxy module based on the acquired request queue parameter includes:

[0030] Based on the acquired request queue parameters and the request information of the other request proxy modules, the request information of the request proxy module is updated.

[0031] In a second aspect, the present disclosure also provides a data acquisition method, including:

[0032] Sending a queue acquisition instruction to a request management system so that the request management system feeds back request queue description information corresponding to a currently stored request queue; the request queue stores data acquisition requests with the same request queue description information; the request queue description information includes domain name indication information;

[0033] Based on the request queue description information corresponding to the currently executed data acquisition task, extracting the target request queue description information from the request queue description information corresponding to the currently stored request queue; wherein the target request queue description information is different from the request queue description information corresponding to the currently executed data acquisition task;

[0034] Sending a request acquisition instruction to the request management system, so that the request management system extracts a target data acquisition request from a target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, and feeds back the target data acquisition request;

[0035] Execute the data acquisition task corresponding to the target data acquisition request.

[0036] In an optional implementation manner, sending a queue acquisition instruction to the request management system includes:

[0037] Sending a queue acquisition instruction to the request proxy module of the request management system, so that the request management system feeds back request queue description information corresponding to each currently stored information group through the request proxy module; the information group is determined by request queue parameters corresponding to multiple users; the request queue parameters include request queue description information of multiple request queues corresponding to the user, and queue length of the request queue of the user;

[0038] The sending a request acquisition instruction to the request management system includes:

[0039] A request acquisition instruction is sent to the request agent module so that the request management system determines the target request queue with the highest queue length among the target request queues through the request agent module when there are multiple target request queues corresponding to the target request queue description information indicated by the request acquisition instruction, and extracts the target data acquisition request from the target request queue with the highest queue length.

[0040] In a third aspect, the present disclosure also provides a data acquisition device, including:

[0041] A generating module, configured to determine, in response to a data acquisition request, request queue description information based on a resource location identifier indicated by the data acquisition request; the request queue description information includes domain name indication information;

[0042] A storage module, used for storing the data acquisition request in a request queue matching the request queue description information;

[0043] A first sending module, configured to send the currently stored request queue description information corresponding to each of the request queues to the downloader in response to the queue acquisition instruction sent by the downloader, so that the downloader extracts the target request queue description information from the request queue description information corresponding to each of the request queues based on the request queue description information corresponding to the currently responded data acquisition request; wherein the target request queue description information is different from the request queue description information corresponding to the data acquisition request in the current response of the downloader;

[0044] The second sending module is used to respond to the request acquisition instruction sent by the downloader, extract the target data acquisition request from the target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, and send the target data acquisition request to the downloader.

[0045] In an optional implementation manner, the generating module determines the request queue description information based on the resource location identifier indicated by the data acquisition request; when the request queue description information includes domain name indication information, it is used to:

[0046] Extracting the domain name information, transmission protocol information and port information corresponding to the data acquisition request from the resource location identifier;

[0047] Determine request queue description information based on the domain name information, the transmission protocol information, and the port information.

[0048] In an optional implementation manner, when storing the data acquisition request in a request queue matching the request queue description information, the storage module is used to:

[0049] Querying a request queue that matches the request queue description information, and storing the data acquisition request in the queried request queue;

[0050] When a request queue matching the request queue description information cannot be found, a request queue matching the request queue description information is generated, and the data acquisition request is stored in the generated request queue.

[0051] In an optional implementation manner, when the second sending module extracts the target data acquisition request from the target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, it is used to:

[0052] Determine a queue type of the target request queue; the queue type includes at least one of a first-in-first-out queue and a priority queue;

[0053] Based on the dequeue mode corresponding to the queue type, a target data acquisition request is determined from the target request queue.

[0054] In an optional implementation manner, the device further includes an agent module, configured to:

[0055] Obtaining request queue parameters corresponding to multiple users through a request proxy module; the request queue parameters include request queue description information of multiple request queues corresponding to the users, and queue lengths of the request queues of the users;

[0056] Based on the acquired request queue parameters, the request information of the request proxy module is updated; the request information includes a plurality of information groups, each of which includes the request queue description information corresponding to the information group, and the queue length of the request queue corresponding to the request queue description information;

[0057] The sending the request queue description information corresponding to each currently stored request queue to the downloader includes:

[0058] In the case where the downloader is the downloader corresponding to the request proxy module, sending the request queue description information corresponding to each currently stored information group to the downloader;

[0059] The step of extracting a target data acquisition request from a target request queue corresponding to the target request queue description information indicated by the request acquisition instruction comprises:

[0060] In the case that there are multiple target request queues, determining the target request queue with the highest queue length among the target request queues;

[0061] The target data acquisition request is extracted from the target request queue with the highest queue length.

[0062] In an optional implementation manner, the proxy module is further used to:

[0063] Obtaining, through the request proxy module, request information of other request proxy modules corresponding to the request proxy module;

[0064] The updating of the request information of the request proxy module based on the acquired request queue parameter includes:

[0065] Based on the acquired request queue parameters and the request information of the other request proxy modules, the request information of the request proxy module is updated.

[0066] In a fourth aspect, the present disclosure also provides a data acquisition device, including:

[0067] A first instruction sending module is used to send a queue acquisition instruction to the request management system, so that the request management system feeds back request queue description information corresponding to the currently stored request queue; the request queue stores data acquisition requests with the same request queue description information; the request queue description information includes domain name indication information;

[0068] A determination module, configured to extract target request queue description information from the request queue description information corresponding to the currently stored request queue based on the request queue description information corresponding to the currently executed data acquisition task; wherein the target request queue description information is different from the request queue description information corresponding to the currently executed data acquisition task;

[0069] A second instruction sending module is used to send a request acquisition instruction to the request management system, so that the request management system extracts a target data acquisition request from a target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, and feeds back the target data acquisition request;

[0070] An execution module is used to execute the data acquisition task corresponding to the target data acquisition request.

[0071] In an optional implementation manner, when the first instruction sending module sends the queue acquisition instruction to the request management system, it is used to:

[0072] Sending a queue acquisition instruction to the request proxy module of the request management system, so that the request management system feeds back request queue description information corresponding to each currently stored information group through the request proxy module; the information group is determined by request queue parameters corresponding to multiple users; the request queue parameters include request queue description information of multiple request queues corresponding to the user, and queue length of the request queue of the user;

[0073] When sending the request acquisition instruction to the request management system, the second instruction sending module is used to:

[0074] A request acquisition instruction is sent to the request agent module so that the request management system determines the target request queue with the highest queue length among the target request queues through the request agent module when there are multiple target request queues corresponding to the target request queue description information indicated by the request acquisition instruction, and extracts the target data acquisition request from the target request queue with the highest queue length.

[0075] In a fifth aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps in the above-mentioned first aspect, second aspect, third aspect, fourth aspect, or any possible implementation of the first aspect, second aspect, third aspect, fourth aspect are performed.

[0076] In a sixth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect, second aspect, third aspect, fourth aspect, or any possible implementation of the first aspect, second aspect, third aspect, fourth aspect are executed.

[0077] The present disclosure discloses a data acquisition method, device, electronic device and storage medium, the method comprising: in response to a data acquisition request, determining request queue description information based on a resource location identifier indicated by the request; storing the data acquisition request in a request queue that matches the request; in response to a queue acquisition instruction sent by a downloader, sending request queue description information corresponding to a currently stored request queue to the downloader, so that the downloader extracts target request queue description information from the request queue description information of each request queue based on the request queue description information of the currently responded data acquisition request; the target request queue description information is different from the request queue description information corresponding to the data acquisition request in the current response of the downloader; in response to the request acquisition instruction sent by the downloader, extracting the target data acquisition request from the target request queue corresponding to the target request queue description information indicated by the instruction, and sending it to the downloader. The disclosed embodiment can respectively store data acquisition requests corresponding to different request queue description information by adopting multiple request queues, that is, data acquisition requests corresponding to different domain names can be stored in different request queues. In this way, the downloader can independently select the request corresponding to the required request queue description information. In this way, the downloader can select the data acquisition requests corresponding to different domain names for parallel processing, thereby maximizing the parallel processing capability of the downloader and improving the data acquisition efficiency.

[0078] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work;

[0080] Figure 1 A schematic diagram of a data acquisition system provided by an embodiment of the present disclosure is shown;

[0081] Figure 2 A flow chart of a data acquisition method provided by an embodiment of the present disclosure is shown;

[0082] Figure 3 One of the schematic diagrams of a request queue in a request management system provided by an embodiment of the present disclosure is shown;

[0083] Figure 4 A second schematic diagram of a request queue in a request management system provided by an embodiment of the present disclosure is shown;

[0084] Figure 5 A third schematic diagram of a request queue in a request management system provided by an embodiment of the present disclosure is shown;

[0085] Figure 6 One of the schematic diagrams of another request management system provided by an embodiment of the present disclosure is shown;

[0086] Figure 7 A second schematic diagram of another request management system provided by an embodiment of the present disclosure is shown;

[0087] Figure 8 A flow chart showing another data acquisition method provided by an embodiment of the present disclosure;

[0088] Fig. 9 A schematic diagram of a data acquisition device provided by an embodiment of the present disclosure is shown;

[0089] Fig.10 A schematic diagram showing another data acquisition device provided by an embodiment of the present disclosure;

[0090] Fig.11 A schematic diagram of a data acquisition electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0092] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0093] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0094] Through research, it is found that the current data acquisition method usually adopts a request queue to store data acquisition requests, and deploys multiple downloaders, adopts a distributed download method, and each downloader obtains data requests from the request queue. Usually, users' data acquisition requests are often concentrated in a certain site, resulting in adjacent requests in the queue belonging to the same site, and a downloader can execute data acquisition tasks of different sites in parallel, but cannot execute data acquisition tasks of the same site at the same time. Due to the characteristics of the request queue, requests of the same site will wait for the downloader of the executable task, while requests of other sites at the back of the queue are blocked by the waiting requests and cannot be obtained by the downloader, resulting in the requests in the queue cannot be dequeued in time, resulting in a waste of download resources.

[0095] Based on the above research, the present disclosure provides a data acquisition system, including a request management system and a downloader. The request management system can determine the request queue description information based on the resource location identifier, and divide the data acquisition requests into different queues based on this, that is, the data acquisition requests corresponding to different domain names can be stored in different request queues. In this way, the downloader can communicate with the request management system, select the data acquisition request with different request queue description information from the currently executing task, and execute the download. In this way, the downloader can select the data acquisition requests corresponding to different domain names for parallel processing, thereby maximizing the parallel processing capability of the downloader and improving the data acquisition efficiency.

[0096] To facilitate understanding of this embodiment, a data acquisition system disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the data acquisition system provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities.

[0097] See also Figure 1 FIG. 1 is a schematic diagram of a data acquisition system provided by an embodiment of the present disclosure, which includes: a request management system 101 and a downloader 102, wherein:

[0098] The request management system 101 can communicate with the downloader 102 .

[0099] The request management system 101 is used to manage data acquisition requests, and after receiving an acquisition instruction from the downloader 102, sends the target data acquisition request to the downloader 102 according to certain rules.

[0100] The downloader 102 is used for the request management system 101 to send an acquisition instruction, and after receiving a target data acquisition request from the request management system 101, executes a data acquisition task corresponding to the acquisition request.

[0101] Based on the above data acquisition system, the embodiment of the present disclosure further provides a data acquisition method, the execution subject of which is a request management system.

[0102] See also Figure 1 FIG. 2 is a flow chart of a data acquisition method provided by an embodiment of the present disclosure, wherein the method includes steps S201 to S204, wherein:

[0103] S201: In response to a data acquisition request, determine request queue description information based on a resource location identifier indicated by the data acquisition request; the request queue description information includes domain name indication information.

[0104] Among them, the resource location identifier is a concise representation of the location and access method of resources obtained on the Internet. It is the address of standard resources on the Internet. The address consists of three parts: protocol header, server address, and file path. The general grammatical format is: protocol + IP address (domain name) + port number + path + parameter + query + information fragment. For example, a typical address is:

[0105] http: / / www.example.com:80 / subject / pythonzly / index.shtml

[0106] Among them, http is the protocol, which is used to tell the browser how to handle the file to be opened. Different protocols represent different resource search and transmission methods. Commonly used protocols on the Internet include file, ftp, http, https and mailto. www.example.com:8080 is the server address and port. The server address refers to the host name or IP address of the server where the resources are stored. Its purpose is to identify the only computer on the Internet and find this computer through this address. The port is the number after the address and the colon, which is used to identify different programs running on a computer. Each network program corresponds to one or more specific port numbers. For example, the default port number of the http program is 80, and the default port number of the https program is 443. subject / pythonzly / index.shtml is the path, which is generally used to specify the location of the resource requested in the server.

[0107] Extracting the domain name information, transmission protocol information and port information corresponding to the data acquisition request from the resource location identifier;

[0108] In the above example, http is the transmission protocol information, www.example.com is the domain name information, and 8080 is the port information. The port information can also be set to a null character. The domain name is a string of names separated by dots, which is the name of a computer or computer group on the Internet. It is used to identify the electronic location of the computer during data transmission. The purpose of a domain name is to become the address of a group of servers that is easy to remember and communicate.

[0109] Then, based on the domain name information, the transmission protocol information and the port information, the request queue description information corresponding to the data acquisition request is determined.

[0110] Exemplarily, the request queue description information may be obtained by connecting the domain name information, transmission protocol information and port information through certain symbols, for example: http-www.example.com-8080, or by flipping the characters of the above information, such as: 0808:moc.elpmaxe.www / / :ptth, or other methods, which are not limited here.

[0111] S202: Store the data acquisition request in a request queue that matches the request queue description information.

[0112] Specifically, in the existing request queues, a request queue matching the request queue description information is searched, and if a request queue matching the request queue description information can be searched, the data acquisition request is stored in the searched request queue;

[0113] If a request queue matching the request queue description information cannot be found, a request queue matching the request queue description information is generated, and the data acquisition request is stored in the generated request queue.

[0114] See also Figure 3 As shown, it is one of the schematic diagrams of a request queue in a request management system provided by an embodiment of the present disclosure, including: queue 1 and data acquisition requests Q1, Q2, Q3 therein; queue 2 and data acquisition requests Q4, Q5, Q6, Q7 therein; and queue 3 and data acquisition requests Q8, Q9 therein. The request queue description information of queue N should be 0808:moc.Nelpmaxe.www / / :ptth.

[0115] Reference Figure 4, which is a second schematic diagram of a request queue in a request management system provided by an embodiment of the present disclosure:

[0116] When the request queue description information of the new data acquisition request Q0 is 0808:moc.3pmaxe.www / / :ptth, a request queue matching its request queue description information, ie, queue 3, is found in the existing request queues, and it is stored in the found queue 3.

[0117] Reference Figure 5 , which is a third schematic diagram of a request queue in a request management system provided by an embodiment of the present disclosure:

[0118] When the request queue description information of the new data acquisition request Q0 is 0808:moc.4pmaxe.www / / :ptth, and a request queue matching the request queue description information cannot be found in the existing request queues, a queue 4 matching the request queue description information is generated, and the data acquisition request is stored in the generated queue 4.

[0119] S203: In response to the queue acquisition instruction sent by the downloader, the request queue description information corresponding to each of the currently stored request queues is sent to the downloader, so that the downloader determines the target request queue description information from the request queue description information corresponding to each of the request queues based on the request queue description information corresponding to the data acquisition request in the current response; wherein the target request queue description information is different from the request queue description information corresponding to the data acquisition request in the downloader's current response.

[0120] Among them, the request queue description information sent to the downloader can be in various forms, including the request queue description information of the request queue and the number of requests therein. The request queue description information of all request queues can be sent, or only a part of it can be sent at a time, such as 5, 10, 20, etc.

[0121] Exemplarily, the request queue description information sending format may be:

[0122] Plain Text

[0123] {"moc.1elpmaxe.www":4,"moc.2elpmaxe.www":4,"moc.3elpmaxe.www":2}

[0124] Among them, moc.1elpmaxe.www is the request queue description information, and the number after the colon is the number of requests in the request queue corresponding to the request queue description information.

[0125] The downloader can execute data acquisition tasks from different sites at the same time, but cannot process data acquisition tasks from the same site at the same time. One site can use multiple domain names, and one domain name usually corresponds to only one site at the same time. In other words, different sites have different domain names, and the request queue description information containing the domain name information is also different. Therefore, the target request queue description information is different from the request queue description information corresponding to the data acquisition request in the downloader's current response.

[0126] S204: In response to the request acquisition instruction sent by the downloader, extract a target data acquisition request from the target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, and send the target data acquisition request to the downloader.

[0127] Among them, the method for determining the target data acquisition request can be: first determine the queue type of the target request queue; the queue type includes at least one of a first-in-first-out queue and a priority queue; and then determine the target data acquisition request from the target request queue based on the dequeue method corresponding to the queue type.

[0128] After the target data acquisition request is sent to the downloader, the number of requests in the request queue corresponding to the target data acquisition request is reduced by one. If the number of requests in the request queue is less than or equal to 0 after reduction by one, the request queue is deleted.

[0129] The above method can meet the needs of a single user to obtain data. On this basis, a proxy module can be added to the request management system to be applied to the scenario where multiple users initiate requests at the same time. The specific method includes:

[0130] The request queue parameters corresponding to the multiple users are obtained through the request proxy module; the request queue parameters include request queue description information of the multiple request queues corresponding to the users, and the queue length of the request queue of the user.

[0131] The request proxy module itself does not store actual requests, but can record the request queues it proxies by numbering.

[0132] Exemplarily, the information obtained by the request proxy module may be: “User 1: {0808:moc.1elpmaxe.www / / :ptth:3, 0808:moc.2elpmaxe.www / / :ptth:4, 0808:moc.3elpmaxe.www / / :ptth:1}”.

[0133] Based on the obtained request queue parameters, the request information of the request proxy module is updated; the request information includes multiple information groups, each information group includes the request queue description information corresponding to the information group, and the queue length of the request queue corresponding to the request queue description information.

[0134] For example, see Figure 6 The above is one of the schematic diagrams of another request management system provided by the embodiment of the present disclosure, including user 1, user 2 and user 3 and their corresponding request queues, and a request proxy module after obtaining the request queue parameters corresponding to the above users and updating the request information of the request proxy module. As can be seen from the figure, there are 4 request queues with different request queue description information of users 1-3, so there are 4 information groups in the request proxy module, and the information group corresponding to queue 1 includes: request queue description information 0808:moc.1elpmaxe.www / / :ptth, queue length 8. Taking queue 1 as an example, the total queue length is 8, including the queue length 3 of user 1, the queue length 2 of user 2 and the queue length 3 of user 1.

[0135] In an optional real-time manner, sending the request queue description information corresponding to the currently stored request queue to the downloader includes:

[0136] In the case where the downloader is the downloader corresponding to the request proxy module, the request queue description information corresponding to each currently stored information group is sent to the downloader.

[0137] The step of determining a target data acquisition request from a target request queue corresponding to the target request queue description information indicated by the request acquisition instruction comprises:

[0138] In the case that there are multiple target request queues, a target request queue with the highest queue length among the target request queues is determined.

[0139] For example, Figure 6 As shown, if the target request queue is queue 1, and there are request queues corresponding to three users in queue 1, the target request queue of user 1 or user 3 with the highest queue length is selected.

[0140] A target data acquisition request is determined from the target request queue with the highest queue length.

[0141] The request proxy module can also proxy other request proxy modules. The specific method is as follows:

[0142] Obtaining, through the request proxy module, request information of other request proxy modules corresponding to the request proxy module;

[0143] The updating of the request information of the request proxy module based on the acquired request queue parameter includes:

[0144] Based on the acquired request queue parameters and the request information of the other request proxy modules, the request information of the request proxy module is updated.

[0145] For example, see Figure 7 The above is the second schematic diagram of another request management system provided by the embodiment of the present disclosure, including user 1, user 2, user 3 and request proxy module 1 and their corresponding request queues, as well as obtaining the request queue parameters corresponding to the above users and request proxy modules, and updating the request information of the request proxy module.

[0146] There are 4 request queues with different request queue description information of users 1-3, so there are 4 information groups in the request proxy module, and the information group corresponding to queue 1 includes: request queue description information 0808:moc.1elpmaxe.www / / :ptth, queue length 8. Taking queue 1 as an example, the total queue length is 8, including the queue length 3 of user 1, the queue length 2 of user 2 and the queue length 3 of user 1.

[0147] Based on the above data acquisition system, the embodiment of the present disclosure also provides another data acquisition method, the execution subject of which is a downloader.

[0148] See also Figure 8 FIG. 8 is a flow chart of a data acquisition method provided by an embodiment of the present disclosure, wherein the method includes steps S801 to S804, wherein:

[0149] S801: Send a queue acquisition instruction to the request management system so that the request management system feeds back request queue description information corresponding to the currently stored request queue; the request queue stores data acquisition requests with the same request queue description information; the request queue description information includes domain name indication information.

[0150] S802: Based on the request queue description information corresponding to the currently executing data acquisition task, extract the target request queue description information from the request queue description information corresponding to the currently stored request queue; wherein the target request queue description information is different from the request queue description information corresponding to the currently executing data acquisition task.

[0151] Specifically, after obtaining the request queue description information corresponding to the request queue sent by the request management module, it is compared with the request queue description information corresponding to the data acquisition request in the downloader's current response to determine whether they are the same. Because the downloader can process data acquisition requests with different target request queue description information in parallel, the target request queue description information different from the data acquisition request in the downloader's current response is selected.

[0152] S803: Send a request acquisition instruction to the request management system, so that the request management system extracts a target data acquisition request from the target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, and feeds back the target data acquisition request.

[0153] S804: Execute the data acquisition task corresponding to the target data acquisition request.

[0154] Specifically, after obtaining the target data acquisition request, the downloader starts downloading the data corresponding to the target data acquisition request.

[0155] After adding the proxy module to the request management system, the specific method also includes:

[0156] Sending a queue acquisition instruction to the request proxy module of the request management system, so that the request management system feeds back request queue description information corresponding to each currently stored information group through the request proxy module; the information group is determined by request queue parameters corresponding to multiple users; the request queue parameters include request queue description information of multiple request queues corresponding to the user, and queue length of the request queue of the user;

[0157] The sending a request acquisition instruction to the request management system includes:

[0158] A request acquisition instruction is sent to the request agent module so that the request management system determines the target request queue with the highest queue length among the target request queues through the request agent module when there are multiple target request queues corresponding to the target request queue description information indicated by the request acquisition instruction, and extracts the target data acquisition request from the target request queue with the highest queue length.

[0159] The downloader executes the data acquisition task corresponding to the target data acquisition request.

[0160] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0161] Based on the same inventive concept, a data acquisition device corresponding to the data acquisition method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned data acquisition method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0162] Reference Fig. 9 FIG. 1 is a schematic diagram of a data acquisition device provided by an embodiment of the present disclosure, wherein the device comprises:

[0163] A generating module 901 is used to determine, in response to a data acquisition request, request queue description information based on a resource location identifier indicated by the data acquisition request; the request queue description information includes domain name indication information;

[0164] A storage module 902, configured to store the data acquisition request in a request queue that matches the request queue description information;

[0165] The first sending module 903 is used to send the request queue description information corresponding to each of the currently stored request queues to the downloader in response to the queue acquisition instruction sent by the downloader, so that the downloader extracts the target request queue description information from the request queue description information corresponding to each of the request queues based on the request queue description information corresponding to the data acquisition request currently responded to; wherein the target request queue description information is different from the request queue description information corresponding to the data acquisition request in the current response of the downloader;

[0166] The second sending module 904 is used to respond to the request acquisition instruction sent by the downloader, extract the target data acquisition request from the target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, and send the target data acquisition request to the downloader.

[0167] In an optional implementation manner, the generating module 901 determines the request queue description information based on the resource location identifier indicated by the data acquisition request; when the request queue description information includes domain name indication information, it is used to:

[0168] Extracting the domain name information, transmission protocol information and port information corresponding to the data acquisition request from the resource location identifier;

[0169] Determine request queue description information based on the domain name information, the transmission protocol information, and the port information.

[0170] In an optional implementation manner, when storing the data acquisition request in a request queue matching the request queue description information, the storage module 902 is configured to:

[0171] Querying a request queue that matches the request queue description information, and storing the data acquisition request in the queried request queue;

[0172] When a request queue matching the request queue description information cannot be found, a request queue matching the request queue description information is generated, and the data acquisition request is stored in the generated request queue.

[0173] In an optional implementation manner, when the second sending module 904 extracts the target data acquisition request from the target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, it is used to:

[0174] Determine a queue type of the target request queue; the queue type includes at least one of a first-in-first-out queue and a priority queue;

[0175] Based on the dequeue mode corresponding to the queue type, a target data acquisition request is determined from the target request queue.

[0176] In an optional implementation manner, the device further includes an agent module, configured to:

[0177] Obtaining request queue parameters corresponding to multiple users through a request proxy module; the request queue parameters include request queue description information of multiple request queues corresponding to the users, and queue lengths of the request queues of the users;

[0178] Based on the acquired request queue parameters, the request information of the request proxy module is updated; the request information includes a plurality of information groups, each of which includes the request queue description information corresponding to the information group, and the queue length of the request queue corresponding to the request queue description information;

[0179] The sending the request queue description information corresponding to each currently stored request queue to the downloader includes:

[0180] In the case where the downloader is the downloader corresponding to the request proxy module, sending the request queue description information corresponding to each currently stored information group to the downloader;

[0181] The step of extracting a target data acquisition request from a target request queue corresponding to the target request queue description information indicated by the request acquisition instruction comprises:

[0182] In the case that there are multiple target request queues, determining the target request queue with the highest queue length among the target request queues;

[0183] The target data acquisition request is extracted from the target request queue with the highest queue length.

[0184] In an optional implementation manner, the proxy module is further used to:

[0185] Obtaining, through the request proxy module, request information of other request proxy modules corresponding to the request proxy module;

[0186] The updating of the request information of the request proxy module based on the acquired request queue parameter includes:

[0187] Based on the acquired request queue parameters and the request information of the other request proxy modules, the request information of the request proxy module is updated.

[0188] Reference Fig.10 FIG. 1 is a schematic diagram of a data acquisition device provided by an embodiment of the present disclosure, wherein the device comprises:

[0189] The first instruction sending module 1001 is used to send a queue acquisition instruction to the request management system, so that the request management system feeds back request queue description information corresponding to the currently stored request queue; the request queue stores data acquisition requests with the same request queue description information; the request queue description information includes domain name indication information;

[0190] A determination module 1002 is used to extract target request queue description information from the request queue description information corresponding to the currently stored request queue based on the request queue description information corresponding to the currently executed data acquisition task; wherein the target request queue description information is different from the request queue description information corresponding to the currently executed data acquisition task;

[0191] The second instruction sending module 1003 is used to send a request acquisition instruction to the request management system, so that the request management system extracts a target data acquisition request from the target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, and feeds back the target data acquisition request;

[0192] The execution module 1004 is used to execute the data acquisition task corresponding to the target data acquisition request.

[0193] In an optional implementation manner, when the first instruction sending module 1001 sends the queue acquisition instruction to the request management system, it is used to:

[0194] Sending a queue acquisition instruction to the request proxy module of the request management system, so that the request management system feeds back request queue description information corresponding to each currently stored information group through the request proxy module; the information group is determined by request queue parameters corresponding to multiple users; the request queue parameters include request queue description information of multiple request queues corresponding to the user, and queue length of the request queue of the user;

[0195] When sending the request acquisition instruction to the request management system, the second instruction sending module 1003 is used to:

[0196] A request acquisition instruction is sent to the request agent module so that the request management system determines the target request queue with the highest queue length among the target request queues through the request agent module when there are multiple target request queues corresponding to the target request queue description information indicated by the request acquisition instruction, and extracts the target data acquisition request from the target request queue with the highest queue length.

[0197] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0198] Corresponds to Figure 1 The data acquisition method in the present disclosure also provides an electronic device 1100, such as Fig.11 FIG. 1 is a schematic diagram of the structure of an electronic device 1100 provided in an embodiment of the present disclosure, including:

[0199] Processor 111, memory 112, and bus 113; memory 112 is used to store execution instructions, including internal memory 1121 and external memory 1122; the internal memory 1121 is also called internal memory, which is used to temporarily store the operation data in the processor 111 and the data exchanged with the external memory 1122 such as a hard disk. The processor 111 exchanges data with the external memory 1122 through the internal memory 1121. When the electronic device 1100 is running, the processor 111 communicates with the memory 112 through the bus 113, so that the processor 111 executes the following instructions:

[0200] In response to the data acquisition request, determining request queue description information based on the resource location identifier indicated by the data acquisition request; the request queue description information includes domain name indication information;

[0201] storing the data acquisition request in a request queue that matches the request queue description information;

[0202] In response to the queue acquisition instruction sent by the downloader, the request queue description information corresponding to each of the currently stored request queues is sent to the downloader, so that the downloader extracts the target request queue description information from the request queue description information corresponding to each of the request queues based on the request queue description information corresponding to the data acquisition request currently responded to; wherein the target request queue description information is different from the request queue description information corresponding to the data acquisition request in the current response of the downloader;

[0203] In response to the request acquisition instruction sent by the downloader, a target data acquisition request is extracted from the target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, and the target data acquisition request is sent to the downloader.

[0204] In an optional implementation manner, in the instruction executed by the processor 111, the resource location identifier indicated by the data acquisition request is used to determine the request queue description information; the request queue description information includes domain name indication information, including:

[0205] Extracting the domain name information, transmission protocol information and port information corresponding to the data acquisition request from the resource location identifier;

[0206] Determine request queue description information based on the domain name information, the transmission protocol information, and the port information.

[0207] In an optional implementation manner, in the instruction executed by the processor 111, storing the data acquisition request in a request queue matching the request queue description information includes:

[0208] Querying a request queue that matches the request queue description information, and storing the data acquisition request in the queried request queue;

[0209] When a request queue matching the request queue description information cannot be found, a request queue matching the request queue description information is generated, and the data acquisition request is stored in the generated request queue.

[0210] In an optional implementation manner, in the instruction executed by the processor 111, extracting the target data acquisition request from the target request queue corresponding to the target request queue description information indicated by the request acquisition instruction includes:

[0211] Determine a queue type of the target request queue; the queue type includes at least one of a first-in-first-out queue and a priority queue;

[0212] Based on the dequeue mode corresponding to the queue type, a target data acquisition request is determined from the target request queue.

[0213] In an optional implementation manner, the processor 111 is further configured to execute:

[0214] Obtaining request queue parameters corresponding to multiple users through a request proxy module; the request queue parameters include request queue description information of multiple request queues corresponding to the users, and queue lengths of the request queues of the users;

[0215] Based on the acquired request queue parameters, the request information of the request proxy module is updated; the request information includes a plurality of information groups, each of which includes the request queue description information corresponding to the information group, and the queue length of the request queue corresponding to the request queue description information;

[0216] The sending the request queue description information corresponding to each currently stored request queue to the downloader includes:

[0217] In the case where the downloader is the downloader corresponding to the request proxy module, sending the request queue description information corresponding to each currently stored information group to the downloader;

[0218] The step of extracting a target data acquisition request from a target request queue corresponding to the target request queue description information indicated by the request acquisition instruction comprises:

[0219] In the case that there are multiple target request queues, determining the target request queue with the highest queue length among the target request queues;

[0220] The target data acquisition request is extracted from the target request queue with the highest queue length.

[0221] In an optional implementation manner, the processor 111 is further configured to execute:

[0222] Obtaining, through the request proxy module, request information of other request proxy modules corresponding to the request proxy module;

[0223] The updating of the request information of the request proxy module based on the acquired request queue parameter includes:

[0224] Based on the acquired request queue parameters and the request information of the other request proxy modules, the request information of the request proxy module is updated.

[0225] In an optional implementation manner, the processor 111 is further configured to execute:

[0226] Sending a queue acquisition instruction to a request management system so that the request management system feeds back request queue description information corresponding to a currently stored request queue; the request queue stores data acquisition requests with the same request queue description information; the request queue description information includes domain name indication information;

[0227] Based on the request queue description information corresponding to the currently executed data acquisition task, extracting the target request queue description information from the request queue description information corresponding to the currently stored request queue; wherein the target request queue description information is different from the request queue description information corresponding to the currently executed data acquisition task;

[0228] Sending a request acquisition instruction to the request management system, so that the request management system extracts a target data acquisition request from a target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, and feeds back the target data acquisition request;

[0229] Execute the data acquisition task corresponding to the target data acquisition request.

[0230] In an optional implementation manner, in the instruction executed by the processor 111, the sending of a queue acquisition instruction to the request management system includes:

[0231] Sending a queue acquisition instruction to the request proxy module of the request management system, so that the request management system feeds back request queue description information corresponding to each currently stored information group through the request proxy module; the information group is determined by request queue parameters corresponding to multiple users; the request queue parameters include request queue description information of multiple request queues corresponding to the user, and queue length of the request queue of the user;

[0232] The sending a request acquisition instruction to the request management system includes:

[0233] A request acquisition instruction is sent to the request agent module so that the request management system determines the target request queue with the highest queue length among the target request queues through the request agent module when there are multiple target request queues corresponding to the target request queue description information indicated by the request acquisition instruction, and extracts the target data acquisition request from the target request queue with the highest queue length.

[0234] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data acquisition method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0235] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the data acquisition method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0236] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0237] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0238] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0239] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0240] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0241] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A data acquisition method, characterized in that: include: In response to the data acquisition request, determining request queue description information based on a resource location identifier indicated by the data acquisition request; The request queue description information includes domain name indication information; storing the data acquisition request in a request queue matching the request queue description information, wherein different request queue description information corresponds to different request queues, and domain name indication information of different request queue description information is different; In response to the queue acquisition instruction sent by the downloader, the request queue description information corresponding to each of the currently stored request queues is sent to the downloader, so that the downloader extracts the target request queue description information from the request queue description information corresponding to each of the request queues based on the request queue description information corresponding to the data acquisition request currently responded to; wherein the target request queue description information is different from the request queue description information corresponding to the data acquisition request in the current response of the downloader; In response to the request acquisition instruction sent by the downloader, a target data acquisition request is extracted from the target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, and the target data acquisition request is sent to the downloader.

2. The method according to claim 1, characterized in that: The determining the request queue description information based on the resource location identifier indicated by the data acquisition request includes: Extracting the domain name information, transmission protocol information and port information corresponding to the data acquisition request from the resource location identifier; Determine request queue description information based on the domain name information, the transmission protocol information, and the port information.

3. The method according to claim 1, characterized in that The storing the data acquisition request in a request queue matching the request queue description information comprises: Querying a request queue that matches the request queue description information, and storing the data acquisition request in the queried request queue; When a request queue matching the request queue description information cannot be found, a request queue matching the request queue description information is generated, and the data acquisition request is stored in the generated request queue.

4. The method according to claim 1, characterized in that: The step of extracting a target data acquisition request from a target request queue corresponding to the target request queue description information indicated by the request acquisition instruction comprises: Determine a queue type of the target request queue; the queue type includes at least one of a first-in-first-out queue and a priority queue; Based on the dequeue mode corresponding to the queue type, a target data acquisition request is determined from the target request queue.

5. The method according to claim 1, characterized in that The method further comprises: Obtaining request queue parameters corresponding to multiple users through a request proxy module; the request queue parameters include request queue description information of multiple request queues corresponding to the users, and queue lengths of the request queues of the users; Based on the acquired request queue parameters, the request information of the request proxy module is updated; the request information includes a plurality of information groups, each of which includes the request queue description information corresponding to the information group, and the queue length of the request queue corresponding to the request queue description information; The sending the request queue description information corresponding to each of the currently stored request queues to the downloader includes: In the case where the downloader is the downloader corresponding to the request proxy module, sending the request queue description information corresponding to each currently stored information group to the downloader; The step of extracting a target data acquisition request from a target request queue corresponding to the target request queue description information indicated by the request acquisition instruction comprises: In the case that there are multiple target request queues, determining the target request queue with the highest queue length among the target request queues; A target data acquisition request is determined from the target request queue with the highest queue length.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining, through the request proxy module, request information of other request proxy modules corresponding to the request proxy module; The updating of the request information of the request proxy module based on the acquired request queue parameter includes: Based on the acquired request queue parameters and the request information of the other request proxy modules, the request information of the request proxy module is updated.

7. A data acquisition method, characterized in that: include: Sending a queue acquisition instruction to the request management system so that the request management system feeds back request queue description information corresponding to the currently stored request queue; The request queue stores data acquisition requests with the same request queue description information; the request queue description information includes domain name indication information, different request queue description information corresponds to different request queues, and the domain name indication information of different request queue description information is different; Based on the request queue description information corresponding to the currently executed data acquisition task, extracting the target request queue description information from the request queue description information corresponding to the currently stored request queue; wherein the target request queue description information is different from the request queue description information corresponding to the currently executed data acquisition task; Sending a request acquisition instruction to the request management system, so that the request management system determines a target data acquisition request from a target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, and feeds back the target data acquisition request; Execute the data acquisition task corresponding to the target data acquisition request.

8. The method according to claim 7, characterized in that The sending of a queue acquisition instruction to the request management system includes: Sending a queue acquisition instruction to the request proxy module of the request management system, so that the request management system feeds back request queue description information corresponding to each currently stored information group through the request proxy module; the information group is determined by request queue parameters corresponding to multiple users; the request queue parameters include request queue description information of multiple request queues corresponding to the user, and queue length of the request queue of the user; The sending a request acquisition instruction to the request management system includes: A request acquisition instruction is sent to the request agent module, so that the request management system determines the target request queue with the highest queue length among the target request queues through the request agent module when there are multiple target request queues corresponding to the target request queue description information indicated by the request acquisition instruction, and determines the target data acquisition request from the target request queue with the highest queue length.

9. A data acquisition device, characterized in that: include: A generating module, which determines the request queue description information based on the resource location identifier indicated by the data acquisition request, wherein the request queue description information includes the domain name indication information; A storage module, used for storing the data acquisition request in a request queue matching the request queue description information, wherein different request queue description information corresponds to different request queues, and domain name indication information of different request queue description information is different; A first sending module, configured to send the currently stored request queue description information corresponding to each of the request queues to the downloader in response to the queue acquisition instruction sent by the downloader, so that the downloader extracts the target request queue description information from the request queue description information corresponding to each of the request queues based on the request queue description information corresponding to the currently responded data acquisition request; wherein the target request queue description information is different from the request queue description information corresponding to the data acquisition request in the current response of the downloader; The second sending module is used to respond to the request acquisition instruction sent by the downloader, extract the target data acquisition request from the target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, and send the target data acquisition request to the downloader.

10. A data acquisition device, characterized in that: include: The first instruction sending module is used to send a queue acquisition instruction to the request management system, so that the request management system feeds back request queue description information corresponding to the currently stored request queue; the request queue stores data acquisition requests with the same request queue description information; the request queue description information includes domain name indication information, different request queue description information corresponds to different request queues, and the domain name indication information of different request queue description information is different; A determination module, configured to extract target request queue description information from the request queue description information corresponding to the currently stored request queue based on the request queue description information corresponding to the currently executed data acquisition task; wherein the target request queue description information is different from the request queue description information corresponding to the currently executed data acquisition task; A second instruction sending module is used to send a request acquisition instruction to the request management system, so that the request management system extracts a target data acquisition request from a target request queue corresponding to the target request queue description information indicated by the request acquisition instruction, and feeds back the target data acquisition request; An execution module is used to execute the data acquisition task corresponding to the target data acquisition request.

11. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the data acquisition method as described in any one of claims 1 to 8 are performed.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data acquisition method according to any one of claims 1 to 8 are executed.

Images