Data acquisition method and device for external interface and medium

By generating a table to be matched and issuing task records on the data processing node, the target key and task information are automatically sent to the distributed architecture, which solves the problem of low data acquisition efficiency and realizes efficient external data interface acquisition and standardized processing , improves the system's concurrent processing capabilities.

CN118984218APending Publication Date: 2024-11-19INSPUR ZHUOSHU BIG DATA IND DEV CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411051360.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-19

Smart Images

  • Figure CN118984218A_ABST
    Figure CN118984218A_ABST
Patent Text Reader

Abstract

The invention discloses a data collection method and device for an external interface and a medium, and the method comprises the steps: carrying out the filling of a to-be-matched table structure with interface access parameters when a data processing node needs to collect target data from the external interface, and generating a to-be-matched table of a collection task; in the task list, according to the to-be-matched table, generating a to-be-issued task record of the collection task; in the API key list, obtaining a plurality of key records of the external interface type, selecting a target key of which the key state is available in the plurality of key records, and updating the key state of the target key in the API key list to be in use; when the collection task meets the issuing condition, to-be-issued task information is extracted from the to-be-issued task record, and the target secret key and the to-be-issued task information are issued to a target collection node in the distributed architecture, so that the target collection node collects target data from an external interface. And the data acquisition efficiency for the external interface is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data acquisition method, device and medium for an external interface. Background Art

[0002] In the process of data governance, in order to achieve data standardization, it is necessary to obtain data through external data interfaces. However, there are some inconveniences in the existing data processing process. For example, the server of the data processing process is usually located in the intranet, which makes it difficult to communicate directly with the external data interface, or because a large number of network requests are sent by a single machine, network congestion or resource limitations occur, resulting in some requests being unable to successfully obtain data, resulting in low data collection efficiency for external interfaces. Summary of the invention

[0003] The embodiments of the present application provide a method, device and medium for collecting data on an external interface, which are used to solve the problem of low efficiency of collecting data on an external interface.

[0004] The present application embodiment adopts the following technical solutions:

[0005] On the one hand, an embodiment of the present application provides a data collection method for an external interface, the method comprising: when a data processing node needs to collect target data from an external interface, filling in interface access parameters in a to-be-matched table structure to generate a to-be-matched table for this collection task; the fields of the to-be-matched table structure include interface input parameters and return results; in a pre-constructed task list, a to-be-sent task record for this collection task is generated according to the to-be-matched table; the fields of the task list include task ID, data interface type, database connection information where the to-be-matched table is located, table name of the to-be-matched table, number of to-be-matched data rows in the to-be-matched table, task creation time, and task execution node name and task completion time; in a pre-constructed API key list, obtaining multiple key records of the external interface type, selecting a target key with an available key status from the multiple key records, and updating the key status of the target key in the API key list to in use; the fields of the API key list include key ID, data interface type, key and key status; when this acquisition task meets the sending conditions, extracting the task information to be sent from the task record to be sent, and sending the target key and the task information to be sent to the target acquisition node in the distributed architecture, so that the target acquisition node can collect the target data from the external interface.

[0006] In one example, in the pre-constructed task list, based on the table to be matched, a task record to be sent for this collection task is generated, specifically including: generating a universal unique identification code, and determining the universal unique identification code as the task ID of this collection task; filling the task ID, the interface type of the external interface, the database connection information where the table to be matched is located, the table name of the table to be matched, the number of rows of data to be matched in the table to be matched, and the current system time into the task list to generate a task record to be sent for this collection task.

[0007] In one example, the method further includes: traversing the task list within a fixed time interval; and determining that the collection task meets the conditions for issuance when it is found that the field contents of the task execution node name and the task completion time corresponding to the current collection task are empty.

[0008] In one example, sending the target key and the task information to be sent to the target collection node in the distributed architecture specifically includes: determining multiple collection nodes in the connected distributed architecture; determining the target collection node according to the preset load balancing, and in the task list, filling the device name of the target collection node into the task execution node name field of this collection task; sending the target key and the task information to be sent to the target collection node.

[0009] In one example, after the target key and the task information to be sent are sent to the target collection node in the distributed architecture, the method further includes: after the return result field of the table to be matched is filled with the target data, determining the task completion time of this collection task; in the task list, filling the task completion time into the task completion time field of this collection task.

[0010] In one example, after the target key and the task information to be sent are sent to the target collection node in the distributed architecture, the method also includes: obtaining the number of rows of data to be matched and the execution time of each historical collection task; the historical collection task has the same external interface type as the current collection task; according to the number of rows of data to be matched and the execution time, determining the task waiting time and the timeout time of the current collection task; within the task waiting time, querying whether the task completion time field is empty; if not, determining that the current collection task has been completed; if so, determining that the current collection task has timed out, and within the timeout time, querying whether the task completion time field is empty; if so, returning a task collection error.

[0011] In one example, determining the task waiting time and the timeout time of the current collection task based on the number of rows of data to be matched and the execution time specifically includes: determining the execution time of each row of each historical collection task based on the number of rows of data to be matched and the execution time of each historical collection task; calculating the average execution time of each row of each historical collection task to obtain the theoretical execution time of each row of the current collection task; multiplying the theoretical execution time of each row by the number of rows of data to be matched to obtain the task waiting time of the current collection task; and determining the timeout time of the current collection task based on the task waiting time.

[0012] In one example, after the target key and the task information to be sent are sent to the target collection node in the distributed architecture, the method further includes: when the quota of the target key is exhausted, receiving a key replacement request from the target collection node, and updating the key status of the target key in the API key list to quota exhausted; selecting other target keys with available key status from the multiple key records, sending the other target keys to the target collection node, and updating the key status of the other target keys in the API key list to in use.

[0013] On the other hand, an embodiment of the present application provides a data acquisition device for an external interface, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above-described data acquisition methods for an external interface.

[0014] On the other hand, an embodiment of the present application provides a non-volatile computer storage medium for data collection for an external interface, storing computer executable instructions, and the computer executable instructions can execute any of the above-mentioned methods for data collection for an external interface.

[0015] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0016] By pre-building the table structure of the to-be-matched table structure, task list and key list, when the data processing node needs to collect target data from the external interface, it can automatically generate the to-be-matched table for this collection task, insert the information of the to-be-matched table into the task list, and generate a record of the task to be sent, so that the target key and the task information to be sent can be automatically sent to the target collection node in the distributed architecture, thereby improving the flexibility and scalability of the system, adapting to different external interfaces and data collection requirements, and efficiently collecting and standardizing external data interfaces in the data governance process, improving data processing efficiency and accuracy, and realizing the execution of collection tasks under the distributed architecture, avoiding network congestion caused by a single server initiating a large number of interface requests at the same time, and improving the system's concurrent processing capabilities and task execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present application, some embodiments of the present application will be described in detail below in conjunction with the accompanying drawings, in which:

[0018] Figure 1 A flow chart of a data collection method for an external interface provided in an embodiment of the present application;

[0019] Figure 2 A structural schematic diagram of a data acquisition device for an external interface provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0021] Some embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0022] Figure 1 A flow chart of a data collection method for an external interface provided in an embodiment of the present application. The method can be applied to different business fields, such as Internet financial business field, e-commerce business field, instant messaging business field, game business field, official business field, etc. The process can be executed by a computing device in the corresponding field, and some input parameters or intermediate results in the process allow manual intervention and adjustment to help improve accuracy.

[0023] The analysis method involved in the embodiments of the present application can be implemented by a terminal device or a server, and the present application does not impose any special restrictions on this. For the convenience of understanding and description, the following embodiments are described in detail by taking a server as an example.

[0024] It should be noted that the server may be a single device or a system consisting of multiple devices, that is, a distributed server, and this application does not make any specific limitation on this.

[0025] Figure 1 The process in includes the following steps:

[0026] S101: When the data processing node needs to collect target data from an external interface, it fills the interface access parameters into the to-be-matched table structure to generate the to-be-matched table for this collection task; the to-be-matched table structure fields include the interface input parameters and the returned results.

[0027] It should be noted that the interface input parameters include the key fields corresponding to the target data, and the target acquisition node retrieves the target data in the external interface based on the interface input parameters. The content of the returned result field is the target data.

[0028] S102: In the pre-constructed task list, according to the table to be matched, generate the task record to be issued for this acquisition task; the fields of the task list include task ID, data interface type, database connection information where the table to be matched is located, table name of the table to be matched, number of rows of data to be matched in the table to be matched, task creation time, task execution node name and task completion time.

[0029] It should be noted that the task list is used to store task information for external interface data collection.

[0030] In some embodiments of the present application, it is necessary to pre-build a table structure of a task list, wherein the table structure of the task list is as follows:

[0031]

[0032]

[0033] Based on this, when generating the task record to be sent for this collection task according to the table to be matched, first, generate a universal unique identification code and determine the universal unique identification code as the task ID of this collection task. Finally, fill the task ID, the interface type of the external interface, the database connection information where the table to be matched is located, the table name of the table to be matched, the number of rows of data to be matched in the table to be matched, and the current system time into the task list to generate the task record to be sent for this collection task.

[0034] It should be noted that the current system time is the content of the task creation time field. In addition, in the task record of this collection task, the task execution node name and task completion time are empty at this time.

[0035] S103: In a pre-built API key list, obtain multiple key records of external interface type, select a target key with a key status of available from the multiple key records, and update the key status of the target key in the API key list to in use; the fields of the API key list include key ID, data interface type, key and key status.

[0036] It should be noted that the API key list is used to store API keys required for external interfaces.

[0037] In some embodiments of the present application, it is necessary to pre-build a table structure of an API key list, wherein the table structure of the API key list is as follows:

[0038]

[0039] That is, when the key status of the target key in the API key list is updated to in use, the key status is updated from 0 to 1.

[0040] S104: When the current collection task meets the sending condition, extract the task information to be sent from the task record to be sent, send the target key and the task information to the target collection node in the distributed architecture, so that the target collection node collects the target data to the external interface.

[0041] In some embodiments of the present application, the task distribution module in the data processing node will query the task list at a fixed time interval (e.g., 10 seconds) to find out whether there is a collection task, and only when the field contents of the task execution node name and the task completion time are both empty, it indicates that the collection task needs to be issued. In addition, the task list may have multiple collection tasks that need to be issued.

[0042] It should be noted that when the query shows that there is a collection task that meets the conditions for issuance, the task ID, data interface type, database connection information where the table to be matched is located, and the table name of the table to be matched in the collection task record are obtained to generate the task information to be issued.

[0043] Based on this, the task list is traversed within a fixed time interval. Then, when the field contents of the task execution node name and task completion time corresponding to this collection task are found to be empty, it is determined that this collection task meets the conditions for issuance.

[0044] In some embodiments of the present application, the collection node is a distributed architecture. Therefore, when issuing, it is necessary to determine the target collection node in the distributed architecture. The target collection node can be any personal computer or server that can communicate with the external interface and task distribution module.

[0045] Based on this, first determine the multiple collection nodes in the connected distributed architecture. Then, determine the target collection node based on the preset load balancing, and fill the device name of the target collection node into the task execution node name field of this collection task in the task list. Finally, send the target key and the task information to be sent to the target collection node.

[0046] In some embodiments of the present application, when the target collection node receives the target key and the task information to be sent, it selects a collection process that meets the type of external interface from the pre-developed data interface collection process, and based on the collection process, obtains the key fields of the interface input parameters in the table to be matched in batches, combines the target key, and splices the collection URL for collection.

[0047] It should be noted that different external interface types correspond to different collection processes, and the collection process is used to represent the rules for collecting data from the external interface.

[0048] Furthermore, the target key may have a quota exhaustion situation, for example, the remaining quota of the target key is less than the quota required by the current target collection node.

[0049] Based on this, when the quota of the target key is exhausted, a key replacement request is received from the target collection node, and the key status of the target key in the API key list is updated to quota exhausted. Then, other target keys with available key status are selected from multiple key records, and the other target keys are sent to the target collection node, and the key status of the other target keys in the API key list is updated to in use.

[0050] In some embodiments of the present application, the target collection node completes the collection, and fills the collected target data into the return result field of the table to be matched. Thus, the data processing node can obtain the target data by checking the table to be matched.

[0051] Based on this, after the return result field of the to-be-matched table is filled with the target data, the task completion time of this collection task is determined. In the task list, the task completion time is filled into the task completion time field of this collection task.

[0052] In some embodiments of the present application, in order to monitor the real-time working status of data collection and prevent the collection task from being stopped due to errors but the problem cannot be discovered in time, the task waiting time is set.

[0053] Based on this, first, obtain the number of rows of data to be matched and the execution time of each historical collection task. The historical collection task has the same external interface type as the current collection task. Then, according to the number of rows of data to be matched and the execution time, determine the task waiting time and timeout time of this collection task.

[0054] During the task waiting time, check whether the task completion time field is empty.

[0055] If not, it is determined that the current collection task has been completed. That is, when it is not empty, it means that the task has been completed.

[0056] If yes, it is determined that the collection task has timed out. Within the timeout period, check whether the task completion time field is empty. It should be noted that the timeout period starts from the deadline of the task waiting period. For example, if the task creation time is 1 o'clock and the task waiting period is 10 minutes, the timeout period is 5 minutes. Then the time interval from 1 o'clock to 1:10 is the task waiting time interval. In addition, the time interval from 1:10 to 1:15 is the time interval of the timeout.

[0057] If so, a task collection error is returned. That is, if the task completion time field is still empty within the timeout period, it means that the task is still not completed.

[0058] It should be noted that if the task completion time field is not empty within the timeout period, it is determined that the collection task has been completed.

[0059] The process of determining the task waiting time and timeout time of this collection task is as follows:

[0060] First, determine the execution time per line of each historical collection task based on the number of rows of data to be matched and the execution time of each historical collection task. Then, calculate the average execution time per line of each historical collection task to obtain the theoretical execution time per line of this collection task. Then, multiply the theoretical execution time per line by the number of rows of data to be matched to obtain the task waiting time of this collection task. Finally, determine the timeout duration of this collection task based on the task waiting time.

[0061] It should be noted that when determining the timeout duration, the task waiting duration can be retrieved in a pre-constructed mapping relationship table to obtain a timeout duration with a mapping relationship. The mapping relationship table includes different timeout durations corresponding to different task waiting duration intervals. Among them, since a longer task waiting duration indicates that the amount of task collection data of the collection task is relatively high, there is a higher possibility of network delay or low data transmission efficiency. Therefore, the mapping relationship change between the task waiting duration and the timeout duration can be set to be longer when the task waiting duration is longer, the timeout duration is longer.

[0062] It should be noted that although the embodiments of the present application are based on Figure 1 Steps S101 to S104 are described in sequence, but this does not mean that steps S101 to S104 must be performed in a strict order. Figure 1 The order shown in the figure is to introduce and explain step S101 to step S104 in sequence, in order to facilitate those skilled in the art to understand the technical solution of the embodiment of the present application. In other words, in the embodiment of the present application, the order between step S101 to step S104 can be appropriately adjusted according to actual needs.

[0063] pass Figure 1 The method can pre-build the table structure of the to-be-matched table, the task list and the key list. When the data processing node needs to collect target data from the external interface, the to-be-matched table of this collection task is automatically generated, the information of the to-be-matched table is inserted into the task list, and the to-be-sent task record is generated, so that the target key and the to-be-sent task information can be automatically sent to the target collection node in the distributed architecture, thereby improving the flexibility and scalability of the system, being able to adapt to different external interfaces and data collection requirements, and efficiently collecting and standardizing external data interfaces in the data governance process, improving data processing efficiency and accuracy, and realizing the execution of collection tasks under the distributed architecture, avoiding network congestion caused by a single server initiating a large number of interface requests at the same time, and improving the system's concurrent processing capability and task execution efficiency.

[0064] Based on the same idea, some embodiments of the present application also provide devices and non-volatile computer storage media corresponding to the above methods.

[0065] Figure 2 A structural diagram of a data acquisition device for an external interface provided in an embodiment of the present application includes:

[0066] at least one processor; and,

[0067] a memory communicatively connected to the at least one processor; wherein,

[0068] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above-mentioned methods for data collection for an external interface.

[0069] Some embodiments of the present application provide a non-volatile computer storage medium for data collection for an external interface, storing computer executable instructions, wherein the computer executable instructions can execute any of the above-mentioned methods for data collection for an external interface.

[0070] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0071] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0072] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0074] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0075] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0076] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0077] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0078] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0079] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0080] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the technical principle of the present application should all fall within the protection scope of the present application.

Claims

1. A data collection method for an external interface, characterized in that: The method comprises: When the data processing node needs to collect target data from an external interface, it fills the interface access parameters into the to-be-matched table structure to generate the to-be-matched table for this collection task; the fields of the to-be-matched table structure include the interface input parameters and the returned results; In the pre-built task list, according to the table to be matched, a task record to be issued for this acquisition task is generated; the fields of the task list include task ID, data interface type, database connection information where the table to be matched is located, table name of the table to be matched, number of rows of data to be matched in the table to be matched, task creation time, task execution node name and task completion time; In a pre-built API key list, multiple key records of external interface types are obtained, a target key whose key status is available is selected from the multiple key records, and the key status of the target key in the API key list is updated to in use; the fields of the API key list include key ID, data interface type, key and key status; When the current collection task meets the sending conditions, the task information to be sent is extracted from the task record to be sent, and the target key and the task information to be sent are sent to the target collection node in the distributed architecture, so that the target collection node collects the target data to the external interface.

2. The method according to claim 1, characterized in that In the pre-built task list, according to the to-be-matched table, a task record to be issued for this acquisition task is generated, specifically including: Generate a universal unique identification code, and determine the universal unique identification code as the task ID of this collection task; The task ID, the interface type of the external interface, the database connection information of the table to be matched, the table name of the table to be matched, the number of rows of data to be matched in the table to be matched, and the current system time are filled into the task list to generate a task record to be issued for this collection task.

3. The method according to claim 2, characterized in that The method further comprises: Traversing the task list at fixed time intervals; When it is found that the field contents of the task execution node name and task completion time corresponding to this collection task are empty, it is determined that this collection task meets the conditions for issuance.

4. The method according to claim 3, characterized in that The sending of the target key and the task information to be sent to the target collection node in the distributed architecture specifically includes: Identify multiple collection nodes in the connected distributed architecture; Determine the target acquisition node according to the preset load balancing, and fill the device name of the target acquisition node into the task execution node name field of this acquisition task in the task list; The target key and the task information to be sent are sent to the target acquisition node.

5. The method according to claim 1, characterized in that After sending the target key and the task information to be sent to the target acquisition node in the distributed architecture, the method further includes: After the return result field of the to-be-matched table is filled with the target data, determining the task completion time of this collection task; In the task list, the task completion time is filled into the task completion time field of this collection task.

6. The method according to claim 1, characterized in that After sending the target key and the task information to be sent to the target acquisition node in the distributed architecture, the method further includes: Obtain the number of rows of data to be matched and the execution time of each historical collection task; the historical collection task has the same external interface type as the current collection task; Determine the task waiting time and timeout time of this collection task according to the number of data rows to be matched and the execution time; Within the task waiting time, query whether the task completion time field is empty; If not, it is determined that this collection task has been completed; If yes, it is determined that the collection task has timed out, and within the timeout period, it is checked whether the task completion time field is empty; If yes, return task collection error.

7. The method according to claim 6, characterized in that Determining the task waiting time and timeout time of the current acquisition task according to the number of rows of data to be matched and the execution time specifically includes: Determine the execution time of each row of each historical collection task based on the number of rows of data to be matched and the execution time of each historical collection task; Calculate the average execution time of each row of each historical collection task to obtain the theoretical execution time of each row of this collection task; Multiply the theoretical execution time per row by the number of rows of data to be matched to obtain the task waiting time of this collection task; The timeout duration of this collection task is determined according to the task waiting duration.

8. The method according to claim 1, characterized in that After sending the target key and the task information to be sent to the target acquisition node in the distributed architecture, the method further includes: When the quota of the target key is exhausted, receiving a key replacement request from the target acquisition node, and updating the key state of the target key in the API key list to quota exhausted; Select other target keys whose key status is available from the multiple key records, send the other target keys to the target acquisition node, and update the key status of the other target keys in the API key list to in use.

9. A data acquisition device for an external interface, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data collection method for an external interface as described in any one of claims 1 to 8.

10. A non-volatile computer storage medium for data acquisition of an external interface, storing computer executable instructions, characterized in that: The computer executable instructions can execute a data collection method for an external interface as described in any one of claims 1 to 8.