Data collection method, device, electronic device, and storage medium

Abstract

The present application provides a data collection method, device, electronic device, and storage medium. The method includes: obtaining each data call process indicated by the task to be executed, and obtaining the downstream service interface information corresponding to each data call process; obtaining the initial call concurrency when the task to be executed starts to be executed, and executing each data call process according to the initial call concurrency; when the interface data fed back by the corresponding downstream service interface is received, adjusting the initial call concurrency according to the interface data, executing each data call process according to the adjusted call concurrency, and adjusting the adjusted call concurrency according to the interface data fed back by the downstream service interface until the target data is obtained and the task to be executed is completed. The method of the present application can solve the problem of how to make full use of downstream resources and improve the efficiency of collecting data from downstream services on the basis of avoiding the transformation of downstream services as much as possible.

Description

Technical Field

[0001] The present application relates to data collection technology for a batch system, and in particular to a data collection method, device, electronic device, and storage medium. Background Art

[0002] A batch processing system, or batch system, refers to a processing system that pulls and assembles data according to established logic within a certain timeframe, performs calculations, and ultimately draws batch conclusions. The server that hosts the batch processing system obtains the required data from internal interface services, third-party services, or the system's local database, and then assembles and calculates the data. Currently, a complete transaction chain often involves calls between internal services and multiple external downstream services. Some downstream services in the chain support different levels of concurrent processing capabilities. To prevent the entire chain from being affected by the crash of a downstream service, resulting in unavailability, the server needs to assess the load capacity of the downstream services.

[0003] Downstream services cyclically add tokens to the token bucket based on their own load. The server determines the downstream service's load capacity by retrieving tokens from the token bucket. Only after a successful retrieval can the downstream service be called. If the server fails to retrieve a token, the request fails. Subsequent requests will attempt to retrieve a token again. If the downstream service is still experiencing an abnormal or high load, subsequent requests will fail again.

[0004] This method of determining the load capacity of downstream services relies on the capabilities of the downstream services themselves. New downstream services require adaptation. Downstream service modifications are uncontrollable, and excessive modifications can be costly. Furthermore, the token bucket upper limit (current limiting threshold) is a preset value, but the load capacity of downstream services changes dynamically. Presetting the current limiting threshold in advance limits the circulation of tokens, resulting in inadequate utilization of downstream resources.

[0005] Therefore, how to make full use of downstream resources and improve the efficiency of collecting data from downstream services while avoiding the transformation of downstream services as much as possible still needs to be solved. Summary of the Invention

[0006] The present application provides a data collection method, device, electronic device, and storage medium to solve the problem of how to fully utilize downstream resources and improve the efficiency of collecting data from downstream services while avoiding the transformation of downstream services as much as possible.

[0007] In one aspect, the present application provides a data collection method, comprising:

[0008] Obtain each data call process indicated by the task to be executed, and obtain downstream service interface information corresponding to each data call process, wherein one data call process corresponds to information of at least one downstream service interface, and the task to be executed is used to instruct to obtain target data from the downstream service interface by executing each data call process;

[0009] Obtain the initial call concurrency when the task to be executed begins, and execute each data call process according to the initial call concurrency, wherein the initial call concurrency is used to represent the number of executions of each data call process. When the data call process is executed, a request message is sent to the corresponding downstream service interface, and the request message is used to request the corresponding downstream service interface to feedback interface data;

[0010] When the interface data fed back by the corresponding downstream service interface is received, the initial call concurrency is adjusted according to the interface data, each data call process is executed according to the adjusted call concurrency, and the adjusted call concurrency is adjusted again according to the interface data fed back by the downstream service interface until the target data is obtained and the task to be executed is completed.

[0011] In one embodiment, adjusting the initial call concurrency according to the interface data includes:

[0012] Obtain the timeliness weight of the tasks to be executed in the running state in the total tasks, obtain the proportion weight of the interface exception, and obtain the urgency weight of the unfinished data call process, wherein the proportion weight is less than or equal to the preset weight, and the urgency weight is greater than or equal to zero;

[0013] Acquire, according to the interface data, a first average time consumed to complete each data call process according to the initial call concurrency, and acquire, as a first number, a number of interface exceptions determined according to the interface data;

[0014] The initial concurrent call volume is adjusted according to the timeliness weight, the proportion weight, the urgency weight, the first average time consumption, and the first number to obtain the adjusted concurrent call volume.

[0015] In one embodiment, adjusting the adjusted call concurrency based on the interface data fed back by the downstream service interface includes:

[0016] Determine, based on the fed-back interface data, a second average time taken to complete each data call process after executing the adjusted call concurrency, and obtain, based on the fed-back interface data, a number of interface exceptions as a second number;

[0017] The adjusted call concurrency is further adjusted according to the timeliness weight, the proportion weight, the urgency weight, the first average time consumption, the second average time consumption, and the second number to obtain a readjusted call concurrency.

[0018] In one embodiment, each concurrent call volume also has a retention count, and adjusting the initial concurrent call volume according to the timeliness weight, the proportion weight, the urgency weight, the first average duration, and the first number to obtain the adjusted concurrent call volume includes:

[0019] Adjusting the initial concurrent call volume according to the timeliness weight, the proportion weight, the urgency weight, the first average duration, the first number, and the number of times the initial concurrent call volume is maintained to obtain the adjusted concurrent call volume;

[0020] The adjusting the adjusted concurrent call volume according to the timeliness weight, the proportion weight, the urgency weight, the first average time consumption, the second average time consumption, and the second number to obtain the adjusted concurrent call volume includes:

[0021] According to the timeliness weight, the proportion weight, the urgency weight, the first average time consumption, the second average time consumption, the second number and the number of times the adjusted call concurrency is maintained, the adjusted call concurrency is adjusted to obtain the readjusted call concurrency.

[0022] In one embodiment, adjusting the initial concurrent call volume according to the timeliness weight, the proportion weight, the urgency weight, the first average duration, the first number, and the number of times the initial concurrent call volume is maintained to obtain the adjusted concurrent call volume includes:

[0023] Determining a first adjustment ratio for a first positive adjustment according to the timeliness weight, the first average time consumption, and the initial concurrent call volume;

[0024] Determining a second adjustment ratio of the first positive adjustment according to the remaining execution time of the current task to be executed, the amount of target data to be acquired for the current task to be executed, and the urgency weight;

[0025] Determining the size of the first reverse adjustment concurrency according to the first number and the urgency weight;

[0026] The adjusted call concurrency is obtained according to the first adjustment ratio of the first forward adjustment, the second adjustment ratio of the first forward adjustment, and the size of the first reverse adjustment concurrency.

[0027] In one embodiment, adjusting the adjusted concurrent call volume according to the timeliness weight, the proportion weight, the urgency weight, the first average time consumption, the second average time consumption, the second number, and the number of times the adjusted concurrent call volume is maintained to obtain the adjusted concurrent call volume includes:

[0028] Determining a first adjustment ratio for a second positive adjustment based on the timeliness weight, the first average time consumption, the second average time consumption, the initial concurrent call volume, and the adjusted concurrent call volume;

[0029] determining a second adjustment ratio for the second positive adjustment according to the remaining execution time of the current task to be executed, the amount of target data to be acquired for the current task to be executed, and the urgency weight;

[0030] Determining the size of the second reverse adjustment concurrency according to the second number and the urgency weight;

[0031] The readjusted call concurrency is obtained according to the first adjustment ratio of the second forward adjustment, the second adjustment ratio of the second forward adjustment, and the size of the second reverse adjustment concurrency.

[0032] In one embodiment, adjusting the initial call concurrency according to the interface data includes:

[0033] When the number of consecutive uses of the initial concurrent call amount is greater than or equal to the number of times the initial concurrent call amount is maintained, adjusting the initial concurrent call amount according to the interface data;

[0034] The adjusting the adjusted call concurrency according to the interface data fed back by the downstream service interface includes:

[0035] When the number of consecutive uses of the adjusted call concurrency is greater than or equal to the number of retention times of the adjusted call concurrency, the adjusted call concurrency is readjusted according to the interface data fed back by the downstream service interface.

[0036] In one embodiment, obtaining the timeliness weight of the tasks to be executed in the running state in the total tasks includes:

[0037] Obtain a fourth quantity as the sum of a first quantity of pending tasks in a running state in the total tasks, a second quantity of unprocessed pending tasks in the total tasks, a priority of each pending task, a third quantity of uncompleted data call processes in the pending tasks in a running state, and the number of uncompleted data call processes in each pending task in the total tasks;

[0038] The timeliness weight of the tasks to be executed in the running state is determined according to the first number, the second number, the priority of each task to be executed, the third number, and the fourth number.

[0039] In one embodiment, the weight of obtaining the proportion of interface exceptions includes:

[0040] Determining a fifth number of downstream service interfaces with feedback exceptions according to the interface data;

[0041] Obtain the total number of downstream service interfaces, the sixth number of abnormal data call processes in the executed data call processes, and the seventh number of completed data call processes;

[0042] A weight of the proportion of interface abnormalities is determined according to the fifth number, the total number, the sixth number, and the seventh number.

[0043] In one embodiment, the urgency weight of obtaining the unfinished data call process includes:

[0044] Determine, according to the interface data, an eighth number of uncompleted data call processes and a total time consumption of completed data call processes as a first time consumption;

[0045] Get the ninth number of tasks that have been completed in the total tasks. The total time taken by the tasks that have been completed is the second time taken.

[0046] An urgency weight of an uncompleted data calling process is determined according to the sixth number, the eighth number, the ninth number, the first time consumption, and the second time consumption.

[0047] In one embodiment, after receiving the interface data fed back by the downstream service interface, the method further includes:

[0048] Obtain the request message received by the downstream service interface and the process identifier carried in the request message;

[0049] According to the process identifier carried by the request message, the request message is stored in the message list of the data call process corresponding to the process identifier;

[0050] Repeat the steps until the call concurrency obtained by the i-th adjustment is equal to zero, and then generate a detection message based on the request message stored in the message list of each data call process;

[0051] The detection message is sent to each downstream service interface, and the call concurrency is re-determined according to the interface data fed back by each downstream service interface, until the re-determined call concurrency is greater than zero, and each data call process is executed according to the re-determined call concurrency.

[0052] In another aspect, the present application provides a data collection device, comprising:

[0053] An acquisition module, configured to acquire each data call process indicated by a to-be-executed task, and to acquire information about a downstream service interface corresponding to each data call process, wherein one data call process corresponds to information about at least one downstream service interface, and the to-be-executed task is configured to instruct the acquisition of target data from the downstream service interface by executing each data call process;

[0054] The acquisition module is further used to obtain the initial call concurrency when the task to be executed starts, and execute each data call process according to the initial call concurrency, wherein the initial call concurrency is used to represent the number of executions of each data call process. When the data call process is executed, a request message is sent to the corresponding downstream service interface, and the request message is used to request the corresponding downstream service interface to feedback interface data;

[0055] A processing module is used to adjust the initial call concurrency according to the interface data fed back by the corresponding downstream service interface when receiving the interface data, execute each data call process according to the adjusted call concurrency, and adjust the adjusted call concurrency according to the interface data fed back by the downstream service interface again, until the target data is obtained and the task to be executed is completed.

[0056] On the other hand, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0057] The memory stores computer-executable instructions;

[0058] The processor executes the computer-executable instructions stored in the memory to implement the data collection method as described in the first aspect.

[0059] On the other hand, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the instructions are executed, the computer executes the data collection method as described in the first aspect.

[0060] On the other hand, the present application provides a computer program product, comprising a computer program, which implements the data collection method as described in the first aspect when executed by a processor.

[0061] The data collection method provided by the embodiment of the present application, when starting to execute the task to be executed, first obtains each data call process indicated by the task to be executed, then obtains the initial call concurrency, and executes each data call process according to the initial call concurrency. After obtaining the interface data fed back by the downstream service interface in real time, the call concurrency is adjusted according to the interface data, the purpose of which is to readjust the call concurrency of the upstream service in accordance with the actual feedback situation of the downstream service interface. By continuously adjusting the call concurrency after continuously obtaining the interface data, that is, dynamically adjusting the call concurrency, the call concurrency can be reduced when the downstream service is overloaded, and the call concurrency can be increased when the downstream service is idle, so as to achieve the purpose of making full use of the downstream service resources and improve the efficiency of collecting data from the downstream service. Moreover, the data collection method provided by the embodiment of the present application does not require the downstream services to be modified one by one, but only requires the upstream service to continuously adjust its own call concurrency, thereby reducing the modification cost required when using downstream resources.

[0062] In summary, the method provided by the embodiments of the present application can solve the problem of how to fully utilize downstream resources and improve the efficiency of collecting data from downstream services while avoiding the transformation of downstream services as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0064] Figure 1 A schematic diagram of an application scenario of the data collection method provided in this application;

[0065] Figure 2 A flowchart of a data collection method provided in one embodiment of the present application;

[0066] Figure 3 A flowchart of a method for adjusting the call concurrency based on interface data provided in one embodiment of the present application;

[0067] Figure 4 A graph showing the changing trend of concurrent calls provided for one embodiment of the present application;

[0068] Figure 5 A trend chart of concurrent calls provided in another embodiment of the present application;

[0069] Figure 6 A trend chart of concurrent calls provided in another embodiment of the present application;

[0070] Figure 7 A trend chart of concurrent calls provided in another embodiment of the present application;

[0071] Figure 8 A schematic diagram of a data collection device provided in accordance with an embodiment of the present application;

[0072] Figure 9 A schematic diagram of an electronic device provided for one embodiment of the present application.

[0073] The above drawings illustrate specific embodiments of the present disclosure, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the present disclosure in any way, but rather to illustrate the concepts of the present disclosure to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0074] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0075] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0076] First, let’s explain the terms involved in this application:

[0077] Batch processing systems, also known as batch processing systems, are systems that pull and assemble data and perform calculations to ultimately derive batch results within a specific timeframe, following established logic. During the data calculation process, the required data must be assembled. This data may be available in the system's local database, pulled in real time by internal interface services, or pulled from third-party services.

[0078] External service: A system service that is independent of the organization and provided by an external third party.

[0079] Current limiting component: A component that limits the processing throughput of the current system service by limiting the number of requests processed by the service to prevent the downstream service system from crashing due to receiving traffic that exceeds its own processing capacity.

[0080] Probe call: A timed task that periodically initiates a simulated call request signal to detect the health status of the service.

[0081] Token bucket: To control the number of concurrent calls between services, a token is issued for each service call. A call can only be initiated after obtaining a token. The component that stores and retains tokens is called a token bucket.

[0082] Currently, a complete transaction chain often involves calls between internal services and multiple external downstream services. Some downstream services in the chain may support different levels of concurrent processing capabilities. To prevent the entire chain from being affected by a crash of a downstream service, causing the entire chain to become unavailable, the server needs to determine the load capacity of the downstream services. Currently, the server determines the load capacity of downstream services by obtaining tokens.

[0083] Specifically, downstream services cyclically add tokens to the token bucket based on their own load. The server determines the downstream service's load capacity by retrieving tokens from the token bucket. Only after a successful retrieval can the downstream service be called. If the server fails to retrieve a token, the request fails. Subsequent requests will attempt to retrieve a token again. If the downstream service is still experiencing an abnormal or high load, subsequent requests will fail again.

[0084] This method of determining the load capacity of downstream services relies on the capabilities of the downstream services themselves. New downstream services require adaptation. Downstream service modifications are uncontrollable, and excessive modifications can be costly. Furthermore, the token bucket upper limit (current limiting threshold) is a preset value, but the load capacity of downstream services changes dynamically. Presetting the current limiting threshold in advance limits the circulation of tokens, resulting in inadequate utilization of downstream resources.

[0085] Based on this, the present application provides a data collection method, device, electronic device, and storage medium. The data collection method includes: obtaining each data call process indicated by the task to be executed, and obtaining the downstream service interface information corresponding to each data call process, wherein a data call process corresponds to the information of at least one downstream service interface, and the task to be executed is used to indicate that the target data is obtained from the downstream service interface by executing each data call process; obtaining the initial call concurrency when the task to be executed starts, and executing each data call process according to the initial call concurrency, wherein the initial call concurrency is used to characterize the number of executions of each data call process, when the data call process is executed, a request message is sent to the corresponding downstream service interface, and the request message is used to request the corresponding downstream service interface to feedback interface data; when the interface data fed back by the corresponding downstream service interface is received, the initial call concurrency is adjusted according to the interface data, each data call process is executed according to the adjusted call concurrency, and the adjusted call concurrency is adjusted according to the interface data fed back by the downstream service interface until the target data is obtained, and the task to be executed is completed.

[0086] After obtaining the interface data fed back by the downstream service interface in real time, the call concurrency is adjusted according to the interface data. The purpose is to readjust the call concurrency of the upstream service in accordance with the actual feedback of the downstream service interface. By continuously adjusting the call concurrency after continuously obtaining the interface data, that is, dynamically adjusting the call concurrency, the call concurrency can be reduced when the downstream service is overloaded, and the call concurrency can be increased when the downstream service is idle, thereby achieving the purpose of making full use of the downstream service resources and improving the efficiency of collecting data from the downstream service. Moreover, the data collection method provided by the embodiment of the present application does not require the downstream services to be modified one by one. It only requires the upstream service to continuously adjust its own call concurrency, which reduces the modification cost required when using downstream resources.

[0087] The data collection method provided in this application is applied to electronic devices, such as computers, servers with batch processing systems, etc. Figure 1 This is a schematic diagram of the application of the data collection method provided by this application. In the figure, when the electronic device receives the execution command of the batch task, it obtains each data call process indicated by the task to be executed, and obtains the downstream service interface information corresponding to each data call process. Figure 1 As shown, different data call processes correspond to different downstream service interfaces. Process 1 corresponds to the interface of downstream data source service A and the interface of downstream data source service B. Process 2 corresponds to the interface of downstream data source service A and the interface of downstream data source service N.

[0088] Then, the initial call concurrency at the start of the pending task is obtained, and each data call process is executed according to the initial call concurrency. When the interface data fed back by the corresponding downstream service interface is received, the initial call concurrency is adjusted according to the interface data, and each data call process is executed according to the adjusted call concurrency. The adjusted call concurrency is further adjusted according to the interface data fed back by the downstream service interface until the target data is obtained, and the pending task is completed.

[0089] See Figure 2 One embodiment of the present application provides a data collection method, comprising:

[0090] S210, obtain each data call process indicated by the task to be executed, and obtain the downstream service interface information corresponding to each data call process, wherein one data call process corresponds to the information of at least one downstream service interface, and the task to be executed is used to indicate that the target data is obtained from the downstream service interface by executing each data call process.

[0091] In the post-loan batch evaluation scenario of user credit rating in the financial field, it is necessary to obtain the user's credit report data, performance data, etc. from a third-party data source service regularly or irregularly for the purpose of verifying the user's credit performance. During the batch evaluation process, the third-party data required for each decision dimension used to evaluate credit performance is different, and each batch evaluation scenario has different requirements for batch evaluation timeliness. For example, some batch evaluation scenarios are emergency batch evaluations, some are regular batch evaluations, and some are bypass batch evaluations. During the batch evaluation process, the process of obtaining the third-party data required for each decision dimension needs to be listed as at least one separate task to be executed. For example, the task to be executed obtains the user's credit report data and performance data from two third-party data source services respectively.

[0092] The pending task may indicate multiple data call processes, each of which corresponds to multiple downstream service interfaces. For example, in the user credit assessment process, data needs to be obtained from the interfaces of downstream data source service A, downstream data source service B, and downstream data source service C. In other words, interfaces A, B, and C need to be called simultaneously to obtain data. In the user pricing assessment process, data needs to be obtained from the interfaces of downstream data source service B, downstream data source service C, and downstream data source service D. In other words, interfaces B, C, and D need to be called simultaneously to obtain data.

[0093] When executing the task to be executed, first obtain the downstream service interface information corresponding to each data call process indicated by the task to be executed to determine the downstream service interface that each data call process needs to interact with when making a data call.

[0094] This pending task is used to instruct the execution of each data call process to obtain target data from the downstream service interface. This target data can be, for example, the user's credit report data and contract performance data described above. Due to the limited load capacity of the downstream data source service, when obtaining the target data, it is necessary to obtain it from the downstream data source service in batches and multiple times. For example, if 10,000 data items are to be obtained directly at once, only 10 data items can be obtained from the downstream data source service at a time.

[0095] S220, obtain the initial call concurrency when the task to be executed starts, and execute each data call process according to the initial call concurrency, wherein the initial call concurrency is used to characterize the number of executions of each data call process. When the data call process is executed, a request message is sent to the corresponding downstream service interface, and the request message is used to request the corresponding downstream service interface to feedback interface data.

[0096] The initial call concurrency indicates the number of times each data call process is executed. For example, if there are three data call processes and the initial call concurrency is 10, it means that the three data call processes are called simultaneously, and each data call process is executed 10 times. Preferably, the initial call concurrency is 1, meaning that one downstream call request is initially initiated.

[0097] When the data call process is executed, a request message is sent to the corresponding downstream service interface. This request message is used to request the corresponding downstream service interface to feedback interface data. The feedback interface data includes the request message received by the interface, the interface time consumption, and the partial target data returned.

[0098] The electronic device is equipped with a dynamic throttling component that connects to the interface of a downstream third-party data source service via a throttling interface. This dynamic throttling component is used to adjust the call concurrency. When a downstream service call is initiated, the dynamic throttling component is first invoked to execute each data call process at the initial call concurrency. Upon receiving interface data fed back by the downstream service interface, the dynamic throttling component determines the new call concurrency based on a pre-configured calculation method.

[0099] S230, when receiving the interface data fed back by the corresponding downstream service interface, adjust the initial call concurrency according to the interface data, execute each data call process according to the adjusted call concurrency, and adjust the adjusted call concurrency according to the interface data fed back by the downstream service interface again, until the target data is obtained, and the task to be executed is completed.

[0100] As described above, due to the limited load capacity of downstream data source services, acquiring target data requires multiple batches from the downstream data source service. For example, if 10,000 pieces of data are needed, they cannot be obtained all at once; only 10 pieces of data can be retrieved from the downstream data source service at a time. Therefore, the fundamental goal of dynamically adjusting the call concurrency based on the load capacity of the downstream data source service is to maximize downstream resource utilization and obtain the target data with the highest efficiency and speed. Once all the target data is obtained, the pending task is completed.

[0101] Data from third-party downstream data sources varies in quality and reliability, and different types of third-party data can have varying impacts on user credit performance assessments. Therefore, when adjusting call concurrency based on data fed back from third-party data source interfaces, a weighting factor should be introduced to comprehensively account for these various factors.

[0102] In an optional embodiment, when adjusting the call concurrency, it is necessary to introduce the timeliness weight of the tasks to be executed in the running state in the total tasks (that is, the tasks that have started to execute the data call process), the proportion weight of interface exceptions, and the urgency weight of the data call process that has not been completed.

[0103] The timeliness weight (a%) reflects the degree of timeliness required for data processing in this batch task. Data processing can be classified into urgent, routine, and bypass types. The timeliness weight (a%) can be determined by combining statistics on the number of pending tasks in the running state in the total task (the entire batch task), the number of unprocessed pending tasks in the total tasks, the priority of each pending task, the number of incomplete data call processes in the running pending tasks, and the number of incomplete data call processes in each pending task in the total tasks.

[0104] Specifically, the first number (RN) of pending tasks in the running state in the total tasks, the second number (WN) of unprocessed pending tasks in the total tasks, the priority (P) of each pending task, the third number (WC) of uncompleted data call processes in the pending tasks in the running state, and the sum of the number of uncompleted data call processes in each pending task in the total tasks is obtained as a fourth number (SUM).

[0105] The timeliness weight of the tasks to be executed in the running state is determined according to the first number, the second number, the priority of each task to be executed, the third number and the fourth number. That is, according to the formula Determine the timeliness weight. In the formula, RNi represents the number of tasks in the i-th category currently being executed, and Pi represents the priority of the i-th category. RNn represents the number of tasks in the n-th category currently being executed (in the running state), WNn represents the number of tasks in the n-th category pending execution, and Pn represents the priority of the n-th category. It should be noted that different types of tasks may be executing or waiting to be executed at the same time, and there may be at least 0 tasks of each type waiting to be processed.

[0106] The proportion weight (b%) reflects the tolerance of batch data to abnormal data in the batch process, that is, the impact of downstream data anomalies on the overall final decision conclusion. Specifically, the fifth number of downstream service interfaces with abnormal feedback is determined based on the interface data. The total number of downstream service interfaces, the sixth number of abnormal data call processes in the executed data call processes, and the seventh number of completed data call processes are obtained. Among them, the fifth number represents the number of downstream service interfaces with abnormal feedback interface data. The sixth number represents the number of data call processes that cannot be executed normally in the data call processes that have started to be executed. For example, when some data call processes are executed and the interface data cannot be fed back, these data call processes are determined to be abnormal data call processes.

[0107] The weight of the proportion of interface abnormality is determined based on the fifth number, the total number, the sixth number, and the seventh number. That is, according to the formula Determine a weight b% of the proportion of interface abnormalities. It should be noted that the weight is less than or equal to a preset weight, which is equal to 1.

[0108] The urgency weight (c%) reflects the urgency of the remaining data to be processed, and the urgency weight is greater than or equal to zero. Specifically, based on the interface data, the eighth number of uncompleted data call processes (taskWaitCount) is determined, and the total time consumed by the completed data call processes is the first time consumption (taskRunCost). The ninth number of completed tasks (lastCount) among the total tasks is then obtained, and the total time consumed by the completed tasks is the second time consumption (lastCost).

[0109] The urgency weight of the unfinished data call process is determined based on the sixth number, the eighth number, the ninth number, the first time consumption, and the second time consumption. Specifically, according to the formula Determine the urgency weight of the data call process that has not been completed.

[0110] After obtaining the timeliness weights of the tasks to be executed in the running state in the total tasks, obtaining the weight of the proportion of interface exceptions, and obtaining the urgency weights of the data call processes that have not been completed. When adjusting the initial call concurrency according to the interface data, specifically, obtaining the first average time required to complete each data call process according to the initial call concurrency according to the interface data, and obtaining the number of interface exceptions determined according to the interface data as a first number. Finally, adjust the initial call concurrency according to the timeliness weight, the proportion weight, the urgency weight, the first average time required, and the first number to obtain the adjusted call concurrency.

[0111] In an optional embodiment, each call concurrency also has a retention number. The initial call concurrency is adjusted based on the timeliness weight, the proportion weight, the urgency weight, the first average time, the first number and the retention number of the initial call concurrency to obtain the adjusted call concurrency.

[0112] A first adjustment ratio for the first forward adjustment is determined based on the timeliness weight, the first average duration, and the initial concurrent call volume. A second adjustment ratio for the first forward adjustment is determined based on the remaining execution time of the currently pending task, the amount of target data to be acquired for the currently pending task, and the urgency weight. A size of the concurrent call volume for the first reverse adjustment is determined based on the first number and the urgency weight. The adjusted concurrent call volume is obtained based on the first adjustment ratio for the first forward adjustment, the second adjustment ratio for the first forward adjustment, and the size of the concurrent call volume for the first reverse adjustment.

[0113] Specifically, according to the formula Adjust the initial call concurrency to obtain the adjusted call concurrency.

[0114] The first adjustment ratio of the first forward adjustment is The second adjustment ratio of the first positive adjustment is The size of the first reverse adjustment concurrency is N1×b%.

[0115] In the formula, L2 represents the adjusted call concurrency, L1 represents the initial call concurrency, T1 represents the first average time, t represents the remaining execution time of the current task to be executed, and M 待 represents the number of target data to be acquired for the current task to be executed, N1 represents the first number, a% represents the timeliness weight, b% represents the proportion weight, and c% represents the urgency weight.

[0116] When adjusting the adjusted call concurrency based on the interface data fed back by the downstream service interface, specifically, determining a second average time required to complete each data call process using the adjusted call concurrency based on the fed back interface data, and obtaining a second number of interface exceptions determined based on the fed back interface data. Adjusting the adjusted call concurrency based on the timeliness weight, the proportion weight, the urgency weight, the first average time required, the second average time required, and the second number to obtain a readjusted call concurrency.

[0117] Furthermore, based on the timeliness weight, the proportion weight, the urgency weight, the first average time consumption, the second average time consumption, the second number and the number of times the adjusted call concurrency is maintained, the adjusted call concurrency is adjusted to obtain the readjusted call concurrency.

[0118] A first adjustment ratio for the second forward adjustment is determined based on the timeliness weight, the first average time consumption, the second average time consumption, the initial call concurrency, and the adjusted call concurrency. A second adjustment ratio for the second forward adjustment is determined based on the remaining execution time of the current task to be executed, the amount of target data to be acquired for the current task to be executed, and the urgency weight. A size of the second reverse adjustment concurrency is determined based on the second number and the urgency weight. The readjusted call concurrency is obtained based on the first adjustment ratio for the second forward adjustment, the second adjustment ratio for the second forward adjustment, and the size of the second reverse adjustment concurrency.

[0119] Specifically, according to the formula The adjusted call concurrency is then adjusted to obtain a readjusted call concurrency.

[0120] The first adjustment ratio of the second positive adjustment is The second adjustment ratio of the second forward adjustment is The second reverse adjustment of the concurrency size is N i ×b%, where i=2.

[0121] Where i represents the number of times the concurrent call is adjusted. When i=2, L i+1 Represents the adjusted call concurrency, L i Represents the adjusted call concurrency, L i -1 represents the initial call concurrency, T i Represents the second average time, T i-1 Represents the first average time, N i represents the second number. t represents the remaining execution time of the current task to be executed, M 待represents the number of target data to be acquired for the current task to be executed, N1 represents the first number, a% represents the timeliness weight, b% represents the proportion weight, and c% represents the urgency weight.

[0122] When the call concurrency is finally dynamically determined based on two positive adjustment ratios (i.e., the ratio of increasing concurrency) and one negative adjustment amount (i.e., the need to reduce concurrency), the two positive adjustment ratios avoid the errors caused by a single dimension, and subtract the value of the negative adjustment amount to further eliminate the interference of the negative factor on the calculation result, thereby improving the accuracy of the concurrency calculation.

[0123] So far, through the above description, you can know how to adjust the call concurrency based on the interface data. Figure 3 This is a rough flow chart of the method described above on how to adjust the call concurrency based on the interface data. That is, when calling the batch service, after obtaining the interface data, the call concurrency is dynamically determined based on the timeliness weight, proportion weight, and urgency weight, and the request message is sent to the third-party data source service based on the determined call concurrency. Figure 3 As shown in the figure, when calling the batch service, interface information such as the number of abnormal interfaces and interface time consumption will also be counted to dynamically calculate the timeliness weight and proportion weight.

[0124] As described above, each call concurrency has a retention number, and the retention number represents the continuous use number of call concurrency. In an optional embodiment, the premise for adjusting the call concurrency is that the continuous use number of call concurrency is greater than or equal to its retention number. For example, when the continuous use number of the initial call concurrency is greater than or equal to the retention number of the initial call concurrency, the initial call concurrency can be adjusted according to the interface data. When the continuous use number of the adjusted call concurrency is greater than or equal to the retention number of the adjusted call concurrency, the adjusted call concurrency is adjusted according to the interface data fed back by the downstream service interface. For example, when the retention number of the initial call concurrency is 2, after the initial call concurrency is used 2 times continuously (i.e., each data call process is executed according to the initial call concurrency twice in a row), the adjusted call concurrency is adjusted according to the interface data fed back by the downstream service interface. When each data call process is executed next time, each data call process is executed according to the adjusted call concurrency.

[0125] In an optional embodiment, the dynamic current limiting component is also connected to a distributed cache (Redis) instance, and the data collection of the dynamic current limiting component is connected to Redis updates, and the query interface of the dynamic current limiting component is connected to an external query interface. When initiating a call to the data call process, the dynamic current limiting component will also collect and cache the request message information of each interface according to the process dimension during the batch run process. When any interface has a message change or an interface is deleted in the process, the message can be updated in time during the current limiting detection.

[0126] Specifically, after receiving the interface data fed back by the downstream service interface, the request message received by the downstream service interface and the process identifier carried in the request message are obtained. Based on the process identifier carried in the request message, the request message is stored in the message list of the data call process corresponding to the process identifier. The message request sent to the downstream service interface carries the process identifier, and the request message in the interface data fed back by the downstream service interface also carries the process identifier.

[0127] When the call concurrency obtained by the i-th adjustment is equal to zero, it proves that a serious anomaly has occurred in the process call and the downstream service interface may be faulty or overloaded. At this time, a detection message is generated based on the request message stored in the message list of each data call process. The detection message is sent to each downstream service interface, and the call concurrency is re-determined based on the interface data fed back by each downstream service interface. If the re-determined call concurrency is still equal to zero, then another detection is performed. That is, the steps are repeatedly performed. When the call concurrency obtained by the i-th adjustment is equal to zero, a detection message is generated based on the request message stored in the message list of each data call process, and the detection message is sent to each downstream service interface. The call concurrency is re-determined based on the interface data fed back by each downstream service interface. Until the re-determined call concurrency is greater than zero, it is determined that the downstream service interface can carry the data call. At this time, each data call process is executed according to the re-determined call concurrency.

[0128] The following combination Figures 4 to 7 Describes changes in the number of concurrent calls.

[0129] Figure 4 The figure shows the growth of call concurrency when the call concurrency is adjusted based on the initial call concurrency. The call concurrency increases in a step-by-step manner.

[0130] Then it enters the regular batch running phase. In this phase, the third-party downstream data source service is stable, the service time tends to be stable, and the processing logic, data collection and dynamic decision-making of the dynamic current limiting component are consistent with the initial phase. Its main feature is that the call concurrency determined by the dynamic current limiting component is constantly changing, but tends to be dynamically balanced. The change trend of the call concurrency in the regular batch running phase is as follows: Figure 5 shown.

[0131] When the third-party downstream data source service is abnormal, the processing logic of the dynamic current limiting component is consistent with the initial stage. Its main feature is that the concurrency determined depends on the weight ratio of the number of abnormalities, the concurrency drops sharply, and then changes dynamically. When the third-party downstream data source service is abnormal, the change trend of the call concurrency is as follows: Figure 6 As shown. At this time, when the call concurrency becomes 0, the detection process is entered, that is, the steps are repeatedly executed. When the call concurrency obtained by the i-th adjustment is equal to zero, a detection message is generated according to the request message stored in the message list of each data call process, and the detection message is sent to each downstream service interface. The call concurrency is re-determined according to the interface data fed back by each downstream service interface. After detecting that the downstream service is normal, the recovery process is entered, that is, until the re-determined call concurrency is greater than zero, it is determined that the downstream service interface can carry the data call. At this time, each data call process is executed according to the re-determined call concurrency.

[0132] Specifically, the request message is extracted from the dynamic throttling component's Redis cache using the process identifier to generate the probe message. If the interface call orchestration required by subsequent processes changes—that is, if the downstream service interface corresponding to the data call process changes—the message content of the data call process is automatically updated during the initial call, thereby automatically maintaining the accuracy of the probe message.

[0133] When the adjusted call concurrency is 0, the dynamic current limiting component switches to heartbeat detection mode. At this time, the dynamic current limiting component automatically obtains the request message with the process ID (Identity document) as the identifier and initiates downstream service detection. For the return content of the detection mode call (that is, the interface data returned by the interface), the conclusion obtained based on the return content is not retained as a formal result. The dynamic current limiting component will only collect the returned interface data and calculate the call concurrency based on the returned interface data. After the dynamic current limiting component receives the result of the detection service, it automatically calculates the formula to determine whether to enter the recovery process. That is, when the calculated call concurrency gradually increases, it enters the recovery process.

[0134] When the third-party downstream data source service gradually recovers from the outage state, the processing logic of the dynamic current limiting component is consistent with the initial stage. Its main feature is that the call concurrency gradually recovers from 0 and increases in steps. The change trend of the call concurrency at this time is as follows: Figure 7 shown.

[0135] In summary Figures 4 to 7The call concurrency is dynamically adjusted based on the status of the third-party downstream data source service. When the third-party downstream data source service is operating abnormally, the call concurrency is continuously reduced to prevent overload and downtime. When the third-party downstream data source service is operating normally, the call concurrency increases in steps to fully utilize the resources of the third-party downstream data source service. Furthermore, when a fault occurs, the status of the third-party downstream data source service is automatically detected by generating a detection message, improving the efficiency of detecting the health status of the third-party downstream data source service and reducing errors.

[0136] The data collection method provided in this embodiment includes: obtaining each data call process indicated by the task to be executed, and obtaining the downstream service interface information corresponding to each data call process, wherein one data call process corresponds to the information of at least one downstream service interface, and the task to be executed is used to indicate that the target data is obtained from the downstream service interface by executing each data call process; obtaining the initial call concurrency when the task to be executed starts, and executing each data call process according to the initial call concurrency, wherein the initial call concurrency is used to characterize the number of executions of each data call process, when the data call process is executed, a request message is sent to the corresponding downstream service interface, and the request message is used to request the corresponding downstream service interface to feedback interface data; when the interface data fed back by the corresponding downstream service interface is received, the initial call concurrency is adjusted according to the interface data, each data call process is executed according to the adjusted call concurrency, and the adjusted call concurrency is adjusted according to the interface data fed back by the downstream service interface, until the target data is obtained, and the task to be executed is completed.

[0137] That is, when starting to execute the task to be executed, first obtain each data call process indicated by the task to be executed, then obtain the initial call concurrency, and execute each data call process according to the initial call concurrency. After obtaining the interface data fed back by the downstream service interface in real time, adjust the call concurrency according to the interface data. The purpose is to readjust the call concurrency of the upstream service in accordance with the actual feedback of the downstream service interface. By continuously adjusting the call concurrency after continuously obtaining the interface data, that is, dynamically adjusting the call concurrency, the call concurrency can be reduced when the downstream service is overloaded, and the call concurrency can be increased when the downstream service is idle, so as to achieve the purpose of making full use of the downstream service resources and improve the efficiency of collecting data from the downstream service. Moreover, the data collection method provided by the embodiment of the present application does not require the downstream services to be modified one by one. It only requires the upstream service to continuously adjust its own call concurrency, which reduces the modification cost required when using downstream resources.

[0138] In summary, the method provided by the embodiments of the present application can solve the problem of how to fully utilize downstream resources and improve the efficiency of collecting data from downstream services while avoiding the transformation of downstream services as much as possible.

[0139] See Figure 8 One embodiment of the present application further provides a data collection device 20, comprising:

[0140] The acquisition module 11 is used to obtain each data call process indicated by the task to be executed, and obtain the downstream service interface information corresponding to each data call process, wherein one data call process corresponds to the information of at least one downstream service interface, and the task to be executed is used to indicate that the target data is obtained from the downstream service interface by executing each data call process.

[0141] The acquisition module 11 is also used to obtain the initial call concurrency when the task to be executed starts, and execute each data call process according to the initial call concurrency, wherein the initial call concurrency is used to characterize the number of executions of each data call process. When the data call process is executed, a request message is sent to the corresponding downstream service interface, and the request message is used to request the corresponding downstream service interface to feedback interface data.

[0142] The processing module 12 is used to adjust the initial call concurrency according to the interface data fed back by the corresponding downstream service interface when receiving the interface data, execute each data call process according to the adjusted call concurrency, and adjust the adjusted call concurrency according to the interface data fed back by the downstream service interface again, until the target data is obtained and the task to be executed is completed.

[0143] The processing module 12 is specifically used to obtain the timeliness weight of the tasks to be executed in the running state in the total tasks, and to obtain the proportion weight of the interface exceptions, and to obtain the urgency weight of the data call process that has not been completed, wherein the proportion weight is less than or equal to the preset weight, and the urgency weight is greater than or equal to zero; obtain the first average time taken to complete each data call process according to the initial call concurrency based on the interface data, and obtain the number of interface exceptions determined according to the interface data as a first number; adjust the initial call concurrency according to the timeliness weight, the proportion weight, the urgency weight, the first average time taken and the first number to obtain the adjusted call concurrency.

[0144] The processing module 12 is specifically used to determine the second average time required to complete each data call process by the adjusted call concurrency based on the re-feedback interface data, and to obtain the number of interface exceptions determined based on the re-feedback interface data as a second number; and to adjust the adjusted call concurrency based on the timeliness weight, the proportion weight, the urgency weight, the first average time required, the second average time required and the second number to obtain the re-adjusted call concurrency.

[0145] Each call concurrency also has a retention count, and the processing module 12 is specifically used to adjust the initial call concurrency based on the timeliness weight, the proportion weight, the urgency weight, the first average time, the first number, and the retention count of the initial call concurrency to obtain the adjusted call concurrency. The processing module 12 is specifically used to adjust the adjusted call concurrency based on the timeliness weight, the proportion weight, the urgency weight, the first average time, the second average time, the second number, and the retention count of the adjusted call concurrency to obtain the readjusted call concurrency.

[0146] The processing module 12 is specifically used to determine the first adjustment ratio of the first forward adjustment based on the timeliness weight, the first average time and the initial call concurrency; determine the second adjustment ratio of the first forward adjustment based on the remaining execution time of the current task to be executed, the amount of target data to be obtained for the current task to be executed and the urgency weight; determine the size of the first reverse adjustment concurrency based on the first number and the urgency weight; and obtain the adjusted call concurrency based on the first adjustment ratio of the first forward adjustment, the second adjustment ratio of the first forward adjustment and the size of the first reverse adjustment concurrency.

[0147] The processing module 12 is specifically used to determine the first adjustment ratio of the second forward adjustment based on the timeliness weight, the first average time consumption, the second average time consumption, the initial call concurrency and the adjusted call concurrency; determine the second adjustment ratio of the second forward adjustment based on the remaining execution time of the current task to be executed, the amount of target data to be obtained for the current task to be executed and the urgency weight; determine the size of the second reverse adjustment concurrency based on the second number and the urgency weight; and obtain the readjusted call concurrency based on the first adjustment ratio of the second forward adjustment, the second adjustment ratio of the second forward adjustment and the size of the second reverse adjustment concurrency.

[0148] The processing module 12 is specifically used to adjust the initial call concurrency according to the interface data when the number of consecutive uses of the initial call concurrency is greater than or equal to the number of times the initial call concurrency is maintained; and the adjustment of the adjusted call concurrency according to the interface data fed back by the downstream service interface includes: when the number of consecutive uses of the adjusted call concurrency is greater than or equal to the number of times the adjusted call concurrency is maintained, the adjustment of the adjusted call concurrency according to the interface data fed back by the downstream service interface.

[0149] The processing module 12 is specifically used to obtain a first number of tasks to be executed in a running state in the total tasks, a second number of unprocessed tasks to be executed in the total tasks, the priority of each task to be executed, a third number of uncompleted data call processes in the tasks to be executed in a running state, and a fourth number of the sum of the number of uncompleted data call processes in each task to be executed in the total tasks; determine the timeliness weight of the tasks to be executed in a running state based on the first number, the second number, the priority of each task to be executed, the third number and the fourth number.

[0150] The processing module 12 is specifically used to determine the fifth number of downstream service interfaces that have feedback abnormalities based on the interface data; obtain the total number of downstream service interfaces, the sixth number of abnormal data call processes in the executed data call processes, and the seventh number of data call processes that have been executed; and determine the proportion weight of the interface abnormality based on the fifth number, the total number, the sixth number and the seventh number.

[0151] The processing module 12 is specifically used to determine the eighth number of unfinished data call processes based on the interface data, and the total time consumed by the completed data call processes is the first time consumed; obtain the ninth number of completed tasks in the total tasks, and the total time consumed by the completed tasks is the second time consumed; determine the urgency weight of the unfinished data call processes based on the sixth number, the eighth number, the ninth number, the first time consumed and the second time consumed.

[0152] The data collection device 10 also includes a storage module 13, which is used to obtain the request message received by the downstream service interface and the process identifier carried by the request message; according to the process identifier carried by the request message, the request message is stored in the message list of the data call process corresponding to the process identifier.

[0153] The data collection device 10 also includes a detection module 14, which is used to repeatedly execute the steps of: when the call concurrency obtained by the i-th adjustment is equal to zero, generating a detection message according to the request message stored in the message list of each data call process; sending the detection message to each downstream service interface, and redetermining the call concurrency according to the interface data fed back by each downstream service interface, until the redetermined call concurrency is greater than zero, and then executing each data call process according to the redetermined call concurrency.

[0154] See Figure 9 One embodiment of the present application further provides an electronic device 20, comprising a processor 21 and a memory 22 in communication with the processor 21. The memory 22 stores computer-executable instructions, and the processor 21 executes the computer-executable instructions stored in the memory 22 to implement the data collection method provided in any of the above embodiments.

[0155] The present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the instructions are executed, the computer-executable instructions are executed by a processor to implement the data collection method provided in any of the above embodiments.

[0156] The present application also provides a computer program product, including a computer program, which implements the data collection method provided in any of the above embodiments when executed by a processor.

[0157] It should be noted that the computer-readable storage medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface mount storage device, an optical disc, or a compact disc read-only memory (CD-ROM). It may also be various electronic devices that include one or any combination of the above memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0158] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0159] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0161] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0164] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data collection method, characterized in that: include: Obtain each data call process indicated by the task to be executed, and obtain downstream service interface information corresponding to each data call process, wherein one data call process corresponds to information of at least one downstream service interface, and the task to be executed is used to instruct to obtain target data from the downstream service interface by executing each data call process; Obtain the initial call concurrency when the task to be executed begins, and execute each data call process according to the initial call concurrency, wherein the initial call concurrency is used to represent the number of executions of each data call process. When the data call process is executed, a request message is sent to the corresponding downstream service interface, and the request message is used to request the corresponding downstream service interface to feedback interface data; When the interface data fed back by the corresponding downstream service interface is received, the initial call concurrency is adjusted according to the interface data, including: obtaining the timeliness weight of the tasks to be executed in the running state in the total tasks, and obtaining the proportion weight of the interface exceptions, and obtaining the urgency weight of the data call process that has not been completed, wherein the proportion weight is less than or equal to the preset weight, and the urgency weight is greater than or equal to zero; obtaining the first average time taken to complete each data call process according to the initial call concurrency according to the interface data, and obtaining the number of interface exceptions determined according to the interface data as a first number; adjusting the initial call concurrency according to the timeliness weight, the proportion weight, the urgency weight, the first average time taken and the first number to obtain the adjusted call concurrency; Execute each data call process according to the adjusted call concurrency, and adjust the adjusted call concurrency according to the interface data fed back by the downstream service interface until the target data is obtained and the task to be executed is completed.

2. The method according to claim 1, characterized in that The adjusting the adjusted call concurrency according to the interface data fed back by the downstream service interface includes: Determine, based on the fed-back interface data, a second average time taken to complete each data call process after executing the adjusted call concurrency, and obtain, based on the fed-back interface data, a number of interface exceptions as a second number; The adjusted call concurrency is further adjusted according to the timeliness weight, the proportion weight, the urgency weight, the first average time consumption, the second average time consumption, and the second number to obtain a readjusted call concurrency.

3. The method according to claim 2, characterized in that Each concurrent call volume also has a retention count, and adjusting the initial concurrent call volume according to the timeliness weight, the proportion weight, the urgency weight, the first average duration, and the first number to obtain the adjusted concurrent call volume includes: Adjusting the initial concurrent call volume according to the timeliness weight, the proportion weight, the urgency weight, the first average duration, the first number, and the number of times the initial concurrent call volume is maintained to obtain the adjusted concurrent call volume; The adjusting the adjusted concurrent call volume according to the timeliness weight, the proportion weight, the urgency weight, the first average time consumption, the second average time consumption, and the second number to obtain the adjusted concurrent call volume includes: According to the timeliness weight, the proportion weight, the urgency weight, the first average time consumption, the second average time consumption, the second number and the number of times the adjusted call concurrency is maintained, the adjusted call concurrency is adjusted to obtain the readjusted call concurrency.

4. The method according to claim 3, characterized in that The adjusting the initial concurrent call volume according to the timeliness weight, the proportion weight, the urgency weight, the first average time consumption, the first number, and the number of times the initial concurrent call volume is maintained to obtain the adjusted concurrent call volume includes: Determining a first adjustment ratio for a first positive adjustment according to the timeliness weight, the first average time consumption, and the initial concurrent call volume; Determining a second adjustment ratio of the first positive adjustment according to the remaining execution time of the current task to be executed, the amount of target data to be acquired for the current task to be executed, and the urgency weight; Determining the size of the first reverse adjustment concurrency according to the first number and the urgency weight; The adjusted call concurrency is obtained according to the first adjustment ratio of the first forward adjustment, the second adjustment ratio of the first forward adjustment, and the size of the first reverse adjustment concurrency.

5. The method according to claim 4, characterized in that The adjusting the adjusted concurrent call volume according to the timeliness weight, the proportion weight, the urgency weight, the first average time consumption, the second average time consumption, the second number, and the number of times the adjusted concurrent call volume is maintained includes: Determining a first adjustment ratio for a second positive adjustment based on the timeliness weight, the first average time consumption, the second average time consumption, the initial concurrent call volume, and the adjusted concurrent call volume; determining a second adjustment ratio for the second positive adjustment according to the remaining execution time of the current task to be executed, the amount of target data to be acquired for the current task to be executed, and the urgency weight; Determining the size of the second reverse adjustment concurrency according to the second number and the urgency weight; The readjusted call concurrency is obtained according to the first adjustment ratio of the second forward adjustment, the second adjustment ratio of the second forward adjustment, and the size of the second reverse adjustment concurrency.

6. The method according to claim 3, characterized in that The adjusting the initial call concurrency according to the interface data includes: When the number of consecutive uses of the initial concurrent call amount is greater than or equal to the number of times the initial concurrent call amount is maintained, adjusting the initial concurrent call amount according to the interface data; The adjusting the adjusted call concurrency according to the interface data fed back by the downstream service interface includes: When the number of consecutive uses of the adjusted call concurrency is greater than or equal to the number of retention times of the adjusted call concurrency, the adjusted call concurrency is readjusted according to the interface data fed back by the downstream service interface.

7. The method according to any one of claims 1 to 6, characterized in that The timeliness weight of the tasks to be executed in the running state in the total tasks is obtained as follows: Obtain a fourth quantity as the sum of a first quantity of pending tasks in a running state in the total tasks, a second quantity of unprocessed pending tasks in the total tasks, a priority of each pending task, a third quantity of uncompleted data call processes in the pending tasks in a running state, and the number of uncompleted data call processes in each pending task in the total tasks; The timeliness weight of the tasks to be executed in the running state is determined according to the first number, the second number, the priority of each task to be executed, the third number, and the fourth number.

8. The method according to claim 7, characterized in that The acquisition interface exception weights include: Determining a fifth number of downstream service interfaces with feedback exceptions according to the interface data; Obtain the total number of downstream service interfaces, the sixth number of abnormal data call processes in the executed data call processes, and the seventh number of completed data call processes; A weight of the proportion of interface abnormalities is determined according to the fifth number, the total number, the sixth number, and the seventh number.

9. The method according to claim 8, characterized in that The urgency weight of the incomplete data call process includes: Determine, according to the interface data, an eighth number of uncompleted data call processes and a total time consumption of completed data call processes as a first time consumption; Get the ninth number of tasks that have been completed in the total tasks. The total time taken by the tasks that have been completed is the second time taken. An urgency weight of an uncompleted data calling process is determined according to the sixth number, the eighth number, the ninth number, the first time consumption, and the second time consumption.

10. The method according to claim 1, characterized in that After receiving the interface data fed back by the downstream service interface, the method further includes: Obtain the request message received by the downstream service interface and the process identifier carried in the request message; According to the process identifier carried by the request message, the request message is stored in the message list of the data call process corresponding to the process identifier; Repeat steps 1 and 2. When the adjusted call concurrency is equal to zero, a detection message is generated based on the request message stored in the message list of each data call process; The detection message is sent to each downstream service interface, and the call concurrency is re-determined according to the interface data fed back by each downstream service interface, until the re-determined call concurrency is greater than zero, and each data call process is executed according to the re-determined call concurrency.

11. A data collection device, characterized in that: include: An acquisition module, configured to acquire each data call process indicated by a to-be-executed task, and to acquire information about a downstream service interface corresponding to each data call process, wherein one data call process corresponds to information about at least one downstream service interface, and the to-be-executed task is configured to instruct the acquisition of target data from the downstream service interface by executing each data call process; The acquisition module is further used to obtain the initial call concurrency when the task to be executed starts, and execute each data call process according to the initial call concurrency, wherein the initial call concurrency is used to represent the number of executions of each data call process. When the data call process is executed, a request message is sent to the corresponding downstream service interface, and the request message is used to request the corresponding downstream service interface to feedback interface data; A processing module is used to obtain, when receiving interface data fed back by the corresponding downstream service interface, the timeliness weight of the tasks to be executed in the running state in the total tasks, the proportion weight of the interface exception, and the urgency weight of the data call process that has not been completed, wherein the proportion weight is less than or equal to the preset weight, and the urgency weight is greater than or equal to zero; obtain, according to the interface data, a first average time consumed to complete each data call process according to the initial call concurrency, and obtain the number of interface exceptions determined according to the interface data as a first number; adjust the initial call concurrency according to the timeliness weight, the proportion weight, the urgency weight, the first average time consumed and the first number to obtain the adjusted call concurrency; The processing module is also used to execute each data call process according to the adjusted call concurrency, and adjust the adjusted call concurrency according to the interface data fed back by the downstream service interface until the target data is obtained and the task to be executed is completed.

12. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the data collection method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed, enable the computer to execute the data collection method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The invention comprises a computer program, which implements the data collection method according to any one of claims 1 to 10 when executed by a processor.

Images