Data acquisition method and device, electronic equipment and storage medium

Abstract

The invention provides a data acquisition method and device, electronic equipment and a storage medium, belongs to the field of data acquisition, and is suitable for the field of financial science and technology. The method comprises the following steps: acquiring a data acquisition instruction for a plurality of target data sources and data source information corresponding to each target data source; configuring a corresponding data acquisition parameter for each target data source according to the data source information and a preset parameter configuration template; wherein the data acquisition parameter comprises a task execution level corresponding to the target data source; constructing data acquisition tasks for the corresponding target data sources based on the data acquisition parameters to obtain a plurality of data acquisition tasks; distributing the data acquisition tasks with the same task execution level to the same acquisition task group to obtain a plurality of acquisition task groups; and executing data acquisition operation on each acquisition task group according to the task execution level to obtain target acquisition data. According to the invention, large-scale and multi-interface financial data acquisition efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data collection and is applicable to the field of financial technology, and in particular to a data collection method and device, an electronic device and a storage medium. Background Art

[0002] In the field of financial technology, data collection is a key link in supporting financial business operations and decision-making. Its work is directly related to the accuracy of investment analysis and the effectiveness of risk control. Due to the requirements of highly digital development, financial business operations need to obtain and process massive data from multiple data sources in real time, including key information such as transaction data, user behavior, and regulatory reports. These data are often distributed in different API interfaces. The timeliness and accuracy of data acquisition directly affect the risk control, investment decision-making and customer service quality of financial activities. With the rapid development of financial technology, the demand for financial data collection is characterized by high frequency, large scale and diversity.

[0003] In related technologies, financial data collection often relies on traditional data stream processing tools for data synchronization, or collection software based on application program interfaces. Traditional data stream processing tools lack targeted optimization for financial scenarios and are difficult to cope with the surge in data brought about by the ever-changing financial scenarios. Collection software based on application program interfaces performs poorly in high-concurrency scenarios and is difficult to meet the real-time and accuracy requirements of modern financial services. Therefore, a more efficient and reliable data collection method is urgently needed. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a data collection method and device, an electronic device and a storage medium, aiming to improve the efficiency of large-scale, multi-interface financial data collection.

[0005] To achieve the above object, a first aspect of an embodiment of the present application provides a data collection method, the method comprising:

[0006] Acquire data collection instructions for multiple target data sources and data source information corresponding to each of the target data sources;

[0007] According to the data source information and a preset parameter configuration template, corresponding data collection parameters are configured for each of the target data sources; wherein the data collection parameters include a task execution level corresponding to the target data source;

[0008] Building a data collection task for the corresponding target data source based on the data collection parameters to obtain a plurality of the data collection tasks;

[0009] Allocating the data collection tasks with the same task execution level to the same collection task group to obtain a plurality of the collection task groups;

[0010] Perform data collection operations on each of the collection task groups according to the task execution level to obtain target collection data.

[0011] In some embodiments, configuring corresponding data collection parameters for each target data source according to the data source information and a preset parameter configuration template includes:

[0012] Parsing the data source information to obtain the corresponding interface address, access credentials and data format information of the target data source;

[0013] Based on the parameter configuration template, the interface address, the access credential and the data format information are standardized and encapsulated;

[0014] According to the data update frequency of the target data source, set a corresponding collection period;

[0015] Based on the collection period, the task execution level is determined.

[0016] In some embodiments, allocating the data collection tasks with the same task execution level to the same collection task group to obtain multiple collection task groups includes:

[0017] Initializing a plurality of the acquisition task groups, each of the acquisition task groups corresponding to one of the task execution levels;

[0018] Traversing the data collection tasks, and assigning each of the data collection tasks to the corresponding collection task group according to the corresponding task execution level;

[0019] The collection task groups are sorted according to the task execution levels corresponding to the collection task groups.

[0020] In some embodiments, when performing data collection operations on each of the collection task groups according to the task execution level, if data collection failure is detected, the method further includes:

[0021] Get the error type of data collection failure;

[0022] Determine a basic retry time and a maximum number of retries according to the error type;

[0023] Calculate an initial retry time interval according to the basic retry time;

[0024] Adding a random jitter value to the initial retry time interval to obtain a target retry time interval;

[0025] After waiting for the target retry time interval, a request retry operation is performed.

[0026] In some embodiments, after executing the request retry operation, if the retry operation still fails, the method further includes:

[0027] Get the current number of retries;

[0028] When the current number of retries is less than the maximum number of retries, the current number of retries is incremented;

[0029] Returning to execute the adding of the random jitter value to the initial retry time interval to obtain a target retry time interval;

[0030] When the current number of retries reaches the maximum number of retries, the corresponding collection task is marked as a failure state.

[0031] In some embodiments, during the process of performing the data collection operation on each of the collection task groups according to the task execution level, the method further includes:

[0032] Collecting execution status information of each of the collection task groups, the execution status information including API request status, response time, and request error rate;

[0033] Based on preset request status analysis rules, the execution status information is analyzed in real time;

[0034] When the execution status information meets the preset alarm condition, an alarm message is generated and the alarm message is pushed to a preset alarm channel;

[0035] The collection task execution strategy is dynamically adjusted according to the execution status information.

[0036] In some embodiments, after obtaining the target acquisition data, the method further includes:

[0037] Performing data integrity check on the target collected data to obtain a data integrity score;

[0038] Performing data format compliance verification on the target collected data to obtain a format compliance score;

[0039] Based on preset business rules, anomaly recognition is performed on the target collected data to obtain anomaly recognition scores;

[0040] Calculating a data quality score based on the data integrity score, the format compliance score, and the anomaly identification score;

[0041] If the data quality score is lower than a preset threshold, the corresponding data is marked as data in a pending state;

[0042] For the state data to be processed, a data quality correction task is generated and executed.

[0043] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a data acquisition device, the device comprising:

[0044] An acquisition module, used to acquire data collection instructions for multiple target data sources and data source information corresponding to each of the target data sources;

[0045] A configuration module, configured to configure corresponding data collection parameters for each of the target data sources according to the data source information and a preset parameter configuration template; wherein the data collection parameters include a task execution level corresponding to the target data source;

[0046] A construction module, used to construct a data collection task for the corresponding target data source based on the data collection parameters to obtain a plurality of the data collection tasks;

[0047] An allocation module, used for allocating the data collection tasks with the same task execution level to the same collection task group to obtain a plurality of the collection task groups;

[0048] The execution module is used to perform data collection operations on each of the collection task groups according to the task execution level to obtain target collection data.

[0049] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0050] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0051] The data collection method and device, electronic device and storage medium proposed in the present application obtain data collection instructions for multiple target data sources and data source information corresponding to each target data source; configure corresponding data collection parameters for each target data source according to the data source information and a preset parameter configuration template; wherein the data collection parameters include a task execution level corresponding to the target data source; construct data collection tasks for the corresponding target data source based on the data collection parameters to obtain multiple data collection tasks; assign data collection tasks with the same task execution level to the same collection task group to obtain multiple collection task groups; perform data collection operations on each collection task group according to the task execution level to obtain target collection data.

[0052] This application obtains the collection instructions and data source information of multiple data sources, and configures the corresponding data collection parameters for each data source in combination with the preset parameter configuration template. It realizes the refined management of data collection tasks through the task execution level, avoiding the efficiency loss caused by the use of a unified collection strategy for all data sources in the traditional method; constructs collection tasks based on data collection parameters, and assigns tasks of the same execution level to the same task group, realizing the reasonable grouping of tasks and resource optimization configuration, and avoiding mutual interference between collection tasks through the task grouping mechanism; performs data collection operations on each collection task group according to the task execution level, establishes a mapping relationship between data importance and execution priority, ensures the priority collection of key business data, and improves the overall collection efficiency. In addition, through the differentiated management of task execution levels, it is possible to flexibly adjust the collection strategy according to the characteristics of different data sources, better adapt to large-scale, multi-interface financial data collection scenarios, significantly improve the concurrent processing capability and system throughput of data collection, and effectively improve the efficiency of financial data collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of a data collection method provided in an embodiment of the present application;

[0054] Figure 2 is another flow chart of the data collection method provided in the embodiment of the present application;

[0055] Figure 3 is another flow chart of the data collection method provided in the embodiment of the present application;

[0056] Figure 4 is another flow chart of the data collection method provided in the embodiment of the present application;

[0057] Figure 5 is another flow chart of the data collection method provided in the embodiment of the present application;

[0058] Figure 6 is another flow chart of the data collection method provided in the embodiment of the present application;

[0059] Figure 7 is another flow chart of the data collection method provided in the embodiment of the present application;

[0060] Figure 8 It is a structural schematic diagram of a data acquisition device provided in an embodiment of the present application;

[0061] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0065] First, some nouns involved in this application are analyzed:

[0066] API (Application Programming Interface) refers to a set of predefined interface specifications that define the methods for program components to interact, including request and response data formats, communication protocols, and access rules. From a functional perspective, API serves as a communication bridge between data providers and data requesters, and implements standardized data transmission. Data providers expose specific data access interfaces through APIs, and data requesters obtain the required data by calling these interfaces.

[0067] In the field of financial technology, data collection is a key link in supporting financial business operations and decision-making, which is directly related to the accuracy of investment analysis and the effectiveness of risk control. Due to the requirements of highly digital development, financial business operations need to obtain and process massive data from multiple data sources in real time, including key information such as transaction data, user behavior, and regulatory reports. These data are often distributed in different API interfaces, and the timeliness and accuracy of data acquisition directly affect the risk control, investment decision-making and customer service quality of financial activities. With the rapid development of financial technology, the demand for financial data collection is characterized by high frequency, large scale and diversity.

[0068] At present, there are many data collection tools and frameworks on the market. Take Apache NiFi as an example. This tool provides powerful data flow processing capabilities and a visual operation interface, which can realize data collection, routing and data transmission between systems. However, when processing high-concurrency API requests, NiFi often encounters performance bottlenecks, and the system response time increases significantly, making it difficult to meet the needs of real-time financial data collection.

[0069] Another type of tools, such as Flink and Kafka Connect, excel in streaming data processing and support high-throughput data processing, but these tools have complex configuration and high development barriers. For financial scenarios that require rapid deployment and flexible adjustment of collection strategies, their high usage costs and maintenance difficulties limit the practical application of the tools. In addition, these tools usually require a large amount of hardware resources to be configured, and the deployment and operation and maintenance costs are high for small and medium-sized financial institutions. Therefore, a more efficient and reliable financial data collection method is urgently needed to solve the performance and reliability problems of existing technologies in large-scale, high-frequency API data collection scenarios.

[0070] Based on this, the embodiments of the present application provide a data collection method and device, an electronic device and a storage medium, aiming to improve the efficiency of large-scale, multi-interface financial data collection.

[0071] The data collection method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the data collection method in the embodiments of the present application is described.

[0072] The data collection method provided in the embodiment of the present application relates to the field of data collection. The data collection method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data collection method, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0075] Figure 1 is an optional flow chart of the data collection method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0076] Step S101: acquiring data collection instructions for multiple target data sources and data source information corresponding to each target data source.

[0077] Step S102: configuring corresponding data collection parameters for each target data source according to the data source information and a preset parameter configuration template.

[0078] Step S103: constructing a data collection task for the corresponding target data source based on the data collection parameters to obtain a plurality of data collection tasks.

[0079] Step S104: Allocate data collection tasks with the same task execution level to the same collection task group to obtain multiple collection task groups.

[0080] Step S105 , performing data collection operations on each collection task group according to the task execution level to obtain target collection data.

[0081] Steps S101 to S105 shown in the embodiment of the present application, by acquiring the collection instructions and data source information of multiple data sources, combined with the preset parameter configuration template, configure the corresponding data collection parameters for each data source, and realize the refined management of data collection tasks through the task execution level, avoiding the efficiency loss caused by the use of a unified collection strategy for all data sources in the traditional method; constructing collection tasks based on data collection parameters, and assigning tasks of the same execution level to the same task group, realizing the reasonable grouping of tasks and resource optimization configuration, and avoiding mutual interference between collection tasks through the task grouping mechanism; performing data collection operations on each collection task group according to the task execution level, establishing a mapping relationship between data importance and execution priority, ensuring the priority collection of key business data, and improving the overall collection efficiency. In addition, through the differentiated management of the task execution level, the collection strategy can be flexibly adjusted according to the characteristics of different data sources, better adapting to large-scale, multi-interface financial data collection scenarios, significantly improving the concurrent processing capability and system throughput of data collection, and effectively improving the efficiency of financial data collection.

[0082] In step S101 of some embodiments, data collection instructions for multiple target data sources and data source information corresponding to each target data source are obtained. For example, in a financial data collection scenario, the system may need to collect data from a stock market data source, a futures trading data source, and a foreign exchange quotation data source at the same time. For a stock market data source, its data source information may include the interface address, access key, data format, etc. of the stock market API; for a futures trading data source, its data source information may include data interface information, authentication method, data structure definition, etc. provided by the futures exchange; for a foreign exchange quotation data source, its data source information includes the access address, authentication token, data protocol, etc. of the foreign exchange quotation server.

[0083] In step S102 of some embodiments, corresponding data collection parameters can be configured for each target data source according to the acquired data source information and the preset parameter configuration template. For example, the task execution level of the stock market data source can be set to the highest level 1, because stock price data is very important for trading decisions; the task execution level of the futures trading data source can be set to level 2, because futures data is important but the real-time requirements are relatively low; based on the comprehensive consideration of the frequency of foreign exchange market transactions and the importance of business, the task execution level of the foreign exchange quotation data source can also be set to level 3.

[0084] See also Figure 2In some embodiments, step S102 may include but is not limited to steps S201 to S204:

[0085] Step S201, parse the data source information to obtain the interface address, access credentials and data format information of the corresponding target data source.

[0086] Step S202: Based on the parameter configuration template, the interface address, access credentials and data format information are standardized and encapsulated.

[0087] Step S203: setting a corresponding collection period according to the data update frequency of the target data source.

[0088] Step S204: determining the task execution level based on the collection cycle.

[0089] In step S201 of some embodiments, the data source information is parsed to obtain the interface address, access credentials and data format information of the corresponding target data source. For example, the interface address obtained after parsing may be "https: / / api.xxx", the access credentials include the API key "STOCK_API_KEY_123" and the access token "ACCESS_TOKEN_456", and the data format information specifies the return format of JSON or XML.

[0090] In step S202 of some embodiments, based on the parameter configuration template, the interface address, access credentials and data format information are standardized and encapsulated. For example, the original interface address of the stock market data source is added with a unified protocol header and version number, the access credentials are standardized in the format of "key type: key value", and the data format is marked as a predefined format code; for the futures trading data source, its interface address is also standardized into a standard format, the authorization code is converted into a unified credential format, and the XML format is marked as the corresponding format code. Through this standardized encapsulation, the parameter formats of different data sources are kept consistent.

[0091] In step S203 of some embodiments, a corresponding collection period is set according to the data update frequency of the target data source. For example, if the stock market data is updated once per second, the collection period is set to 1 second; if the futures trading data is updated every 5 minutes, the collection period is set to 5 minutes; if the foreign exchange quotation data is updated once every hour, the collection period is set to 1 hour. The setting of the collection period needs to match the update characteristics of the data source, to ensure the timeliness of the data and to avoid invalid repeated collection.

[0092] In step S204 of some embodiments, the task execution level is determined based on the collection cycle. Specifically, the execution level can be divided according to the length of the collection cycle. For example, a data source with a collection cycle of less than 1 second (such as stock quotes) is set to the highest level 1, indicating that real-time collection is required; a data source with a collection cycle between 1 minute and 10 minutes (such as futures trading data) is set to level 2, indicating quasi-real-time collection; a data source with a collection cycle of more than 10 minutes (such as foreign exchange quotes) is set to level 3, indicating scheduled collection. This level division method based on the collection cycle can reasonably reflect the real-time requirements of the data source.

[0093] Through the above steps S201 to S204, standardized configuration of data source parameters and reasonable division of task levels are achieved. First, the core parameters are obtained by parsing the data source information to ensure that the collection task has complete access information; then, the parameter configuration template is used for standardized packaging to improve the standardization and consistency of parameter configuration; then, the collection cycle is set based on the data update frequency to avoid resource waste and ensure data timeliness; finally, the task execution level is determined according to the collection cycle, and the corresponding relationship between data real-time requirements and execution priorities is established. This parameter configuration and task classification method not only ensures the timeliness and accuracy of financial data collection, but also improves the utilization efficiency of system resources. It is especially suitable for financial data collection scenarios with different real-time requirements such as stocks, futures, and foreign exchange.

[0094] In step S103 of some embodiments, based on the configured data collection parameters, a corresponding data collection task can be constructed for each target data source. For example, for a stock market data collection task, parameters such as the interface address, access key, data format, and task execution level 1 can be packaged to form a complete collection task; for a futures trading data collection task, related parameters and task execution level 2 can also be packaged; for a foreign exchange quotation data collection task, the corresponding parameters and task execution level 3 are assembled into a collection task.

[0095] In step S104 of some embodiments, data collection tasks of the same task execution level are assigned to the same collection task group. For example, all stock market data collection tasks with a task execution level of 1 are assigned to the first collection task group; all futures trading data collection tasks with a task execution level of 2 are assigned to the second collection task group; and all foreign exchange quotation data collection tasks with a task execution level of 3 are assigned to the third collection task group.

[0096] See also Figure 3 In some embodiments, step S104 may include but is not limited to steps S301 to S303:

[0097] Step S301, initializing multiple collection task groups, each collection task group corresponds to a task execution level.

[0098] Step S302, traverse the data collection tasks, and assign each data collection task to a corresponding collection task group according to the corresponding task execution level.

[0099] Step S303: sorting the collection task groups according to the task execution levels corresponding to the collection task groups.

[0100] In step S301 of some embodiments, multiple acquisition task groups are initialized, each of which corresponds to a task execution level. For example, three acquisition task groups may be created, corresponding to task execution levels 1, 2, and 3, respectively. The number of acquisition task groups may be determined based on the actual divided task execution levels. The initialization operation is performed to prepare for subsequent task allocation.

[0101] In step S302 of some embodiments, the data collection tasks are traversed, and each data collection task is assigned to a corresponding collection task group according to the corresponding task execution level. Specifically, the task execution level attribute of each data collection task can be checked. If it is level 1, the task is added to the collection task group corresponding to level 1; if it is level 2, it is assigned to the collection task group corresponding to level 2; and tasks at level 3 are assigned to the collection task group corresponding to level 3. By this traversal and assignment method, task grouping based on execution level is achieved.

[0102] In step S303 of some embodiments, each collection task group is sorted according to the task execution level corresponding to the collection task group. For example, the task execution level can be used as the basis for sorting, and the higher the level, the higher the sorting. Assuming that the task group of level 1 is named "group1", the one of level 2 is named "group2", and the one of level 3 is named "group3", the sorting result is {"group1", "group2", "group3"}. This sorting allows high-priority task groups to be executed first.

[0103] Through the above steps S301 to S303, the grouping and sorting of the collection tasks are realized. First, by initializing the collection task group, a basic container is provided for task allocation; then, the data collection tasks are traversed and divided into corresponding task groups according to their execution level attributes, ensuring the aggregation of tasks at the same level; finally, the collection task groups are sorted according to the execution level to determine the execution order of the task groups. This task grouping and sorting mechanism not only realizes the hierarchical management of tasks, but also clarifies the execution sequence of high-priority tasks, thereby realizing the orderly collection of financial data under limited resource conditions and improving the overall efficiency of data collection.

[0104] In step S105 of some embodiments, data collection operations are performed on each collection task group according to the task execution level. For example, the stock market data collection task in the first collection task group may be executed first to ensure that the most important market data can be obtained in a timely manner; the futures transaction data collection task in the second collection task group may be executed second; and the foreign exchange quotation data collection task in the third collection task group may be executed last. Through this hierarchical execution method, the collection needs of high-priority data can be guaranteed in priority when resources are limited.

[0105] See also Figure 4 In some embodiments, when performing data collection operations on each collection task group according to the task execution level, if data collection failure is detected, the process may also include but is not limited to steps S401 to S405:

[0106] Step S401, obtaining the error type of data collection failure.

[0107] Step S402, determining a basic retry time and a maximum number of retries according to the error type.

[0108] Step S403: Calculate an initial retry time interval according to the basic retry time.

[0109] Step S404: Add a random jitter value to the initial retry time interval to obtain a target retry time interval.

[0110] Step S405: After waiting for the target retry time interval, execute the request retry operation.

[0111] In step S401 of some embodiments, the error type of data acquisition failure is obtained. For example, the possible error types include network connection timeout, server return error code, data format parsing error, etc. By capturing and identifying different error types, a decision basis is provided for subsequent retry strategies.

[0112] In step S402 of some embodiments, the basic retry time and the maximum number of retries are determined according to the error type. For example, for a network connection timeout error, a shorter basic retry time (such as 1 second) and a larger maximum number of retries (such as 5 times) can be set; for an error code returned by the server, a longer basic retry time (such as 30 seconds) and a smaller maximum number of retries (such as 3 times) can be set; for a data format parsing error, a moderate basic retry time (such as 10 seconds) and a maximum number of retries (such as 2 times) can be set. This method of setting retry parameters according to the error type can adopt appropriate retry strategies for different failure causes.

[0113] In step S403 of some embodiments, the initial retry time interval is calculated according to the basic retry time. Specifically, the retry time interval can be calculated based on the basic retry time in combination with the exponential backoff algorithm. For example, assuming that the basic retry time is 1 second and the exponential factor of the exponential backoff is 2, the time interval for the first retry is 1 second, the second is 2 seconds, the third is 4 seconds, and so on. This exponential backoff retry time calculation method can effectively avoid server pressure caused by instantaneous concurrent retries.

[0114] In step S404 of some embodiments, a random jitter value is added to the initial retry time interval to obtain a target retry time interval. For example, a random jitter time can be added on the basis of the initial retry time interval, and the jitter range can be set to a certain proportion (such as 20%) of the initial retry time interval. Assuming that the initial retry time interval is 10 seconds, the target retry time interval after adding jitter may be 9 seconds or 12 seconds. By introducing random jitter, the time points of concurrent retries are further dispersed, reducing the instantaneous pressure on the server.

[0115] In step S405 of some embodiments, after waiting for the target retry time interval, a request retry operation is executed. That is, the system first suspends execution and waits for a period of time (i.e., the target retry time interval), and then initiates a data collection request for the failed task again. If the retry fails again, the system repeats steps S401 to S404 until the maximum number of retries is reached or the collection is successful. Through this fault-tolerant processing of waiting for retry, the success rate of data collection is effectively improved.

[0116] See also Figure 5 In some embodiments, after step 405, if it is detected that the retry operation still fails, the following steps may also include but are not limited to steps S501 to S504:

[0117] Step S501, obtaining the current number of retries.

[0118] Step S502: when the current number of retries is less than the maximum number of retries, the current number of retries is incremented.

[0119] Step S503, returning to execute adding a random jitter value to the initial retry time interval to obtain a target retry time interval.

[0120] Step S504: When the current number of retries reaches the maximum number of retries, the corresponding collection task is marked as a failure state.

[0121] In step S501 of some embodiments, the current number of retries is obtained. For example, a retry counter can be maintained for each data acquisition task, and the initial value is 0. When the retry operation of step S405 fails for the first time, step S501 is entered to obtain the current number of retries. This number of retries reflects the number of times the current task has been retried.

[0122] In step S502 of some embodiments, it is determined whether the current number of retries is less than the maximum number of retries. The maximum number of retries is determined in step S402 according to the error type and indicates the upper limit of the allowed retries. If the current number of retries has not reached the maximum number of retries, the current number of retries is incremented and recorded. This increment operation is used to update the retry counter to prepare for the next retry.

[0123] In step S503 of some embodiments, if the current number of retries is less than the maximum number of retries, the process returns to step S404, i.e., a random jitter value is added to the initial retry time interval to obtain a new target retry time interval. This means that the exponential backoff time interval and random jitter time calculated in steps S403 and S404 will be used for the next retry. By returning to step S404, a dynamic update of the retry time interval is achieved.

[0124] In step S504 of some embodiments, if the current number of retries has reached the maximum number of retries, the corresponding collection task is marked as a failure. This indicates that the task still cannot successfully complete data collection after multiple retries, and it is marked as a failure and no further retries are made. This failure mark can be used for subsequent statistics and analysis to optimize the data collection process.

[0125] Through the above steps S501 to S504, a complete retry termination mechanism is implemented. This mechanism dynamically controls the continuation or termination of the retry by obtaining the current number of retries and comparing it with the maximum number of retries. If the maximum number of retries has not been reached, the counter is incremented and the calculation and adjustment of the retry time interval are returned; if the maximum number of retries has been reached, the task is marked as a failure and the retry is terminated. This mechanism avoids the waste of system resources by unlimited retries while repeatedly trying to compensate for data collection failures. For financial data collection, timely identification and processing of failed collection tasks helps maintain data integrity and consistency and improves the robustness of the system.

[0126] Through the above steps S401 to S405, an intelligent data collection retry mechanism is implemented. The mechanism first obtains the error type, sets different retry parameters according to different errors, then uses the exponential backoff algorithm to calculate the retry time interval, introduces random jitter to disperse the concurrent pressure, and finally executes the fault-tolerant operation of waiting for retry. This retry mechanism comprehensively considers the retry time overhead and server performance, improves fault tolerance while taking into account the stability of the system, and is particularly suitable for the complex and changeable network environment in the financial data collection scenario, and can significantly improve the success rate and robustness of data collection.

[0127] See also Figure 4 In some embodiments, when performing data collection operations on each collection task group according to the task execution level, if data collection failure is detected, the process may also include but is not limited to steps S401 to S405:

[0128] See also Figure 6 In some embodiments, during the execution of step S105, the following steps may also be included but are not limited to steps S601 to S604:

[0129] Step S601: Collect the execution status information of each collection task group.

[0130] Step S602: Perform real-time analysis on the execution status information based on preset request status analysis rules.

[0131] Step S603: when the execution status information meets the preset alarm condition, alarm information is generated and pushed to the preset alarm channel.

[0132] Step S604: dynamically adjust the collection task execution strategy according to the execution status information.

[0133] In step S601 of some embodiments, the execution status information of each collection task group is collected. The execution status information includes three key indicators: API request status, response time and request error rate. The API request status reflects the execution result of the collection task, such as success, failure, timeout, etc.; the response time measures the response speed of the API interface and is used to evaluate performance; the request error rate counts the proportion of request failures and reflects the stability of the collection process. By continuously collecting these status information, the execution of the data collection task can be monitored in real time.

[0134] In step S602 of some embodiments, the execution status information is analyzed in real time based on the preset request status analysis rules. The request status analysis rules define the criteria for judging and warning the execution of the collection task. For example, threshold rules such as the API request success rate is less than 90%, the average response time is more than 500 milliseconds, and the request error rate is higher than 5% can be set. The collected execution status information is compared with the request status analysis rules in real time to detect abnormal situations in a timely manner.

[0135] In step S603 of some embodiments, when the execution status information meets the preset alarm condition, the alarm mechanism is triggered and the alarm information is generated. The alarm condition can be set based on the request status analysis rule of step S602, such as the request error rate of three consecutive cycles exceeds 10%. Once the execution status reaches the alarm condition, it means that there is a more serious problem with the data collection task, and it is necessary to promptly notify the relevant personnel to handle it, generate alarm information, and push the alarm information to the preset alarm channel. The alarm information usually includes the abnormal collection task group, specific abnormal indicators, and triggered alarm rules, so that the operation and maintenance personnel can quickly locate the problem. The alarm channel can be a text message, email, instant messaging tool, etc., which is used to notify the relevant responsible person as soon as possible.

[0136] In step S604 of some embodiments, the collection task execution strategy is dynamically adjusted according to the execution status information. This is an adaptive optimization mechanism based on monitoring feedback. For example, if the average response time of a collection task group is too long, you can consider increasing the concurrency; if the request error rate of a data source is continuously high, you can lower its collection frequency or suspend collection. By dynamically adjusting the strategy, you can achieve automatic tuning of the collection task and improve the efficiency and quality of data collection.

[0137] Through the above steps S601 to S604, a set of real-time monitoring and dynamic optimization mechanisms are constructed to ensure the smooth operation of data collection tasks. This mechanism takes the collection task group as the monitoring object, and through the real-time collection and analysis of key indicators such as API request status, response time, and request error rate, abnormal situations are discovered in time and alarms are triggered. At the same time, the collection strategy is dynamically adjusted according to monitoring feedback to achieve adaptive optimization of task execution. This monitoring and optimization mechanism is particularly critical for financial data collection, which can help quickly discover and deal with collection failures, and continuously improve collection performance and stability, thereby providing high-quality data support for upper-level financial business applications.

[0138] See also Figure 7 In some embodiments, after step S105, the following steps may also be performed, including but not limited to steps S701 to S706:

[0139] Step S701, performing data integrity check on the target collected data to obtain a data integrity score.

[0140] Step S702: verify the data format compliance of the target collected data to obtain a format compliance score.

[0141] Step S703: perform anomaly identification on the target collected data based on preset business rules to obtain anomaly identification scores.

[0142] Step S704, calculating the data quality score according to the data integrity score, the format compliance score and the anomaly recognition score.

[0143] Step S705: if the data quality score is lower than a preset threshold, the corresponding data is marked as data in a pending state.

[0144] Step S706: Generate and execute a data quality correction task for the status data to be processed.

[0145] In step S701 of some embodiments, a data integrity check is performed on the target collected data to obtain a data integrity score. The data integrity check mainly checks whether the collected data is missing or incomplete. For example, the field integrity rate of the statistical record can be counted, that is, the proportion of non-empty fields in each record; the missing of key fields (such as transaction time, amount, etc.) can also be checked. A data integrity score can be calculated based on the integrity check results to quantitatively reflect the completeness of the data.

[0146] In step S702 of some embodiments, data format compliance verification is performed on the target collected data to obtain a format compliance score. Data format compliance verification is used to check whether the collected data complies with predefined format specifications. For example, whether the date field complies with the format of "YYYY-MM-DD", whether the amount field is a numeric type, whether the string field exceeds the length limit, etc. Through format verification, a format compliance score is calculated to measure the standardization of the data format.

[0147] In step S703 of some embodiments, anomaly identification is performed on the target collected data based on preset business rules to obtain anomaly identification scores. Anomaly identification is to check the business logic rationality of the data based on some anomaly judgment rules set for business scenarios. For example, for stock trading data, rules can be set to determine whether the trading volume is abnormal (such as exceeding 3 times the standard deviation of the past 30 days); for bank transaction data, rules can be set to check whether the transaction amount exceeds the account balance. Based on the anomaly identification results, an anomaly identification score is generated to reflect the compliance of the data in business logic.

[0148] In step S704 of some embodiments, a data quality score is calculated based on the data integrity score, the format compliance score, and the anomaly recognition score. The data quality score is a quantitative indicator for comprehensive evaluation of data quality, and can be calculated based on the weighted average of the first three scores. For example, the data integrity score is weighted 40%, the format compliance score is weighted 30%, and the anomaly recognition score is weighted 30%, and then the weighted sum is obtained to obtain the final data quality score. This score fully reflects the availability and reliability of the collected data.

[0149] In step S705 of some embodiments, if the data quality score is lower than a preset threshold, the corresponding data is marked as pending. The preset threshold is a quality standard for measuring whether the data can be used directly, which can be set according to business needs, such as 80 points. If the quality score of a batch of collected data is lower than 80 points, it will be marked as pending, indicating that this part of the data needs further verification and correction and cannot be directly put into use.

[0150] In step S706 of some embodiments, a data quality correction task is generated and executed for the data in the pending state. The data quality correction task is a series of processing operations designed to further improve the data quality, which may include data completion, data format conversion, abnormal data removal, etc. The system automatically generates these correction tasks and schedules them for execution until the data quality meets the standard. For example, for data with a large number of missing fields, they can be completed by tracing back historical records; for data that does not conform to the format, format conversion can be performed; for data that is obviously abnormal, it can be directly removed.

[0151] Through the above steps S701 to S706, a data quality assurance mechanism is constructed to ensure that the collected financial data can meet the business usage requirements. The mechanism comprehensively evaluates the data quality through three dimensions: data integrity, format compliance, and anomaly identification, and calculates the data quality score, automatically identifies the data to be processed and generates corresponding quality correction tasks, thereby realizing the automatic detection and repair of data quality. This is crucial for financial data collection, and can effectively prevent business risks caused by data defects, improve the security and reliability of data use, and provide a high-quality data foundation for subsequent business analysis and decision-making.

[0152] Combining steps S101 to S105, the present application provides a full-process data collection method. This method introduces a task grouping mechanism based on the task execution level, and realizes the refined scheduling of collection tasks through standardized configuration of data source parameters and hierarchical management of tasks; at the same time, an intelligent fault-tolerant retry mechanism is designed, which can dynamically adjust the retry strategy for different data collection failure scenarios, thereby improving the success rate of data collection; in addition, a real-time monitoring and dynamic optimization mechanism is constructed, which can timely discover and deal with collection failures, and continuously optimize collection efficiency and quality. This full-process data collection method fully considers the key requirements of financial data collection such as real-time, accuracy, and fluency, and well balances the performance and cost of data collection, providing financial institutions with a set of reliable, efficient, and flexible data collection solutions.

[0153] The data collection method and device, electronic device and storage medium proposed in the present application obtain data collection instructions for multiple target data sources and data source information corresponding to each target data source; configure corresponding data collection parameters for each target data source according to the data source information and a preset parameter configuration template; wherein the data collection parameters include a task execution level corresponding to the target data source; construct data collection tasks for the corresponding target data source based on the data collection parameters to obtain multiple data collection tasks; assign data collection tasks with the same task execution level to the same collection task group to obtain multiple collection task groups; perform data collection operations on each collection task group according to the task execution level to obtain target collection data.

[0154] This application obtains the collection instructions and data source information of multiple data sources, and configures the corresponding data collection parameters for each data source in combination with the preset parameter configuration template. It realizes the refined management of data collection tasks through the task execution level, avoiding the efficiency loss caused by the use of a unified collection strategy for all data sources in the traditional method; constructs collection tasks based on data collection parameters, and assigns tasks of the same execution level to the same task group, realizing the reasonable grouping of tasks and resource optimization configuration, and avoiding mutual interference between collection tasks through the task grouping mechanism; performs data collection operations on each collection task group according to the task execution level, establishes a mapping relationship between data importance and execution priority, ensures the priority collection of key business data, and improves the overall collection efficiency. In addition, through the differentiated management of task execution levels, it is possible to flexibly adjust the collection strategy according to the characteristics of different data sources, better adapt to large-scale, multi-interface financial data collection scenarios, significantly improve the concurrent processing capability and system throughput of data collection, and effectively improve the efficiency of financial data collection.

[0155] See also Figure 8The present application also provides a data acquisition device that can implement the above data acquisition method. The device includes:

[0156] An acquisition module, used to acquire data collection instructions for multiple target data sources and data source information corresponding to each target data source;

[0157] A configuration module, configured to configure corresponding data collection parameters for each target data source according to data source information and a preset parameter configuration template; wherein the data collection parameters include a task execution level corresponding to the target data source;

[0158] A construction module is used to construct a data collection task for a corresponding target data source based on the data collection parameters to obtain multiple data collection tasks;

[0159] An allocation module, used for allocating data collection tasks with the same task execution level to the same collection task group, thereby obtaining multiple collection task groups;

[0160] The execution module is used to perform data collection operations on each collection task group according to the task execution level to obtain target collection data.

[0161] The specific implementation of the data acquisition device is basically the same as the specific implementation of the above-mentioned data acquisition method, and will not be repeated here.

[0162] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data collection method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0163] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0164] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0165] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the data acquisition method of the embodiment of this application;

[0166] Input / output interface 903, used to implement information input and output;

[0167] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0168] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0169] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0170] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned data collection method when executed by a processor.

[0171] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0172] The data collection method and device, electronic device and storage medium proposed in the present application obtain data collection instructions for multiple target data sources and data source information corresponding to each target data source; configure corresponding data collection parameters for each target data source according to the data source information and a preset parameter configuration template; wherein the data collection parameters include a task execution level corresponding to the target data source; construct data collection tasks for the corresponding target data source based on the data collection parameters to obtain multiple data collection tasks; assign data collection tasks with the same task execution level to the same collection task group to obtain multiple collection task groups; perform data collection operations on each collection task group according to the task execution level to obtain target collection data.

[0173] This application obtains the collection instructions and data source information of multiple data sources, and configures the corresponding data collection parameters for each data source in combination with the preset parameter configuration template. It realizes the refined management of data collection tasks through the task execution level, avoiding the efficiency loss caused by the use of a unified collection strategy for all data sources in the traditional method; constructs collection tasks based on data collection parameters, and assigns tasks of the same execution level to the same task group, realizing the reasonable grouping of tasks and resource optimization configuration, and avoiding mutual interference between collection tasks through the task grouping mechanism; performs data collection operations on each collection task group according to the task execution level, establishes a mapping relationship between data importance and execution priority, ensures the priority collection of key business data, and improves the overall collection efficiency. In addition, through the differentiated management of task execution levels, it is possible to flexibly adjust the collection strategy according to the characteristics of different data sources, better adapt to large-scale, multi-interface financial data collection scenarios, significantly improve the concurrent processing capability and system throughput of data collection, and effectively improve the efficiency of financial data collection.

[0174] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0175] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0176] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0178] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0179] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one item (one) of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0181] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0183] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0184] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data collection method, characterized in that: include: Acquire data collection instructions for multiple target data sources and data source information corresponding to each of the target data sources; According to the data source information and a preset parameter configuration template, corresponding data collection parameters are configured for each of the target data sources; wherein the data collection parameters include a task execution level corresponding to the target data source; Building a data collection task for the corresponding target data source based on the data collection parameters to obtain a plurality of the data collection tasks; Allocating the data collection tasks with the same task execution level to the same collection task group to obtain a plurality of the collection task groups; Perform data collection operations on each of the collection task groups according to the task execution level to obtain target collection data.

2. The data collection method according to claim 1, characterized in that: The configuring corresponding data collection parameters for each target data source according to the data source information and a preset parameter configuration template includes: Parsing the data source information to obtain the corresponding interface address, access credentials and data format information of the target data source; Based on the parameter configuration template, the interface address, the access credential and the data format information are standardized and encapsulated; According to the data update frequency of the target data source, set a corresponding collection period; Based on the collection period, the task execution level is determined.

3. The data collection method according to claim 1, characterized in that: The data collection tasks having the same task execution level are assigned to the same collection task group to obtain a plurality of collection task groups, including: Initializing a plurality of the acquisition task groups, each of the acquisition task groups corresponding to one of the task execution levels; Traversing the data collection tasks, and assigning each of the data collection tasks to the corresponding collection task group according to the corresponding task execution level; The collection task groups are sorted according to the task execution levels corresponding to the collection task groups.

4. The data collection method according to claim 1, characterized in that: When performing the data collection operation on each of the collection task groups according to the task execution level, if data collection failure is detected, the method further includes: Get the error type of data collection failure; Determine a basic retry time and a maximum number of retries according to the error type; Calculate an initial retry time interval according to the basic retry time; Adding a random jitter value to the initial retry time interval to obtain a target retry time interval; After waiting for the target retry time interval, a request retry operation is performed.

5. The data collection method according to claim 4, characterized in that: After executing the request retry operation, if the retry operation still fails, the method further includes: Get the current number of retries; When the current number of retries is less than the maximum number of retries, the current number of retries is incremented; Returning to execute the adding of the random jitter value to the initial retry time interval to obtain a target retry time interval; When the current number of retries reaches the maximum number of retries, the corresponding collection task is marked as a failure state.

6. The data collection method according to claim 1, characterized in that: In the process of performing data collection operations on each of the collection task groups according to the task execution level, the method further includes: Collecting execution status information of each of the collection task groups; wherein the execution status information includes API request status, response time and request error rate; Based on preset request status analysis rules, the execution status information is analyzed in real time; When the execution status information meets the preset alarm condition, an alarm message is generated and the alarm message is pushed to a preset alarm channel; The collection task execution strategy is dynamically adjusted according to the execution status information.

7. The data collection method according to claim 1, characterized in that: After obtaining the target collection data, it also includes: Performing data integrity check on the target collected data to obtain a data integrity score; Performing data format compliance verification on the target collected data to obtain a format compliance score; Based on preset business rules, anomaly recognition is performed on the target collected data to obtain anomaly recognition scores; Calculating a data quality score based on the data integrity score, the format compliance score, and the anomaly identification score; If the data quality score is lower than a preset threshold, the corresponding data is marked as data in a pending state; For the state data to be processed, a data quality correction task is generated and executed.

8. A data acquisition device, characterized in that: include: An acquisition module, used to acquire data collection instructions for multiple target data sources and data source information corresponding to each of the target data sources; A configuration module, configured to configure corresponding data collection parameters for each of the target data sources according to the data source information and a preset parameter configuration template; wherein the data collection parameters include a task execution level corresponding to the target data source; A construction module, used to construct a data collection task for the corresponding target data source based on the data collection parameters to obtain a plurality of the data collection tasks; An allocation module, used for allocating the data collection tasks with the same task execution level to the same collection task group to obtain a plurality of the collection task groups; The execution module is used to perform data collection operations on each of the collection task groups according to the task execution level to obtain target collection data.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor implements the data collection method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the data collection method according to any one of claims 1 to 7.

Images