Data collection method, device, computer equipment and storage medium

Abstract

The present application provides a data collection method, device, computer equipment and storage medium, which belongs to the field of data management technology. In the present application, by automatically configuring reliability enhancement strategies for data interfaces of different interface types, it can be ensured that no data is missed when executing data collection tasks, data loss is avoided, and data integrity is enhanced. For data sources with multiple copies, the above method can ensure the consistency of data between multiple copies of the data source. Furthermore, on the basis of data collection based on the reliability enhancement strategy, the collected data is checked for data integrity, which further ensures the integrity of the data. Then, the collected data is deduplicated to improve the data quality. The above methods are used in combination to ensure the reliability of the data.

Description

Technical Field

[0001] The present application relates to the field of data management technology, and in particular to a data acquisition method, device, computer equipment and storage medium. Background Art

[0002] Database is an important technical platform for data management, and data collection, as the core link of data management, is of great significance to data management.

[0003] Current data collection methods are all based on fixed rules. For example, if the target data is data from a specific time, the server uses the specified time as an input parameter to the data source system and collects data from the data source system updated at that specified time through the data interface.

[0004] However, when using the above data collection methods to collect data, if the system encounters an abnormality or is in a critical state, data may be missed during the data collection task, resulting in data loss and affecting data integrity. If there are multiple copies of the data source, due to network delays or untimely data synchronization, the data stored in multiple copies may be inconsistent, resulting in inconsistent collected data. In summary, data reliability is poor. Summary of the Invention

[0005] The embodiments of the present application provide a data collection method, apparatus, computer device, and storage medium, which ensure the integrity and consistency of data and enhance the reliability of data. The technical solution is as follows:

[0006] In one aspect, a data collection method is provided, comprising:

[0007] In response to receiving the incoming parameters of the data acquisition task, optimizing the incoming parameters based on a target reliability enhancement strategy of the data interface, where the target reliability enhancement strategy is an incoming parameter processing method corresponding to the interface type to which the data interface belongs;

[0008] According to the optimized input parameters, the data interface is called to execute the data collection task;

[0009] Process the collected data and store the processed data in the database.

[0010] In another aspect, a data acquisition device is provided, comprising:

[0011] an incoming parameter optimization module, configured to, in response to receiving incoming parameters of a data acquisition task, optimize the incoming parameters based on a target reliability enhancement strategy for the data interface, wherein the target reliability enhancement strategy is an incoming parameter processing method corresponding to the interface type to which the data interface belongs;

[0012] A data interface calling module, configured to call the data interface according to the optimized input parameters to execute the data collection task;

[0013] The data processing module is used to process the collected data and store the processed data in the database.

[0014] In one possible implementation, the apparatus further includes:

[0015] The policy configuration module is used to configure a target reliability enhancement policy for the data interface based on the interface type of the data interface of the data management platform. The target reliability enhancement policy is an input parameter processing method corresponding to the interface type of the data interface.

[0016] In one possible implementation, the policy configuration module is configured to, for a first interface type, which refers to a data interface using natural time as an increment field, configure a first reliability enhancement policy for the data interface if the data acquisition protocol supported by the data interface is data whose update time is greater than the input parameter, wherein the first reliability enhancement policy is to subtract one time unit from the input parameter of the data acquisition task;

[0017] For the second interface type, the second interface type refers to a data interface with a start and end time period as an incremental field, and the data acquisition protocol supported by the data interface is data whose transmission update time is within the start and end time period. A second reliability enhancement strategy is configured for the data interface, and the second reliability enhancement strategy is to use the input parameters as a baseline and move forward and backward by specified time steps respectively.

[0018] In one possible implementation, the incoming parameter optimization module is configured to, for the first interface type, subtract one time unit from the incoming parameter of the data acquisition task if the data acquisition protocol supported by the data interface is data whose transmission update time is greater than the incoming parameter;

[0019] For the second interface type, the data acquisition protocol supported by the data interface is data whose update time is within the start and end time periods, and the input parameters are used as a baseline to move forward and backward by a specified time step respectively.

[0020] In one possible implementation, the data processing module is configured to perform a corresponding data integrity check on the collected data based on the interface type of the data interface; if the integrity check passes, a data deduplication step is performed; if the integrity check fails, the data collection task is re-executed;

[0021] The collected data is compared with the data in the buffer area to remove duplicate data.

[0022] In one possible implementation, the apparatus further includes:

[0023] A scheme configuration module is used to obtain a target data integrity verification scheme from a mapping relationship between the interface type and the data integrity verification scheme based on the interface type of the data interface, and configure the target data integrity verification scheme for the data interface, wherein the target data integrity verification scheme is the data integrity verification scheme corresponding to the interface type.

[0024] In one possible implementation, the data processing module is configured to, for a third interface type, which refers to a data interface using a version number as an increment field, compare the version number of the collected data with the version number of the data in the cache area one by one; if the version number of the collected data is the same as the version number of the data in the cache area, remove duplicate data from the collected data; if the version number of the collected data is different from the version number of the data in the cache area, retain the data in the collected data;

[0025] For other interface types, the primary key of the collected data is compared one by one with the primary key of the cache data. If the primary key of the collected data is the same as the primary key of the cache data, the duplicate data in the collected data is removed; if the primary key of the collected data is different from the primary key of the cache data, the data in the collected data is retained.

[0026] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the data acquisition method in the optional implementation of the present application.

[0027] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed by the data acquisition method in the optional implementation of the present application.

[0028] On the other hand, a computer program product or computer program is provided, which includes computer program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device performs the operations performed by the data acquisition method provided in the various optional implementations mentioned above.

[0029] In an embodiment of the present application, by automatically configuring reliability enhancement strategies for data interfaces of different interface types, it is possible to ensure that no data is missed when executing data collection tasks, avoid data loss, and enhance data integrity. Moreover, for a data source with multiple copies, the above method can ensure the consistency of data between multiple copies of the data source. Furthermore, on the basis of data collection based on the reliability enhancement strategy, a data integrity check is performed on the collected data to further ensure the integrity of the data. The collected data is then deduplicated to improve data quality. The above methods are used in combination to ensure the reliability of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] Figure 1 This is an implementation environment of a data collection method provided according to an embodiment of the present application;

[0032] Figure 2 This is a flow chart of a data collection method provided according to an embodiment of the present application;

[0033] Figure 3 This is a flow chart of a method for configuring reliability enhancement strategies for data interfaces of different interface types provided in an embodiment of the present application;

[0034] Figure 4 is a block diagram of a data acquisition device provided according to an embodiment of the present application;

[0035] Figure 5 This is a structural block diagram of a system architecture of a data acquisition method provided in an embodiment of the present application;

[0036] Figure 6 It is a structural diagram of a server provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0038] In the present application, the term "at least one" means one or more, and the term "plurality" means two or more.

[0039] In this application, the term "and / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0040] It should be noted that the data involved in this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0041] Figure 1 This is a schematic diagram of an implementation environment of a data collection method provided in accordance with an embodiment of the present application. Figure 1 , the implementation environment includes a data management platform 101 and multiple data source systems 102 .

[0042] The data management platform 101 is used to collect data from various data source systems 102 and process and store the collected data to provide data services. In some embodiments, the data management platform 101 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), big data and artificial intelligence platforms.

[0043] The multiple data source systems 102 are used to provide data to the data management platform 101 , and the multiple data source systems 102 transmit data with the data management platform 101 via a data interface.

[0044] In some embodiments, the servers in the data management platform 101 can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0045] In some embodiments, the wireless or wired network described above uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to any combination of a LAN (Local Area Network), a MAN (Metropolitan Area Network), a WAN (Wide Area Network), a mobile, wired or wireless network, a dedicated network, or a virtual private network. In some embodiments, technologies and / or formats including HTML (Hypertext Markup Language) and XML (Extensible Markup Language) are used to represent data exchanged over the network. Conventional encryption technologies such as SSL (Secure Socket Layer), TLS (Transport Layer Security), VPN (Virtual Private Network), and IPsec (Internet Protocol Security) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used in place of or in addition to the aforementioned data communication technologies.

[0046] Figure 2 This is a flow chart of a data collection method provided in accordance with an embodiment of the present application. Figure 2 As shown, taking the execution subject as the server of the data management platform as an example, the method includes the following steps:

[0047] 201. Based on the interface type of the data interface of the data management platform, configure a target reliability enhancement policy for the data interface. The target reliability enhancement policy is a reliability enhancement policy corresponding to the interface type.

[0048] In the embodiment of the present application, the reliability enhancement strategy refers to a new method of processing incoming parameters to enhance the reliability of the data interface, wherein the incoming parameters refer to the collection conditions corresponding to the data collection task.

[0049] The interface type is determined through automatic identification or manual selection. The automatic identification process includes identifying the interface type of the data interface based on the protocol fields supported by the data interface. The manual selection process includes providing multiple optional interface types for the data interface and, upon selection of any interface type by the administrator, determining the selected interface type as the interface type of the data interface.

[0050] In some embodiments, the above-mentioned step 201 includes: based on the interface type, obtaining the target reliability enhancement policy from the mapping relationship between the interface type and the reliability enhancement policy, and configuring the target reliability enhancement policy for the data interface, wherein the target reliability enhancement policy is the reliability enhancement policy corresponding to the interface type.

[0051] The above step 201 is a process of optimizing the data interface strategy, which can be executed at any stage. For each data collection process, it is not necessary to execute the strategy optimization process every time data is collected, but it can be executed based on the currently configured strategy.

[0052] See also Figure 3 ,For different interface types, different reliability enhancement strategies are configured.,The following describes the reliability enhancement strategies based on different,interface types.

[0053] The incremental fields involved in the first four situations below refer to the fields containing the newly added data in the data source system.

[0054] Case 1: The interface type is the first interface type, which refers to a data interface using natural time as the increment field.

[0055] Among them, natural time refers to time units with larger granularity such as years, months, days, seasons, weeks, hours, and minutes.

[0056] For the first interface type, based on the data acquisition protocol supported by the data interface, determine whether to configure a reliability enhancement policy for the data interface. If the data acquisition protocol supported by the data interface is data with a transmission update time greater than or equal to the input parameter, then no reliability enhancement policy is configured for the data interface, and the original data interface is used to transmit data. If the data acquisition protocol supported by the data interface is data with a transmission update time greater than the input parameter, then a first reliability enhancement policy is configured for the data interface, where the first reliability enhancement policy refers to subtracting one time unit from the input parameter of the data acquisition task to obtain an optimized input parameter.

[0057] For data interfaces that support data collection protocols that transmit data with an update time greater than or equal to the input parameters, if the data they transmit meets the consistency and integrity requirements, there is no need to configure a reliability enhancement policy. However, for data interfaces that support data collection protocols that transmit data with an update time greater than the input parameters, under certain critical conditions, for example, when the start time in the input parameters issued by the data collection task is the same as the data update time of the data source system, data collected based on the above data collection protocol may omit data updated during the critical state time. Therefore, it is necessary to configure a reliability enhancement policy to prevent the omission of data updated during the critical state time.

[0058] The second case: the interface type is the second interface type, which refers to a data interface with a start and end time period as an increment field.

[0059] The start and end time periods may be determined by a start time point and an end time point, or by a start time point and a data collection duration, and are not limited thereto.

[0060] For the second interface type, the data acquisition protocol supported by the data interface is data with a transmission update time within the start and end time period, and a second reliability enhancement strategy is configured for the data interface, wherein the second reliability enhancement strategy refers to taking the input parameters as the baseline, moving forward and backward by specified time steps respectively to obtain new input parameters, and obtaining new start and end time periods based on the new input parameters.

[0061] For data interfaces that support data collection protocols that transmit data updated within the start and end time periods, only partial data may be collected at the start and end time points of the data collection task based on the above data collection protocols. Therefore, a reliability enhancement strategy must be configured to obtain all data updated by the data source system within the start and end time periods.

[0062] The third case: the interface type is the third interface type, which refers to a data interface with a version number as an incremental field.

[0063] For the third interface type, the data acquisition protocol supported by this data interface is data that transmits version numbers corresponding to the input parameters. Since the data in the data source system has its own unique version number, the integrity and consistency of the data are guaranteed. Therefore, this type of data interface does not need to be configured with a reliability enhancement strategy and uses the original data interface to transmit data.

[0064] The fourth case: the interface type is the fourth interface type, which refers to a data interface with a specified time as the increment field.

[0065] For the fourth interface type, the data acquisition protocol supported by this data interface is to transmit data with the same update time as the input parameters. Since the update time of the data in the data source system is unique, the data interface transmits data at the specified time each time, ensuring the integrity and consistency of the data. Therefore, this type of data interface does not need to be configured with a reliability enhancement strategy and uses the original data interface to transmit data.

[0066] Case 5: The interface type is other interface types.

[0067] For data interfaces of other interface types, you can configure customized reliability enhancement policies for them.

[0068] 202. In response to receiving an input parameter of the data acquisition task, optimize the input parameter based on a target reliability enhancement strategy of the data interface.

[0069] In some embodiments, for the first interface type, if the data acquisition protocol supported by the data interface is data with a transmission update time greater than or equal to the input parameters, the reliability enhancement strategy is not configured for the data interface, and accordingly, the input parameters are not optimized. If the data acquisition protocol supported by the data interface is data with a transmission update time greater than the input parameters, the first reliability enhancement strategy configured for the data interface is to subtract one time unit from the input parameters of the data acquisition task, and optimize the input parameters based on the first reliability enhancement strategy.

[0070] For example, the data collection task is to collect data updated by the data source system after 8:30, where the input parameter is 8:30, the time unit is minutes, and the data collection protocol supported by the data interface is to transmit data with an update time greater than the input parameter. Based on the first reliability enhancement strategy, the input parameter of the data collection task is subtracted by one time unit, and the input parameter is adjusted from 8:30 to 8:29 to optimize the input parameter.

[0071] In some embodiments, for the second interface type, the data acquisition protocol supported by the data interface is data whose transmission update time is within the start and end time period. The second reliability enhancement strategy configured for the data interface is to use the input parameters as the baseline, move forward and backward by specified time steps respectively, and optimize the input parameters based on the second reliability enhancement strategy.

[0072] For example, the data collection task is to collect data updated by the data source system between 8:30 and 8:40, where the input parameters are 8:30 and 8:40, the specified time step is 1 minute, and the data collection protocol supported by the data interface is to transmit data with update time in the start and end time periods. Based on the second reliability enhancement strategy, with the input parameters as the baseline, the specified time steps are moved forward and backward respectively, and the input parameters are adjusted from 8:30 and 8:40 to 8:29 and 8:41 to optimize the input parameters.

[0073] In some embodiments, for the third interface type, the data acquisition protocol supported by the data interface is data that transmits version numbers corresponding to the input parameters. Since the data in the data source system have their own unique version numbers, the integrity and consistency of the data are guaranteed. Therefore, a reliability enhancement strategy is not configured for the data interface, and accordingly, the input parameters are not optimized.

[0074] In some embodiments, for the fourth interface type, the data acquisition protocol supported by the data interface is to transmit data with the same update time as the input parameters. Since the update time of the data in the data source system is unique, the data interface transmits data at the specified time each time, ensuring the integrity and consistency of the data. Therefore, a reliability enhancement strategy is not configured for the data interface, and accordingly, the input parameters are not optimized.

[0075] 203. Call the data interface according to the optimized input parameters to execute the data collection task.

[0076] In some embodiments, if communication failures, network connection interruptions, and other problems occur during data update in the data source system, the server may collect abnormal data. Accordingly, after collecting the data, the following process is also included: after collecting the data, the collected data is preliminarily detected to determine whether the collected data has abnormal conditions such as the primary key being empty. If the collected data has abnormal conditions, the incoming parameters are adjusted according to the content of the incremental field in the abnormal data to obtain new incoming parameters, or the content of the incremental field in the abnormal data is used as the new incoming parameter, and a new data collection task is generated based on the new incoming parameter. By executing the new data collection task, data is recollected from the data source system to complete the abnormal data.

[0077] For example, for the second interface type, three time points are specified as T A 、T B 、T C , and time point T B At time point T A and T C The input parameters of the data collection task are T Ais the starting time point, T C is the time period of the deadline, accordingly, the data collection task is used to collect T A to T C Data updated by the data source system within the time period.

[0078] After collecting the data, if the update time is T B to T C If the data within the time period has an abnormal situation such as the primary key being empty, the starting time point T in the input parameter of the data collection task will be A Adjust to T B , keep T C To keep the deadline unchanged, obtain new input parameters, and generate a new data collection task based on the new input parameters. The new data collection task is used to collect T B to T C The data updated in the time period is collected again from the data source system by executing the new data collection task. B to T C Data updated within the time period to complete abnormal data;

[0079] If only the update time is at time point T B If the data has abnormal conditions such as the primary key being empty, the input parameters of the data collection task are adjusted to the time point T B , based on the new input parameters, a new data collection task is generated. The new data collection task is used to collect data at time point T B Updated data is collected from the data source system at time point T by executing the new data collection task. B It should be noted that the interface type of the data interface for transmitting data is changed from the second interface type to the fourth interface type.

[0080] 204. Based on the interface type of the data interface, perform corresponding data integrity check on the collected data. If the integrity check passes, execute step 205. If the integrity check fails, re-execute the data collection task.

[0081] In some embodiments, the above-mentioned step 204 includes: based on the interface type of the data interface, obtaining the target data integrity verification scheme from the mapping relationship between the interface type and the data integrity verification scheme, and configuring the target data integrity verification scheme for the data interface, wherein the target data integrity verification scheme is the data integrity verification scheme corresponding to the interface type.

[0082] Different data integrity verification schemes are configured for different interface types. The following describes the data integrity verification schemes based on different interface types.

[0083] Case 1: The interface type is the third interface type, which refers to an interface with a version number as an increment field.

[0084] For the third interface type, since the data in the data source system has a unique version number corresponding to the data, the data integrity can be determined by checking whether the version number is missing. Accordingly, based on the interface type of the data interface, the corresponding data integrity check of the collected data includes: for the third interface type, after the server collects the data, it checks whether the version number corresponding to the data is missing. If the version numbers in the data are continuous and not missing, it means that the data is complete, the data integrity check passes, and step 205 is executed; if the version number in the data is missing, it means that the data is missing, the data integrity check fails, and the data collection task needs to be re-executed to collect the data corresponding to the missing version number.

[0085] In the second case, the interface type is the fifth interface type, which refers to a data interface that supports a check code.

[0086] For the fifth interface type, the data in the data source system already carries a checksum, which we call the first checksum. After the server collects the data, it recalculates the checksum for the data, which we call the second checksum. The first checksum is compared with the second checksum to determine whether the data has data integrity. Accordingly, based on the interface type of the data interface, performing a corresponding data integrity check on the collected data includes: for the fifth interface type, after the server collects the data, it calculates the second checksum for the data. If the second checksum is the same as the first checksum for the data, it indicates that the data is complete and the data integrity check passes, and step 205 is executed; if the second checksum is different from the first checksum for the data, it indicates that the data is missing and the data integrity check fails. The data collection task needs to be re-executed to collect data whose second checksum is different from the first checksum.

[0087] Through the method provided in step 204 above, different data integrity verification schemes are selected according to the interface type of the data interface to perform data integrity verification on the collected data, thereby ensuring data integrity, improving data quality, and enhancing data reliability.

[0088] 205. Compare the collected data with the data in the buffer area to remove duplicate data.

[0089] When executing the data collection task based on the above steps 201 - 203 , duplicate data may appear in the collected data, which will affect the efficiency of subsequent data processing. Therefore, the collected data must be compared with the data in the buffer area to remove duplicate data.

[0090] In some embodiments, the cache is used to temporarily store all data collected from various data source systems. Temporary storage refers to eliminating data stored in the cache based on an elimination algorithm. In some embodiments, the elimination algorithm is based on a first-in, first-out rule, and for each data source, the data last stored in the cache by that data source system will not be eliminated until the next time that data source system stores data in the cache.

[0091] In some embodiments, the size of the cache area can be dynamically adjusted according to the most recent data collection amount, and the data stored in the cache area is part of the data in the database.

[0092] In some embodiments, the above data deduplication includes the following two implementation methods:

[0093] The first implementation method: For the third interface type, since the data in the data source system has a unique version number corresponding to the data, it is possible to determine whether there is duplicate data based on the version number of the collected data. The above-mentioned step of comparing the collected data with the data in the cache area to remove duplicate data includes: comparing the version number of the collected data with the version number of the data in the cache area one by one; if the version number of the collected data is the same as the version number of the data in the cache area, removing the duplicate data in the collected data; if the version number of the collected data is different from the version number of the data in the cache area, retaining the data in the collected data.

[0094] The second implementation method: For other interface types, it is possible to determine whether there is duplicate data based on the primary key of the collected data, wherein the primary key refers to a field in the data that can uniquely identify the data, and the primary key is unique to the data. The above-mentioned step of comparing the collected data with the data in the cache area to remove duplicate data includes: comparing the primary key of the collected data with the primary key of the cache area data one by one, if the primary key of the collected data is the same as the primary key of the cache area data, removing the duplicate data in the collected data; if the primary key of the collected data is different from the primary key of the cache area data, retaining the data in the collected data.

[0095] 206. The deduplicated data is stored in a database, and the deduplicated data is stored in a cache area.

[0096] In some embodiments, after deduplication of the collected data, the deduplication data is stored in a database, and the deduplication data is stored in a cache area to prepare for the next data deduplication.

[0097] In an embodiment of the present application, by automatically configuring reliability enhancement strategies for data interfaces of different interface types, it is possible to ensure that no data is missed when executing data collection tasks, avoid data loss, and enhance data integrity. Moreover, for a data source with multiple copies, the above method can ensure the consistency of data between multiple copies of the data source. Furthermore, on the basis of data collection based on the reliability enhancement strategy, a data integrity check is performed on the collected data to further ensure the integrity of the data. The collected data is then deduplicated to improve data quality. The above methods are used in combination to ensure the reliability of the data.

[0098] Figure 4 This is a block diagram of a data acquisition device provided according to an embodiment of the present application. The device is used to perform the above-mentioned data acquisition method, see Figure 4 , the device comprises:

[0099] An incoming parameter optimization module 401 is configured to, in response to receiving incoming parameters of a data acquisition task, optimize the incoming parameters based on a target reliability enhancement strategy for the data interface, where the target reliability enhancement strategy is an incoming parameter processing method corresponding to the interface type to which the data interface belongs;

[0100] The data interface calling module 402 is used to call the data interface according to the optimized input parameters to execute the data collection task;

[0101] The data processing module 403 is used to process the collected data and store the processed data in a database.

[0102] In one possible implementation, the apparatus further includes:

[0103] The policy configuration module is used to configure a target reliability enhancement policy for the data interface based on the interface type of the data interface of the data management platform. The target reliability enhancement policy is an input parameter processing method corresponding to the interface type of the data interface.

[0104] In one possible implementation, the policy configuration module is configured to, for a first interface type, which refers to a data interface using natural time as an increment field, configure a first reliability enhancement policy for the data interface if the data acquisition protocol supported by the data interface is data whose update time is greater than the input parameter, wherein the first reliability enhancement policy is to subtract one time unit from the input parameter of the data acquisition task;

[0105] For the second interface type, the second interface type refers to a data interface with a start and end time period as an incremental field, and the data acquisition protocol supported by the data interface is data whose transmission update time is within the start and end time period. A second reliability enhancement strategy is configured for the data interface, and the second reliability enhancement strategy is to use the input parameters as a baseline and move forward and backward by specified time steps respectively.

[0106] In one possible implementation, the incoming parameter optimization module is configured to, for the first interface type, subtract one time unit from the incoming parameter of the data acquisition task if the data acquisition protocol supported by the data interface is data whose transmission update time is greater than the incoming parameter;

[0107] For the second interface type, the data acquisition protocol supported by the data interface is data whose update time is within the start and end time periods, and the input parameters are used as a baseline to move forward and backward by a specified time step respectively.

[0108] In one possible implementation, the data processing module is configured to perform a corresponding data integrity check on the collected data based on the interface type of the data interface; if the integrity check passes, a data deduplication step is performed; if the integrity check fails, the data collection task is re-executed;

[0109] The collected data is compared with the data in the buffer area to remove duplicate data.

[0110] In one possible implementation, the apparatus further includes:

[0111] A scheme configuration module is used to obtain a target data integrity verification scheme from a mapping relationship between the interface type and the data integrity verification scheme based on the interface type of the data interface, and configure the target data integrity verification scheme for the data interface, wherein the target data integrity verification scheme is the data integrity verification scheme corresponding to the interface type.

[0112] In one possible implementation, the data processing module is configured to, for a third interface type, which refers to a data interface using a version number as an increment field, compare the version number of the collected data with the version number of the data in the cache area one by one; if the version number of the collected data is the same as the version number of the data in the cache area, remove duplicate data from the collected data; if the version number of the collected data is different from the version number of the data in the cache area, retain the data in the collected data;

[0113] For other interface types, the primary key of the collected data is compared one by one with the primary key of the cache data. If the primary key of the collected data is the same as the primary key of the cache data, the duplicate data in the collected data is removed; if the primary key of the collected data is different from the primary key of the cache data, the data in the collected data is retained.

[0114] In an embodiment of the present application, by automatically configuring reliability enhancement strategies for data interfaces of different interface types, it is possible to ensure that no data is missed when executing data collection tasks, avoid data loss, and enhance data integrity. Moreover, for a data source with multiple copies, the above method can ensure the consistency of data between multiple copies of the data source. Furthermore, on the basis of data collection based on the reliability enhancement strategy, a data integrity check is performed on the collected data to further ensure the integrity of the data. The collected data is then deduplicated to improve data quality. The above methods are used in combination to ensure the reliability of the data.

[0115] It should be noted that the data acquisition device provided in the above embodiment is only illustrated by the division of the above functional modules when running an application program. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data acquisition device provided in the above embodiment and the data acquisition method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0116] Alternatively, the above data collection method can be performed as follows Figure 5 The architecture is implemented, Figure 5 It is a structural block diagram of the system architecture of a data acquisition method provided according to an embodiment of the present application, including a task scheduling unit 501 and a data acquisition unit 502, wherein the data acquisition unit 502 includes an interface identification unit 503, a reliability enhancement strategy application unit 504, a data integrity verification unit 505, a data deduplication unit 506, a data cache unit 507 and a data storage unit 508.

[0117] The task scheduling unit 501 is used for the server to schedule the data collection task according to a fixed scheduling rule to start the data collection task.

[0118] The interface identification unit 503 is used to identify the interface type of the data interface. The interface type is determined by automatic identification or manual selection. The specific process is shown in the above step 201.

[0119] The reliability enhancement strategy application unit 504 is used to configure a target reliability enhancement strategy for the data interface based on the interface type of the data interface of the data management platform, optimize the incoming parameters based on the target reliability enhancement strategy of the data interface, and call the data interface according to the optimized incoming parameters to execute the data collection task. The specific process is shown in the above steps 201-203.

[0120] The data integrity check unit 505 is used to perform corresponding data integrity check on the collected data based on the interface type of the data interface. If the integrity check passes, the above step 205 is executed. If the integrity check fails, the data collection task is re-executed. The specific process is shown in the above step 204.

[0121] The data deduplication unit 506 is used to compare the collected data with the data in the buffer area to remove duplicate data. The specific process is shown in the above step 205.

[0122] The data cache unit 507 is used to store the deduplicated data into a cache area in preparation for the next data deduplication.

[0123] The data storage unit 508 is used to store the deduplicated data into the database.

[0124] In the embodiment of the present application, the computer device can be configured as a server, and the server serves as the execution entity to implement the technical solution provided in the embodiment of the present application.

[0125] Figure 6 This is a schematic diagram of the structure of a server provided in accordance with an embodiment of the present application. The server 600 may vary significantly due to different configurations or performances, and may include one or more processors (Central Processing Units, CPUs) 601 and one or more memories 602, wherein the memories 602 store at least one computer program, which is loaded and executed by the processor 601 to implement the data acquisition methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The server may also include other components for implementing device functions, which will not be described in detail here.

[0126] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0127] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one computer program, which is executed by the processor 601 to implement the data acquisition method provided in the method embodiment of the present application.

[0128] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the computer device 600, and the computer device 600 may include more or fewer components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0129] The present application also provides a computer-readable storage medium that stores at least one computer program, which is loaded and executed by a processor of a computer device to implement the operations performed by the computer device in the method of the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0130] In some embodiments, the computer program involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected through a communication network. Multiple computer devices distributed at multiple locations and interconnected through a communication network may constitute a blockchain system.

[0131] The present application also provides a computer program product or computer program, which includes computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the data collection method provided in the various optional implementations described above.

[0132] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0133] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data collection method, characterized in that: The method comprises: Based on the interface type of the data interface of the data management platform, configuring a target reliability enhancement strategy for the data interface; In response to receiving the incoming parameters of the data acquisition task, optimizing the incoming parameters based on a target reliability enhancement strategy of the data interface, where the target reliability enhancement strategy is an incoming parameter processing method corresponding to the interface type to which the data interface belongs; According to the optimized input parameters, the data interface is called to execute the data collection task; Process the collected data and store the processed data in the database; The configuring of a target reliability enhancement strategy for the data interface based on the interface type of the data interface of the data management platform includes: For a first interface type, the first interface type refers to a data interface that uses natural time as an increment field. If the data acquisition protocol supported by the data interface is data that transmits update time greater than the input parameter, a first reliability enhancement strategy is configured for the data interface, wherein the first reliability enhancement strategy is to subtract one time unit from the input parameter of the data acquisition task. For the second interface type, the second interface type refers to a data interface with a start and end time period as an incremental field. If the data acquisition protocol supported by the data interface is data whose update time is within the start and end time period, a second reliability enhancement strategy is configured for the data interface. The second reliability enhancement strategy is to use the input parameters as a baseline and move forward and backward by specified time steps respectively.

2. The method according to claim 1, characterized in that In response to receiving the input parameters of the data acquisition task, optimizing the input parameters based on the target reliability enhancement strategy of the data interface includes: For the first interface type, if the data acquisition protocol supported by the data interface is data whose transmission update time is greater than the input parameter, then the input parameter of the data acquisition task is reduced by one time unit; For the second interface type, if the data acquisition protocol supported by the data interface is data whose update time is within the start and end time periods, the input parameters are used as a baseline and the specified time steps are moved forward and backward respectively.

3. The method according to claim 1, characterized in that The processing of the collected data includes: Based on the interface type of the data interface, performing a corresponding data integrity check on the collected data; if the integrity check passes, executing a data deduplication step; if the integrity check fails, re-executing the data collection task; The collected data is compared with the data in the buffer area to remove duplicate data.

4. The method according to claim 3, characterized in that The method further comprises: Based on the interface type of the data interface, a target data integrity verification scheme is obtained from a mapping relationship between the interface type and the data integrity verification scheme, and the target data integrity verification scheme is configured for the data interface, wherein the target data integrity verification scheme is the data integrity verification scheme corresponding to the interface type.

5. The method according to claim 3, characterized in that The comparing the collected data with the data in the buffer area to remove duplicate data includes: For the third interface type, the third interface type refers to a data interface with a version number as an incremental field, comparing the version number of the collected data with the version number of the data in the cache area one by one, and if the version number of the collected data is the same as the version number of the data in the cache area, removing duplicate data from the collected data; if the version number of the collected data is different from the version number of the data in the cache area, retaining the data in the collected data; For other interface types, the primary key of the collected data is compared one by one with the primary key of the cache data. If the primary key of the collected data is the same as the primary key of the cache data, the duplicate data in the collected data is removed; if the primary key of the collected data is different from the primary key of the cache data, the data in the collected data is retained.

6. A data acquisition device, characterized in that: The device comprises: A policy configuration module, configured to configure a target reliability enhancement policy for the data interface based on the interface type of the data interface of the data management platform; an incoming parameter optimization module, configured to, in response to receiving incoming parameters of a data acquisition task, optimize the incoming parameters based on a target reliability enhancement strategy for the data interface, wherein the target reliability enhancement strategy is an incoming parameter processing method corresponding to the interface type to which the data interface belongs; A data interface calling module, configured to call the data interface according to the optimized input parameters to execute the data collection task; A data processing module is used to process the collected data and store the processed data in a database; Among them, the policy configuration module is used for the first interface type, which refers to a data interface with natural time as the incremental field. If the data acquisition protocol supported by the data interface is data with a transmission update time greater than the input parameter, then the first reliability enhancement strategy is configured for the data interface, and the first reliability enhancement strategy is to subtract one time unit from the input parameter of the data acquisition task; for the second interface type, which refers to a data interface with a start and end time period as the incremental field, if the data acquisition protocol supported by the data interface is data with a transmission update time within the start and end time period, then the second reliability enhancement strategy is configured for the data interface, and the second reliability enhancement strategy is to use the input parameter as the baseline and move forward and backward by a specified time step respectively.

7. A computer device, characterized in that: The computer device includes a processor and a memory, the memory is used to store at least one computer program, and the at least one computer program is loaded by the processor to execute the data collection method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store at least one computer program, and the at least one computer program is used to execute the data collection method according to any one of claims 1 to 5.

Images