Multi-source data acquisition method and system and computer program

Abstract

The invention discloses a multi-source data acquisition method and system and a computer program, and the method comprises the steps: receiving a data reporting request sent by a data source, and carrying out the processing and scheduling of the obtained data reporting request; according to the scheduling arrangement of the data reporting request, receiving original data sent by a data source; checking the original data; according to a preset data source attribute, selecting a proper adapter from preset adapters to standardize the original data to obtain standard data; and storing the standard data in a database. According to the technical scheme provided by the invention, the data acquired by various heterogeneous data sources are scheduled and received, and the appropriate adapter is selected for standardized processing, so that a data acquisition technical scheme which does not need to be redeveloped and deployed and naturally supports multiple data sources is realized; according to the scheme, the pre-developed adapter can be utilized to automatically adapt to the data source for processing, so that the data acquisition process is greatly simplified, and the data acquisition efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data collection, and in particular to a multi-source data collection method, system and computer program. Background Art

[0002] Data collection is the foundation of informatization and the key to continuously improving the informatization and automation level of enterprises. At the same time, with the continuous enrichment of production and life scenarios, the complexity of data collection is also increasing.

[0003] In the field of data collection, the technical solutions currently used are data collection systems based on a single data source or simple timed task schedulers. Although these solutions can meet certain data collection needs, they have obvious shortcomings. They are not friendly to the access of multiple data sources, and the deployment is complicated. Each new data source must be redeveloped and deployed. It is impossible to quickly and effectively schedule and process heterogeneous data from multiple different data sources, and there are limitations in the real-time and accuracy of data collection. In addition, these solutions also lack sufficient flexibility and scalability, and it is difficult to adapt to the rapidly changing data collection needs and environment. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the current technology, the present invention provides a multi-source data collection method, which schedules and receives data collected from multiple data sources, and selects appropriate adapters for standardized processing, without the need for redevelopment and deployment, and naturally supports multiple data sources.

[0005] To achieve the above object, the embodiments of the present invention adopt the following technical solutions: A multi-source data collection method comprises the following steps: Receive data reporting requests sent by data sources, and process and schedule the obtained data reporting requests; Receive the original data sent by the data source according to the scheduling arrangement of the data reporting request; Verify the original data; According to the preset data source attributes, select a suitable adapter from the preset adapters to standardize the original data to obtain standard data; Store standard data in a database; The processing and scheduling of the obtained data reporting request includes: Dynamically adjust the priority of data reporting requests from data sources based on preset priority rules, characteristics of data sources, and statistical records of historical reported data; The step of selecting a suitable adapter from preset adapters to standardize the raw data includes: The machine learning-based quality assessment model dynamically evaluates data quality during the standardization process and automatically adjusts adapter parameters or prompts manual intervention based on the assessment results.

[0006] According to one aspect of the present invention, the verifying of the original data includes: verifying the format, integrity and legality of the original data according to preset verification rules.

[0007] According to one aspect of the present invention, the data source attributes are set via a configuration file.

[0008] According to one aspect of the present invention, the data source attributes include a data source identifier, a data source type, and a requirement field.

[0009] According to one aspect of the present invention, the data source types include: relational databases, NoSQL databases, APIs, and file systems.

[0010] According to one aspect of the present invention, the multi-source data collection method further includes: Generic adapters can be pre-developed based on commonly used data source types.

[0011] According to one aspect of the present invention, the multi-source data collection method further includes: When a new data source needs to be connected, if there is an adapter that is compatible with the data source type among the existing adapters, the data source can be directly connected; if there is no adapter that is compatible with the data source type among the existing adapters, the corresponding adapter developed according to the data source type is used to connect to the data source.

[0012] According to one aspect of the present invention, the standard data is in Json format.

[0013] A multi-source data acquisition system, based on the multi-source data acquisition method as described above, comprises: The scheduling module is used to receive the data reporting request sent by the data source, and process and schedule the obtained data reporting request; A receiving module, used to receive the original data sent by the data source according to the scheduling order of the data reporting request; A verification module, used for verifying the original data; A standardization module is used to select a suitable adapter from preset adapters to standardize the original data according to preset data source attributes to obtain standard data; The storage module is used to store standard data in the database.

[0014] A computer program, which implements the steps of the multi-source data acquisition method as described above when executed.

[0015] Advantages of the present invention: The present invention provides a multi-source data collection method, which schedules and receives data collected from multiple heterogeneous data sources, and selects appropriate adapters for standardized processing, thereby realizing a data collection technical solution that does not require redevelopment and deployment and naturally supports multiple data sources; the solution can use pre-developed adapters to automatically adapt data sources for processing, greatly simplifying the data collection process and improving the efficiency of data collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A flowchart of a multi-source data acquisition method according to the present invention; Figure 2 This is an architecture diagram of a multi-source data acquisition method described in the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Embodiment 1

[0020] like Figure 1 and 2 As shown, a multi-source data collection method comprises the following steps: S1: Receive data reporting requests sent by data sources, and process and schedule the obtained data reporting requests.

[0021] The processing and scheduling of the obtained data reporting request includes: dynamically adjusting the priority of the data reporting request of the data source according to preset priority rules, characteristics of the data source, and statistical records of historical reported data.

[0022] In the specific implementation, this method is implemented through the data acquisition system, and the API gateway, as the first entry point of the entire system, undertakes the important task of receiving and processing reporting requests from different data sources. The responsibilities of the API gateway are not limited to receiving requests, but also to perform preliminary processing and scheduling of these requests to ensure that the system can efficiently respond to the reporting needs of various data sources. During the processing, the system will dynamically adjust the priority of the data source's data reporting request according to the preset priority rules, the characteristics of the data source, and the statistical records of historical reported data, reasonably allocate resources, and optimize the process of data reception and processing. For example, the higher the preset priority, the higher the actual priority; the more urgent and critical the data source report, the higher the priority; the higher the priority is when the historical reporting record is abnormal; you can also set weights for data sources, comprehensively calculate priority scores, and prioritize them according to scores.

[0023] In addition, intelligent algorithms can be used to dynamically adjust the priority of data processing to adapt to changes in data traffic in different scenarios, thereby improving the data processing capabilities and response speed of the entire system. For example, a decision tree can be built in advance based on historical data reporting, and the decision tree can be used to obtain the priority ranking of data sources in real time based on the current data source situation.

[0024] In addition, the system can also monitor the real-time status of data reporting requests to ensure the continuity and stability of data processing. Through such a design, this method can effectively process a large amount of data from different data sources and ensure the efficient and stable operation of the system.

[0025] S2: Receive the original data sent by the data source according to the scheduling arrangement of the data reporting request.

[0026] S3: Verify the original data.

[0027] Step S3 includes: verifying the format, integrity, legality, etc. of the original data according to preset verification rules.

[0028] The raw data collected by the data source often has various problems, and it is necessary to conduct a comprehensive check on the raw data to ensure that its format is correct, its content is complete, and it complies with all relevant legal requirements. The quality and integrity of the data directly affect the results and efficiency of subsequent processing, so it is essential to verify the data in all aspects.

[0029] S4: According to the preset data source attributes, select a suitable adapter from the preset adapters to standardize the original data to obtain standard data.

[0030] The step of selecting a suitable adapter from preset adapters to standardize the raw data includes: Machine learning-based quality assessment models, such as logistic regression, support vector machine (SVM), random forest, etc., dynamically evaluate data quality during the standardization process and automatically adjust adapter parameters or prompt manual intervention based on the evaluation results.

[0031] The role of the adapter is to convert and process the data according to the characteristics and requirements of the data, including but not limited to format adjustment, missing value filling, outlier detection and correction, etc., to convert the original data into a unified standard format data. In practical applications, standard data is generally in json format for easy interpretation and analysis.

[0032] Preferably, the data source attributes are set via a configuration file.

[0033] The data source attributes include data source identification, data source type, and required fields, etc.

[0034] The data source types include: relational databases, NoSQL databases, APIs, file systems, etc. Identify the data source type and select a suitable adapter to process the raw data.

[0035] In the specific implementation, the parameters in the configuration file are set as follows: dataSources: An array containing the configuration of all data sources.

[0036] name: The name of the data source, used for identification.

[0037] type: The type of data source, such as relational database, NoSQL database, API, or file system.

[0038] connectionString / endpoint / apiUrl / path: Depending on the type of data source, provide a connection string, endpoint URL, API URL, or file path.

[0039] fields: An object that defines the type, description, required or not, and other constraints (such as maximum length, minimum value, format) of the fields in the data source.

[0040] validationRules: defines the rules for data validation, which can be simple text descriptions or more complex validation logic.

[0041] In the specific implementation, the processed standard data in json format is as follows: { "metaData": { "requestId": "unique-request-id", "timestamp": "2024-06-06T12:00:00Z", "dataSource": "source-identifier", "dataSourceType": "type-of-datasource" }, "payload": { "data": { "recordId": "unique-record-id", "attributes": { / / Define attributes here according to data types and business requirements } } / / There can be more records }, "validation": { "status": "valid / invalid", "messages": "validation-message-1", / / There can be more verification messages } } S5: Store the standard data in a database.

[0042] Preferably, the multi-source data collection method further includes: Generic adapters can be pre-developed based on commonly used data source types.

[0043] Preferably, the multi-source data collection method further includes: When a new data source needs to be accessed, if there is an adapter that is compatible with the data source type among the existing adapters, the data source can be directly accessed.

[0044] In actual applications, the raw data collected and reported by the data collectors developed by various manufacturers generally adopt some common formats or meet the industry specifications. Therefore, some general adapters can be developed in advance according to various common or industry-specified data source types to facilitate access to different data sources. These adapters can support multiple data formats and protocols, thereby ensuring seamless integration and use in various application scenarios.

[0045] In specific implementation, the adapter mode can be used to develop data source adapters to provide a unified access interface for different types of data sources (such as relational databases, NoSQL databases, APIs, file systems, etc.). In this way, when a new data source is connected, only the data type and field description need to be selected to complete data collection and verification without the need for redevelopment and deployment. In this way, the data integration process can be greatly simplified, development efficiency can be improved, and system complexity can be reduced.

[0046] The beneficial effects of this embodiment are as follows: this method can schedule the reception of data collected from a variety of heterogeneous data sources, and select appropriate adapters for standardized processing, thereby realizing a data collection technology solution that does not require redevelopment and deployment and naturally supports multiple data sources; this solution can use pre-developed adapters to automatically adapt data sources for processing, greatly simplifying the data collection process and improving the efficiency of data collection.

[0047] Embodiment 2

[0048] like Figure 1 and 2 As shown, a multi-source data collection method comprises the following steps: S1: Receive data reporting requests sent by data sources, and process and schedule the obtained data reporting requests.

[0049] The processing and scheduling of the obtained data reporting request includes: dynamically adjusting the priority of the data reporting request of the data source according to preset priority rules, characteristics of the data source, and statistical records of historical reported data.

[0050] In the specific implementation, this method is implemented through the data acquisition system, and the API gateway, as the first entry point of the entire system, undertakes the important task of receiving and processing reporting requests from different data sources. The responsibilities of the API gateway are not limited to receiving requests, but also to perform preliminary processing and scheduling of these requests to ensure that the system can efficiently respond to the reporting needs of various data sources. During the processing, the system will dynamically adjust the priority of the data source's data reporting request according to the preset priority rules, the characteristics of the data source, and the statistical records of historical reported data, reasonably allocate resources, and optimize the process of data reception and processing. For example, the higher the preset priority, the higher the actual priority; the more urgent and critical the data source report, the higher the priority; the higher the priority is when the historical reporting record is abnormal; you can also set weights for data sources, comprehensively calculate priority scores, and prioritize them according to scores.

[0051] In addition, intelligent algorithms can be used to dynamically adjust the priority of data processing to adapt to changes in data traffic in different scenarios, thereby improving the data processing capabilities and response speed of the entire system. For example, a decision tree can be built in advance based on historical data reporting, and the decision tree can be used to obtain the priority ranking of data sources in real time based on the current data source situation.

[0052] In addition, the system can also monitor the real-time status of data reporting requests to ensure the continuity and stability of data processing. Through such a design, this method can effectively process a large amount of data from different data sources and ensure the efficient and stable operation of the system.

[0053] S2: Receive the original data sent by the data source according to the scheduling arrangement of the data reporting request.

[0054] S3: Verify the original data.

[0055] Step S3 includes: verifying the format, integrity, legality, etc. of the original data according to preset verification rules.

[0056] The raw data collected by the data source often has various problems, and it is necessary to conduct a comprehensive check on the raw data to ensure that its format is correct, its content is complete, and it complies with all relevant legal requirements. The quality and integrity of the data directly affect the results and efficiency of subsequent processing, so it is essential to verify the data in all aspects.

[0057] S4: According to the preset data source attributes, select a suitable adapter from the preset adapters to standardize the original data to obtain standard data.

[0058] The step of selecting a suitable adapter from preset adapters to standardize the raw data includes: Machine learning-based quality assessment models, such as logistic regression, support vector machine (SVM), random forest, etc., dynamically evaluate data quality during the standardization process and automatically adjust adapter parameters or prompt manual intervention based on the evaluation results.

[0059] The role of the adapter is to convert and process the data according to the characteristics and requirements of the data, including but not limited to format adjustment, missing value filling, outlier detection and correction, etc., to convert the original data into a unified standard format data. In practical applications, standard data is generally in json format for easy interpretation and analysis.

[0060] Preferably, the data source attributes are set via a configuration file.

[0061] The data source attributes include data source identification, data source type, and required fields, etc.

[0062] The data source types include: relational databases, NoSQL databases, APIs, file systems, etc. Identify the data source type and select a suitable adapter to process the raw data.

[0063] In the specific implementation, the parameters in the configuration file are set as follows: dataSources: An array containing the configuration of all data sources.

[0064] name: The name of the data source, used for identification.

[0065] type: The type of data source, such as relational database, NoSQL database, API, or file system.

[0066] connectionString / endpoint / apiUrl / path: Depending on the type of data source, provide a connection string, endpoint URL, API URL, or file path.

[0067] fields: An object that defines the type, description, required or not, and other constraints (such as maximum length, minimum value, format) of the fields in the data source.

[0068] validationRules: defines the rules for data validation, which can be simple text descriptions or more complex validation logic.

[0069] In the specific implementation, the processed standard data in json format is as follows: { "metaData": { "requestId": "unique-request-id", "timestamp": "2024-06-06T12:00:00Z", "dataSource": "source-identifier", "dataSourceType": "type-of-datasource" }, "payload": { "data": { "recordId": "unique-record-id", "attributes": { / / Define attributes here according to data types and business requirements } } / / There can be more records }, "validation": { "status": "valid / invalid", "messages": "validation-message-1", / / There can be more verification messages } } S5: Store the standard data in a database.

[0070] Preferably, the multi-source data collection method further includes: Generic adapters can be pre-developed based on commonly used data source types.

[0071] Preferably, the multi-source data collection method further includes: When a new data source needs to be connected, if there is an adapter that is compatible with the data source type among the existing adapters, the data source can be directly connected; if there is no adapter that is compatible with the data source type among the existing adapters, the corresponding adapter developed according to the data source type is used to connect to the data source.

[0072] In actual applications, the raw data collected and reported by the data collectors developed by various manufacturers generally adopt some common formats or meet the industry specifications. Therefore, some general adapters can be developed in advance according to various common or industry-specified data source types to facilitate access to different data sources. These adapters can support multiple data formats and protocols, thereby ensuring seamless integration and use in various application scenarios.

[0073] In specific implementation, the adapter mode can be used to develop data source adapters to provide a unified access interface for different types of data sources (such as relational databases, NoSQL databases, APIs, file systems, etc.). In this way, when a new data source is connected, only the data type and field description need to be selected to complete data collection and verification without the need for redevelopment and deployment. In this way, the data integration process can be greatly simplified, development efficiency can be improved, and system complexity can be reduced.

[0074] Preferably, the multi-source data collection method further includes: The database is connected to multiple business parties to allow different external business parties to read data.

[0075] The beneficial effects of this embodiment are: this method provides a unified interface for the data source, which is convenient for accessing multiple data sources and adapting appropriate adapters to process the data; and when there is no suitable adapter, a corresponding adapter can also be developed for processing according to the data source type.

[0076] Embodiment 3

[0077] A multi-source data acquisition system, based on the multi-source data acquisition method as described in Embodiment 1 or 2, comprises: The scheduling module is used to receive the data reporting request sent by the data source, and process and schedule the obtained data reporting request; A receiving module, used to receive the original data sent by the data source according to the scheduling order of the data reporting request; A verification module, used for verifying the original data; A standardization module is used to select a suitable adapter from preset adapters to standardize the original data according to preset data source attributes to obtain standard data; The storage module is used to store standard data in the database.

[0078] Embodiment 4

[0079] A computer program, when executed, implements the steps of the multi-source data acquisition method as described in embodiment 1 or 2.

[0080] Embodiment 5

[0081] A readable storage medium stores a computer program as described in Example 4, and when the computer program is executed, the steps of the multi-source data acquisition method as described in Example 1 or 2 are implemented.

[0082] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multi-source data collection method, characterized in that: The following steps are involved: Receive data reporting requests sent by data sources, and process and schedule the obtained data reporting requests; Receive the original data sent by the data source according to the scheduling arrangement of the data reporting request; Verify the original data; According to the preset data source attributes, select a suitable adapter from the preset adapters to standardize the original data to obtain standard data; Store standard data in a database; The processing and scheduling of the obtained data reporting request includes: Dynamically adjust the priority of data reporting requests from data sources based on preset priority rules, characteristics of data sources, and statistical records of historical reported data; The step of selecting a suitable adapter from preset adapters to standardize the raw data includes: The machine learning-based quality assessment model dynamically evaluates data quality during the standardization process and automatically adjusts adapter parameters or prompts manual intervention based on the assessment results.

2. The multi-source data acquisition method according to claim 1, characterized in that: The verifying of the original data includes: verifying the format, integrity and legality of the original data according to preset verification rules.

3. The multi-source data acquisition method according to claim 1, characterized in that: The data source properties are set through a configuration file.

4. The multi-source data acquisition method according to claim 1, characterized in that: The data source attributes include a data source identifier, a data source type, and a requirement field.

5. The multi-source data acquisition method according to claim 4, characterized in that: The data source types include: relational databases, NoSQL databases, APIs, and file systems.

6. The multi-source data acquisition method according to claim 5, characterized in that: The multi-source data collection method further comprises: Generic adapters can be pre-developed based on commonly used data source types.

7. The multi-source data acquisition method according to claim 6, characterized in that: The multi-source data collection method further comprises: When a new data source needs to be connected, if there is an adapter that is compatible with the data source type among the existing adapters, the data source can be directly connected; if there is no adapter that is compatible with the data source type among the existing adapters, the corresponding adapter developed according to the data source type is used to connect to the data source.

8. The multi-source data acquisition method according to claim 1, characterized in that: The standard data is in Json format.

9. A multi-source data acquisition system, characterized in that: The multi-source data acquisition method according to any one of claims 1 to 8 comprises: The scheduling module is used to receive the data reporting request sent by the data source, and process and schedule the obtained data reporting request; A receiving module, used to receive the original data sent by the data source according to the scheduling order of the data reporting request; A verification module, used for verifying the original data; A standardization module is used to select a suitable adapter from preset adapters to standardize the original data according to preset data source attributes to obtain standard data; The storage module is used to store standard data in the database.

10. A computer program, characterized in that When the computer program is executed, the steps of the multi-source data acquisition method according to any one of claims 1 to 8 are implemented.

Images