Data collection method, device, equipment and medium for industrial multi-source heterogeneous data sources

Abstract

The present application provides a data collection method, device, equipment and medium for industrial multi-source heterogeneous data sources. The method according to the present application includes: determining the business function module corresponding to the current industrial application platform, determining the multi-level storage structure corresponding to the current industrial application platform, generating corresponding data collection tasks according to the change data and the business demand data corresponding to the current industrial application platform; assigning the data collection tasks to the corresponding processing threads; collecting multi-source heterogeneous data according to the data collection protocol corresponding to the data interface; storing the cleaned multi-source heterogeneous data in the corresponding database based on the data type corresponding to the multi-source heterogeneous data, and completing the collection of the multi-source heterogeneous data. The technical solution provided by the present application can realize the collection of industrial multi-source heterogeneous data.

Description

Technical Field

[0001] This document relates to the field of data acquisition technology, and in particular to a data acquisition method, device, equipment and medium for industrial multi-source heterogeneous data sources. Background Art

[0002] The equipment, products, and application systems in industrial enterprises continuously generate massive amounts of data, and there is an urgent need for analytical insights and value creation of industrial data. It is urgent to achieve extensive connection, unified management, and application development of equipment assets, transform industrial data into valuable digital services for industrial enterprises, release the potential of industrial data, and improve the efficiency of industrial resource management. Building an industrial operating system is an important way for enterprises to reduce costs, improve efficiency, reduce environmental pollution, effectively utilize resources, and respond quickly to decisions. The basic support for the extensive connection of industrial resources is diverse in industrial industry equipment, protocols, and communication standards, which brings severe challenges to the connection of industrial equipment assets and data assets. Summary of the invention

[0003] One or more embodiments of the present specification provide a method, device, equipment and medium for collecting data from industrial multi-source heterogeneous data sources to solve the problem of collecting data from industrial multi-source heterogeneous data sources.

[0004] The embodiment of the present invention provides a data collection method for industrial multi-source heterogeneous data sources, comprising:

[0005] S1. Determine the business function module corresponding to the current industrial application platform, store the business data corresponding to each business function module in layers according to the business type corresponding to each business function module, and determine the multi-level storage structure corresponding to the current industrial application platform;

[0006] S2. acquiring the change data of the multi-level storage structure in real time, so as to generate corresponding data collection tasks according to the change data and the business demand data corresponding to the current industrial application platform;

[0007] S3. Allocate the data collection task to the corresponding processing thread according to the thread parameters of each processing thread corresponding to the current industrial application platform;

[0008] S4, sequentially acquiring the data interface corresponding to each data collection task in the processing thread, so as to collect multi-source heterogeneous data according to the data collection protocol corresponding to the data interface;

[0009] S5. Clean the acquired multi-source heterogeneous data according to a preset standard format;

[0010] S6. Based on the data types corresponding to the multi-source heterogeneous data, the cleaned multi-source heterogeneous data are stored in a corresponding database to complete the collection of the multi-source heterogeneous data; wherein the corresponding database includes: a relational database and a time series database.

[0011] An embodiment of the present invention provides a data acquisition device for industrial multi-source heterogeneous data sources, comprising:

[0012] Determine a storage structure module, used to determine the business function module corresponding to the current industrial application platform, so as to hierarchically store the business data corresponding to each business function module according to the business type corresponding to each business function module, and determine the multi-level storage structure corresponding to the current industrial application platform;

[0013] A collection task generation module, used for acquiring the change data of the multi-level storage structure in real time, so as to generate corresponding data collection tasks according to the change data and the business demand data corresponding to the current industrial application platform;

[0014] A thread generation module, used to allocate the data collection task to the corresponding processing thread according to the thread parameters of each processing thread corresponding to the current industrial application platform;

[0015] A data acquisition module, used to sequentially acquire the data interface corresponding to each data acquisition task in the processing thread, so as to acquire multi-source heterogeneous data according to the data acquisition protocol corresponding to the data interface;

[0016] The data cleaning module is used to clean the acquired multi-source heterogeneous data according to the preset standard format;

[0017] A data storage module is used to store the cleaned multi-source heterogeneous data in a corresponding database based on the data types corresponding to the multi-source heterogeneous data, thereby completing the collection of the multi-source heterogeneous data; wherein the corresponding database includes: a relational database and a time series database.

[0018] An embodiment of the present invention further provides an electronic device, including:

[0019] processor; and,

[0020] A memory is arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps of the above-mentioned industrial multi-source heterogeneous data source data collection method.

[0021] An embodiment of the present invention further provides a storage medium for storing computer executable instructions, wherein the computer executable instructions, when executed, implement the steps of the above-mentioned method for collecting data from industrial multi-source heterogeneous data sources.

[0022] By adopting the embodiment of the present invention, corresponding data collection tasks are generated according to business demand data, and data collection is achieved by setting the data source system, setting the data collection connection method, setting the scheduling rules, setting the target data type, data format and data content, setting the data content verification rules, and setting the data collection trigger method of the target data. The collected data is cleaned and stored in the corresponding database, so that when a new data source is connected, only corresponding configuration needs to be performed to achieve the collection of the new data source. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 A flowchart of a method for collecting data from industrial multi-source heterogeneous data sources according to an embodiment of the present invention;

[0025] Figure 2 It is a schematic diagram of a data acquisition device for industrial multi-source heterogeneous data sources according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0027] Method Embodiment

[0028] According to an embodiment of the present invention, a data collection method for industrial multi-source heterogeneous data sources is provided. Figure 1 Flow chart of the data collection method for industrial multi-source heterogeneous data sources according to an embodiment of the present invention. Figure 1 As shown, the data collection method for industrial multi-source heterogeneous data sources in an embodiment of the present invention specifically includes:

[0029] S1. Determine the business function module corresponding to the current industrial application platform, store the business data corresponding to each business function module in layers according to the business type corresponding to each business function module, and determine the multi-level storage structure corresponding to the current industrial application platform;

[0030] Identification of business function modules: First, we need to conduct a comprehensive business analysis of the industrial application platform to identify all business function modules. We understand each business process through workflow diagrams, user stories, and other methods. Based on the business process, we divide the platform into several function modules, such as production management, quality management, supply chain management, equipment maintenance, etc.

[0031] Business type analysis: For each business function module, analyze its corresponding business type, which may include:

[0032] Transactional data: such as order processing and inventory changes.

[0033] Analytical data: such as production efficiency analysis and market trend analysis.

[0034] Real-time data: such as equipment status monitoring and production progress tracking.

[0035] Historical data: such as production records, maintenance logs.

[0036] S2, acquiring the change data of the multi-level storage structure in real time, so as to generate corresponding data collection tasks according to the change data and the business demand data corresponding to the current industrial application platform; S2 specifically includes:

[0037] The generating of the corresponding data collection task includes: setting the data source system and data collection parameters of the data collection task. The setting of the data source system and data collection parameters of the data collection task specifically includes:

[0038] Set up the data source system and configure the business systems of all data sources;

[0039] Set the docking mode for data collection and configure the corresponding interface parameters according to different docking modes;

[0040] Set scheduling rules, including: support for task priority sorting, support for weighting deadlines and turnaround time criteria;

[0041] Set the target data type, data format and data content;

[0042] Set data content verification rules;

[0043] Set the data collection trigger mode of the target data, including event triggering or timed collection. The event triggering mode configures the associated events of data collection, and collects the corresponding business result data immediately and synchronously when the associated event processing is completed; the timed collection mode sets the timed collection mode, including setting the data collection time and collection frequency.

[0044] S3. Allocate the data collection task to the corresponding processing thread according to the thread parameters of each processing thread corresponding to the current industrial application platform;

[0045] The S3 specifically includes:

[0046] Load the data collection plan task parameters;

[0047] Parse the trigger mode configuration information in the data collection plan task parameters and assemble batch processing scheduling rules;

[0048] Initialize the task scheduling module;

[0049] The single data collection task is verified. If the trigger condition is met, step S4 is executed.

[0050] S4, sequentially obtaining the data interface corresponding to each data collection task in the processing thread, so as to collect multi-source heterogeneous data according to the data collection protocol corresponding to the data interface; S4 specifically includes:

[0051] Analyze data connection mode, interface parameters and target data type, and assemble target data interface parameters;

[0052] Calling a target data interface to obtain response data of the target data interface;

[0053] Analyze the target data format and data content, parse and extract the target data.

[0054] S5. Clean the acquired multi-source heterogeneous data according to a preset standard format; S5 specifically includes:

[0055] Multi-source heterogeneous data cleaning includes missing value cleaning, format content cleaning, and logical error cleaning.

[0056] S6. Based on the data types corresponding to the multi-source heterogeneous data, the cleaned multi-source heterogeneous data are stored in a corresponding database to complete the collection of the multi-source heterogeneous data; wherein the corresponding database includes: a relational database and a time series database.

[0057] The embodiments of the present invention have the following beneficial effects:

[0058] Generate corresponding data collection tasks based on business demand data, and collect data by setting the data source system, setting the data collection connection method, setting the scheduling rules, setting the target data type, data format and data content, setting the data content verification rules, and setting the data collection trigger method of the target data. After cleaning the collected data, store it in the corresponding database, so that when accessing a new data source, you only need to perform corresponding configuration to realize the collection of the new data source.

[0059] Device Example 1

[0060] According to an embodiment of the present invention, a data acquisition device for industrial multi-source heterogeneous data sources is provided. Figure 2 Schematic diagram of a data acquisition device for industrial multi-source heterogeneous data sources according to an embodiment of the present invention. Figure 2 As shown, the data acquisition device for industrial multi-source heterogeneous data sources according to the embodiment of the present invention specifically includes:

[0061] A storage structure determination module 20 is used to determine the business function module corresponding to the current industrial application platform, so as to hierarchically store the business data corresponding to each business function module according to the business type corresponding to each business function module, and determine the multi-level storage structure corresponding to the current industrial application platform;

[0062] The acquisition task generation module 21 is used to obtain the change data of the multi-level storage structure in real time, so as to generate corresponding data acquisition tasks according to the change data and the business demand data corresponding to the current industrial application platform;

[0063] A thread generation module 22, configured to allocate the data collection task to a corresponding processing thread according to thread parameters of each processing thread corresponding to the current industrial application platform;

[0064] The data acquisition module 23 is used to sequentially acquire the data interface corresponding to each data acquisition task in the processing thread, so as to acquire multi-source heterogeneous data according to the data acquisition protocol corresponding to the data interface;

[0065] The data cleaning module 24 is used to clean the acquired multi-source heterogeneous data according to a preset standard format;

[0066] The data storage module 25 is used to store the cleaned multi-source heterogeneous data into a corresponding database based on the data type corresponding to the multi-source heterogeneous data, so as to complete the collection of the multi-source heterogeneous data; wherein the corresponding database includes: a relational database and a time series database.

[0067] Device Example 2

[0068] According to an embodiment of the present invention, an electronic device is provided, including:

[0069] processor; and,

[0070] A memory is arranged to store computer executable instructions, which when executed cause the processor to perform the steps of the above method embodiments.

[0071] Device Example 3

[0072] A storage medium is used to store computer executable instructions, and the computer executable instructions, when executed, implement the steps of the above method embodiment.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data collection method for industrial multi-source heterogeneous data sources, characterized in that include: S1. Determine the business function module corresponding to the current industrial application platform, store the business data corresponding to each business function module in layers according to the business type corresponding to each business function module, and determine the multi-level storage structure corresponding to the current industrial application platform; S2. acquiring the change data of the multi-level storage structure in real time, so as to generate corresponding data collection tasks according to the change data and the business demand data corresponding to the current industrial application platform; S3. Allocate the data collection task to the corresponding processing thread according to the thread parameters of each processing thread corresponding to the current industrial application platform; S4, sequentially acquiring the data interface corresponding to each data collection task in the processing thread, so as to collect multi-source heterogeneous data according to the data collection protocol corresponding to the data interface; S5. Clean the acquired multi-source heterogeneous data according to a preset standard format; S6. Based on the data types corresponding to the multi-source heterogeneous data, the cleaned multi-source heterogeneous data are stored in a corresponding database to complete the collection of the multi-source heterogeneous data; wherein the corresponding database includes: a relational database and a time series database.

2. The method according to claim 1, characterized in that: The generating of the corresponding data collection task includes: setting the data source system and data collection parameters of the data collection task.

3. The method according to claim 2, characterized in that The data source system and data collection parameters for setting the data collection task specifically include: Set up the data source system and configure the business systems of all data sources; Set the docking mode for data collection and configure the corresponding interface parameters according to different docking modes; Set scheduling rules, including: support for task priority sorting, support for weighting deadlines and turnaround time criteria; Set the target data type, data format and data content; Set data content verification rules; Set the data collection trigger mode of the target data, including event triggering or timed collection.

4. The method according to claim 3, characterized in that: The event triggering mode configures the associated events of data collection, and collects the corresponding business result data immediately and synchronously when the associated event processing is completed; the scheduled collection sets the scheduled collection mode, including setting the data collection time and collection frequency.

5. The method according to claim 1, characterized in that The S3 specifically includes: Load the data collection plan task parameters; Parse the trigger mode configuration information in the data collection plan task parameters and assemble batch processing scheduling rules; Initialize the task scheduling module; The single data collection task is verified. If the trigger condition is met, step S4 is executed.

6. The method according to claim 3, characterized in that: The S4 specifically includes: Analyze data connection mode, interface parameters and target data type, and assemble target data interface parameters; Calling a target data interface to obtain response data of the target data interface; Analyze the target data format and data content, parse and extract the target data.

7. The method according to claim 1, characterized in that: The S5 specifically includes: Multi-source heterogeneous data cleaning includes missing value cleaning, format content cleaning, and logical error cleaning.

8. A data acquisition device for industrial multi-source heterogeneous data sources, comprising: Determine a storage structure module, used to determine the business function module corresponding to the current industrial application platform, so as to hierarchically store the business data corresponding to each business function module according to the business type corresponding to each business function module, and determine the multi-level storage structure corresponding to the current industrial application platform; A collection task generation module, used for acquiring the change data of the multi-level storage structure in real time, so as to generate corresponding data collection tasks according to the change data and the business demand data corresponding to the current industrial application platform; A thread generation module, used for allocating the data collection task to the corresponding processing thread according to the thread parameters of each processing thread corresponding to the current industrial application platform; A data acquisition module, used to sequentially acquire the data interface corresponding to each data acquisition task in the processing thread, so as to acquire multi-source heterogeneous data according to the data acquisition protocol corresponding to the data interface; The data cleaning module is used to clean the acquired multi-source heterogeneous data according to the preset standard format; A data storage module is used to store the cleaned multi-source heterogeneous data in a corresponding database based on the data types corresponding to the multi-source heterogeneous data, thereby completing the collection of the multi-source heterogeneous data; wherein the corresponding database includes: a relational database and a time series database.

9. An electronic device, comprising: processor; as well as, A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps of the data collection method for industrial multi-source heterogeneous data sources as claimed in any one of claims 1 to 8.

10. A storage medium for storing computer executable instructions, wherein the computer executable instructions, when executed, implement the steps of the data collection method for industrial multi-source heterogeneous data sources as described in any one of claims 1 to 8.

Images