Identifier-based data acquisition system and data analysis method

Abstract

The invention relates to the technical field of data acquisition and processing, and discloses an identifier-based data acquisition system, which comprises a system module assembly, and is characterized in that the system module assembly comprises a gateway configuration module, a metadata setting module, a data management module and a gateway identification module; the invention further provides an identification-based data analysis method which comprises the following steps: S1, gateway identification: identifying the gateway by utilizing identification information, reading equipment operation data and identification data from the gateway, and permanently recording equipment operation core data by utilizing an identification system; and S2, collecting messages. According to the method, the accuracy and the consistency of the data are ensured through an accurate identification technology, a solid foundation is provided for data analysis and decision making, secondly, the security of the data is remarkably enhanced through application of the identification system, the risks of unauthorized access and data leakage are effectively prevented, analysis of various data formats and protocols is supported, and the method is suitable for popularization and application. And the traceability of the data is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition and processing, and in particular to a data acquisition system and a data analysis method based on identification. Background Art

[0002] Data collection and the realization of all-round digital management have become mainstream demands. How to achieve data sharing, high performance of data collection, and then extract core value is the key to enterprise informatization construction.

[0003] General data collection systems collect data directly and cannot identify the data, thus failing to ensure the accuracy and consistency of the data. Moreover, direct data collection can easily lead to data leakage, and there are certain risks in data security, which cannot meet people's needs. Therefore, a data collection system and data analysis method based on identification are proposed to solve such problems. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the deficiencies in the prior art, the present invention provides a data acquisition system and data analysis method based on identification, which are mainly used to solve the problems that general data acquisition systems directly collect data and cannot identify the data, thereby failing to ensure the accuracy and consistency of the data, and that direct data collection is prone to data leakage, and there are certain risks in data security, which cannot meet people's needs.

[0006] (II) Technical solution

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A data acquisition system based on identification includes a system module assembly, wherein the system module assembly includes a gateway configuration module, a metadata setting module, a data management module, and a gateway identification module. The gateway configuration module is connected to the metadata setting module, and is connected to a gateway device via a data line. The metadata setting module performs metadata definition configuration on the gateway device and its location. The gateway identification module identifies the data of the metadata setting module and the gateway configuration module, and uses the identification information to identify the gateway. The gateway identification module is connected to the data management module, and the gateway identification module uses the identification information to identify the gateway, and reads the device operation data from the gateway, identifies the data, and uses the identification system to permanently record the device operation core data assets and upload them to the data management module. The user uses the data management module to configure the data acquisition related metadata and triggers the gateway to perform data acquisition.

[0009] Furthermore, the metadata setting module can customize metadata definitions and support parsing of multiple data formats and protocols.

[0010] Based on the above solution, the gateway identification module includes a tracing module that supports parsing of multiple data formats and protocols.

[0011] As a further solution of the present invention, the system module assembly includes a cloud service data warehouse, which builds an efficient, scalable and responsive data collection and analysis solution through the cloud service data warehouse, streaming processing, and time-driven methods, and is achieved by integrating the cloud service data warehouse, streaming data processing system and event-driven architecture.

[0012] Based on the above solution, the integration capability of the cloud service data warehouse allows data from different sources to be standardized to form a unified data view, thereby reducing data redundancy and improving data consistency.

[0013] The present invention also proposes a data analysis method based on identification, comprising the following steps:

[0014] S1: Gateway identification, using identification information to identify the gateway, and reading device operation data from the gateway, identification data, and using the identification system to permanently record the core data of device operation;

[0015] S2: Message collection, collecting and processing data through message middleware;

[0016] S3: Data analysis, analyze the collected data through the data analysis plug-in, and dynamically load the data analysis plug-in by identifying the gateway identifier and using the strategy mode to achieve data analysis flexibly and efficiently.

[0017] As a further solution of the present invention, S2 includes a data processing module, which improves the efficiency of processing a large amount of data, thereby optimizing the speed of data processing.

[0018] Furthermore, a stream data processing system is introduced into S2 to perform preliminary cleaning, aggregation and conversion of the data to prepare for subsequent storage and analysis.

[0019] Based on the above solution, S3 includes an event-driven architecture (EDA) as the coordination center of the entire solution.

[0020] (III) Beneficial effects

[0021] Compared with the prior art, the present invention provides a data collection system and data analysis method based on identification, which has the following beneficial effects:

[0022] 1. The present invention provides strong support for the enterprise's data strategy planning and long-term management by permanently recording the core data assets of the enterprise's equipment operation. The metadata setting module can customize metadata definitions and support the parsing of multiple data formats and protocols, so that the system can adapt to changing application scenarios and needs.

[0023] 2. The system of the present invention ensures the accuracy and consistency of data through precise identification technology, providing a solid foundation for data analysis and decision-making. Secondly, the application of the identification system significantly enhances the security of data and effectively prevents the risk of unauthorized access and data leakage.

[0024] 3. The present invention supports the parsing of multiple data formats and protocols through the gateway identification module, improves the traceability of data, and makes the source and flow process of data easily traceable, which is very beneficial for auditing, troubleshooting and performance optimization. The system supports the parsing of multiple data formats and protocols, making it suitable for a wider range of industrial and commercial environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the framework structure of a data acquisition system based on identification proposed by the present invention;

[0026] Figure 2 A schematic diagram of the process structure of a data parsing method based on identification proposed by the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Example 1

[0029] Reference Figure 1-Figure 2, a data acquisition system based on identification, including a system module assembly, the system module assembly includes a gateway configuration module, a metadata setting module, a data management module, and a gateway identification module. The gateway configuration module is connected to the metadata setting module, and is connected to the gateway device through a data line. The metadata setting module performs metadata definition configuration on the gateway device and its stepping point. The gateway identification module identifies the data of the metadata setting module and the gateway configuration module, and uses the identification information to identify the gateway. The gateway identification module is connected to the data management module. The gateway identification module uses the identification information to identify the gateway, and reads the device operation data from the gateway, identifies the data, and uses the identification system to permanently record the core data assets of the device operation and upload them to the data management module. By permanently recording the core data assets of the enterprise equipment operation, it provides strong support for the enterprise's data strategic planning and long-term management. The user uses the data management module to configure the data acquisition related metadata and triggers the gateway to collect data. The system ensures the accuracy and consistency of the data through precise identification technology, providing a solid foundation for data analysis and decision-making. Secondly, the application of the identification system significantly enhances the security of the data and effectively prevents the risk of unauthorized access and data leakage.

[0030] In the present invention, the metadata setting module can customize metadata definitions and supports the parsing of multiple data formats and protocols, so that the system can adapt to changing application scenarios and needs. The gateway identification module includes a tracing module, which supports the parsing of multiple data formats and protocols, improves the traceability of data, and makes the source and flow process of data easily traceable, which is very beneficial for auditing, troubleshooting and performance optimization. The system supports the parsing of multiple data formats and protocols, making it suitable for a wider range of industrial and commercial environments.

[0031] The present invention also proposes a data analysis method based on identification, comprising the following steps:

[0032] S1: Gateway identification, using identification information to identify the gateway, and reading device operation data from the gateway, identification data, and using the identification system to permanently record the core data of device operation;

[0033] S2: Message collection, collecting and processing data through message middleware;

[0034] S3: Data analysis, analyze the collected data through the data analysis plug-in, and dynamically load the data analysis plug-in by identifying the gateway identifier and using the strategy mode to achieve data analysis flexibly and efficiently.

[0035] In particular, S2 includes a data processing module, which improves the efficiency of processing large amounts of data, thereby optimizing the speed of data processing.

[0036] Example 2

[0037] Reference Figure 1-Figure 2 , a data acquisition system based on identification, including a system module assembly, the system module assembly includes a gateway configuration module, a metadata setting module, a data management module, and a gateway identification module. The gateway configuration module is connected to the metadata setting module, and is connected to the gateway device through a data line. The metadata setting module performs metadata definition configuration on the gateway device and its stepping point. The gateway identification module identifies the data of the metadata setting module and the gateway configuration module, and uses the identification information to identify the gateway. The gateway identification module is connected to the data management module. The gateway identification module uses the identification information to identify the gateway, and reads the device operation data from the gateway, identifies the data, and uses the identification system to permanently record the core data assets of the device operation and upload them to the data management module. By permanently recording the core data assets of the enterprise equipment operation, it provides strong support for the enterprise's data strategic planning and long-term management. The user uses the data management module to configure the data acquisition related metadata and triggers the gateway to collect data. The system ensures the accuracy and consistency of the data through precise identification technology, providing a solid foundation for data analysis and decision-making. Secondly, the application of the identification system significantly enhances the security of the data and effectively prevents the risk of unauthorized access and data leakage.

[0038] In the present invention, the metadata setting module can customize metadata definitions and supports the parsing of multiple data formats and protocols, so that the system can adapt to changing application scenarios and needs. The gateway identification module includes a tracing module, which supports the parsing of multiple data formats and protocols, improves the traceability of data, and makes the source and flow process of data easily traceable, which is very beneficial for auditing, troubleshooting and performance optimization. The system supports the parsing of multiple data formats and protocols, making it suitable for a wider range of industrial and commercial environments. The system module assembly includes a cloud service data warehouse, which is used to build an efficient, scalable and responsive data collection and analysis solution through the cloud service data warehouse, streaming processing, and time-driven methods. By integrating the cloud service data warehouse, the streaming data processing system and the event-driven architecture, the integration capability of the cloud service data warehouse allows data from different sources to be standardized to form a unified data view, thereby reducing data redundancy and improving data consistency.

[0039] The present invention also proposes a data analysis method based on identification, comprising the following steps:

[0040] S1: Gateway identification, using identification information to identify the gateway, and reading device operation data from the gateway, identification data, and using the identification system to permanently record the core data of device operation;

[0041] S2: Message collection, collecting and processing data through message middleware;

[0042] S3: Data analysis, analyze the collected data through the data analysis plug-in, and dynamically load the data analysis plug-in by identifying the gateway identifier and using the strategy mode to achieve data analysis flexibly and efficiently.

[0043] In particular, S2 includes a data processing module, which improves the efficiency of processing large amounts of data, thereby optimizing the speed of data processing.

[0044] In particular, the introduction of stream data processing systems in S2, such as Apache Kafka or Apache Flink, enables the solution to process high-speed data streams in real time. These systems are designed for low-latency data processing and are suitable for application scenarios with extremely high real-time requirements, such as monitoring systems, real-time analysis, and transaction processing. The stream data processing system can perform preliminary cleaning, aggregation, and transformation of data to prepare for subsequent storage and analysis. S3 includes an event-driven architecture (EDA) as the coordination center of the entire solution, providing a flexible data processing mechanism. In EDA, data is regarded as a series of events, and the system can dynamically trigger the corresponding processing flow based on these events. This mechanism not only improves the response speed of data processing, but also allows the system to flexibly adjust the data processing logic according to changes in business needs.

[0045] In the description of this article, it should be noted that relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

Claims

1. A data acquisition system based on identification, comprising a system module assembly, characterized in that: The system module assembly includes a gateway configuration module, a metadata setting module, a data management module, and a gateway identification module. The gateway configuration module is connected to the metadata setting module and is connected to the gateway device via a data line. The metadata setting module defines and configures metadata for the gateway device and its location. The gateway identification module identifies the data of the metadata setting module and the gateway configuration module, and uses the identification information to identify the gateway. The gateway identification module is connected to the data management module. The gateway identification module uses the identification information to identify the gateway, reads the device operation data from the gateway, identifies the data, and uses the identification system to permanently record the core data assets of the device operation and upload them to the data management module. The user uses the data management module to configure data acquisition-related metadata and trigger the gateway to perform data acquisition.

2. The data acquisition system based on identification according to claim 1, characterized in that: The metadata setting module can customize metadata definitions and supports parsing of multiple data formats and protocols.

3. The data acquisition system based on identification according to claim 1, characterized in that: The gateway identification module includes a tracing module that supports parsing of multiple data formats and protocols.

4. The data acquisition system based on identification according to claim 1, characterized in that: The system module assembly includes a cloud service data warehouse, which uses the cloud service data warehouse, stream processing, and time-driven methods to build an efficient, scalable and responsive data collection and analysis solution, which is achieved by integrating the cloud service data warehouse, stream data processing system and event-driven architecture.

5. The data acquisition system based on identification according to claim 4, characterized in that: The integration capability of the cloud service data warehouse allows data from different sources to be standardized to form a unified data view, thereby reducing data redundancy and improving data consistency.

6. A data analysis method based on identification, characterized in that: The following steps are involved: S1: Gateway identification, using identification information to identify the gateway, and read the device operation data from the gateway, identification data, and use the identification system to permanently record the core data of the device operation; S2: Message collection, collecting and processing data through message middleware; S3: Data parsing, parses the collected data through the data parsing plug-in, and dynamically loads the data parsing plug-in using the strategy mode by identifying the gateway identifier to achieve data parsing flexibly and efficiently.

7. The data analysis method based on identification according to claim 6, characterized in that: The S2 includes a data processing module, which improves the efficiency of processing large amounts of data, thereby optimizing the speed of data processing.

8. The data analysis method based on identification according to claim 6, characterized in that: The stream data processing system is introduced into S2 to perform preliminary cleaning, aggregation and conversion of the data to prepare for subsequent storage and analysis.

9. The data analysis method based on identification according to claim 6, characterized in that: The S3 includes an event-driven architecture (EDA) as the coordination center of the entire solution.

Images