Multi-source heterogeneous data fusion and real-time interaction system and method based on SpringBoot

Abstract

The invention relates to a multi-source heterogeneous data fusion and real-time interaction system and method based on SpringBoot, and the system comprises a data collection layer which is responsible for collecting data from all information systems, and supports a plurality of data formats and protocols; the data preprocessing layer is used for cleaning, converting and standardizing the collected data; the data interaction layer is used for constructing a RESTfulAPI interface based on a SpringBoot framework to realize real-time transmission and interaction of data; the data storage layer is used for storing fused data by adopting a distributed database; and the application service layer is used for providing application services such as data visualization, analysis and early warning and supporting management decisions. The method comprises the steps of data acquisition, data preprocessing, data interaction, data storage and application. The method has the advantages of higher real-time performance, better compatibility and lower data processing cost. And meanwhile, through the flexibility and expandability of the SpringBoot framework, the system can be easily integrated and expanded with other systems.

Description

Technical Field

[0001] This invention relates to a multi-source heterogeneous data fusion and real-time interaction system and method based on Spring Boot, belonging to the technical field of data processing. Background Technology

[0002] Currently, safety production management in chemical industrial parks faces challenges such as diverse data sources, inconsistent formats, and high real-time requirements. Zhenqu 3.X Safety Production Management Platform, as a mature safety management software for chemical industrial parks, provides rich data monitoring and management functions. However, different chemical industrial parks or enterprises may use different information systems, leading to serious data silos and making real-time data sharing and interaction difficult. Traditional data integration methods are often inefficient and cannot meet the requirements of modern chemical industrial parks for data real-time performance, accuracy, and security.

[0003] Current problems: Data silos: The inability to effectively share data between different systems leads to a lack of comprehensive data support for management decisions.

[0004] Poor real-time performance: Traditional methods are difficult to achieve real-time data collection and transmission, which affects the speed of emergency response.

[0005] Format incompatibility: The inconsistent formats of multi-source heterogeneous data make processing difficult and costly.

[0006] Low security: There are security risks during data transmission, making it vulnerable to attacks or leaks. Summary of the Invention

[0007] To address the aforementioned problems, this invention discloses a multi-source heterogeneous data fusion and real-time interaction system and method based on Spring Boot, the specific technical solution of which is as follows: A multi-source heterogeneous data fusion and real-time interaction system based on Spring Boot includes: Data Acquisition Layer: Responsible for collecting data from various information systems, supporting multiple data formats and protocols; Data preprocessing layer: Cleans, transforms, and standardizes the collected data to ensure consistent data format; Data interaction layer: Based on the Spring Boot framework, a RESTful API interface is built to realize real-time data transmission and interaction; Data storage layer: A distributed database is used to store the merged data, ensuring data security and scalability; Application service layer: Provides application services for data visualization, analysis and early warning, supporting management decision-making; The system collects raw data from various information systems through the data acquisition layer. After being cleaned and transformed by the data preprocessing layer, the data is transmitted in real time to the data interaction layer through the Spring Boot interface service. The data interaction layer is responsible for receiving, parsing and forwarding the data, ensuring seamless data integration between different systems. The merged data is stored in a distributed database for application service layer to call and analyze.

[0008] The method for a multi-source heterogeneous data fusion and real-time interaction system based on Spring Boot, as described above, includes the following steps: Step 1: Data Acquisition: The data acquisition layer connects to different data sources through the adapter pattern, and the raw data is pushed to the preprocessing layer in the form of JSON / XML / binary stream; Step 2: Data preprocessing: including Data cleaning: filtering out missing / outlier values; Format conversion: Converting unstructured data into structured data; Standardized output: uniformly in Avro / Protobuf format; Step 3: Data Interaction: including RESTful API: The application receives pre-processed data; WebSocket: Real-time push of alarm information; Security controls: JWT authentication + AES-256 message encryption; Step 4: Data storage: Kafka streaming write → Flink real-time computation → storage of historical data in HBase + hot data in Redis; Step 5: Data application: GraphQL interface for on-demand querying.

[0009] The specific process of converting the original data into JSON / XML / binary stream format in step 1 is as follows: For structured data in MySQL, the process involves retrieving data through queries: executing an SQL query (SELECT FROM table), converting each returned row into a JSON object, with each column as a property of the object, and combining all the JSON objects from all rows into a JSON array; for semi-structured data in MongoDB, the process reads documents directly from MongoDB and serializes them into JSON strings; and for binary data, the process encodes the entire binary file (such as an image) into a Base64 string and then uses this string as a property in the JSON object.

[0010] Furthermore, the specific process of filtering missing / outlier values ​​in step 2 is as follows: As a relational database, MySQL has a strict table structure and handles missing values, which are usually represented by NULL. Cleanup operations are based on SQL, and outlier values ​​are handled by filtering and deletion. Cleanup uses an aggregation framework and query operations. Missing values ​​are represented by fields not existing, fields with null values, empty strings "", or empty arrays [].

[0011] Furthermore, the specific process of converting unstructured data into structured data in step 2 is as follows: writing regular expressions to define the extracted information.

[0012] Furthermore, the specific process of standardizing the output in step 2 is as follows: 2.1 First, add the necessary dependencies to pom.xml. 2.2 Define the data schema: Create a schema file in the src / main / avro / directory; 2.3 Avro serialization service, 2.4 Protobuf serialization service, 2.5 Unified Format Conversion Controller 2.6 Finally, perform batch conversion.

[0013] The beneficial effects of this invention are: This invention breaks down data silos: enabling real-time sharing and interaction of data between different systems, thereby improving the scientific nature of management decisions.

[0014] This invention improves real-time performance: by using Spring Boot interface services, it ensures real-time data collection and transmission, thereby improving emergency response speed.

[0015] This invention unifies the data format: the data preprocessing layer standardizes multi-source heterogeneous data, reducing the difficulty and cost of data processing.

[0016] This invention enhances security by employing security measures such as encrypted transmission and access control to ensure the security of data transmission.

[0017] Compared to traditional data integration methods, this invention offers higher real-time performance, better compatibility, and lower data processing costs. Furthermore, the flexibility and scalability of the Spring Boot framework facilitate integration and expansion with other systems. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the system framework of the present invention. Detailed Implementation

[0019] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0020] This invention provides a data interaction system and method for a safety production management and control platform in a chemical industrial park based on the Spring Boot interface. It aims to solve problems such as data silos, poor real-time performance, format incompatibility, and low security in existing technologies. The system utilizes the Spring Boot framework to build efficient and stable interface services, enabling seamless integration between the Zhenqu 3.X safety production management and control platform and existing information systems in the Hexian Chemical Industrial Park (or other chemical industrial parks), achieving real-time collection, fusion, and interaction of multi-source heterogeneous data.

[0021] 2. System Structure: Data Acquisition Layer: Responsible for collecting data from the Zhenqu 3.X platform and various information systems in Hexian Chemical Industrial Park, supporting multiple data formats and protocols.

[0022] Data preprocessing layer: Cleans, transforms, and standardizes the collected data to ensure a consistent data format.

[0023] Data interaction layer: Based on the Spring Boot framework, a RESTful API interface is built to realize real-time data transmission and interaction.

[0024] Data storage layer: A distributed database is used to store the merged data, ensuring data security and scalability.

[0025] Application service layer: Provides application services such as data visualization, analysis and early warning to support management decision-making.

[0026] 3. Working principle: The system collects raw data from the Zhenqu 3.X platform and various information systems in the Hexian Chemical Industrial Park through the data acquisition layer. After cleaning and transformation by the data preprocessing layer, the data is transmitted in real time to the data interaction layer via Spring Boot interface services. The data interaction layer is responsible for receiving, parsing, and forwarding data, ensuring seamless data integration between different systems. The merged data is stored in a distributed database for the application service layer to access and analyze.

[0027] Combined with appendix Figure 1 The implementation process of this invention is described below: 1. Experimental setup: Experimental environment: A test environment was set up that included the Zhenqu 3.X platform, the existing information system of Hexian Chemical Industrial Park, and the data interaction system of this invention.

[0028] Experimental parameters: Set different parameters such as data acquisition frequency, transmission rate and number of concurrent users to simulate actual operating scenarios.

[0029] I. Data Acquisition Layer → Data Preprocessing Layer Interaction method: 1. Connect to different data sources (Zhenqu 3.X platform, Hexian Chemical Industrial Park system, etc.) through the adapter pattern.

[0030] The data acquisition layer connects to different data sources via the adapter pattern. It processes structured data from MySQL by executing SQL queries (SELECT FROM table), converting each returned row into a JSON object, with each column as an attribute of the object, and all JSON objects from all rows forming a JSON array. For semi-structured data from MongoDB, it reads documents directly from MongoDB as serialized JSON strings. For binary data, it encodes the entire binary file (such as an image) into a Base64 string, then uses this string as an attribute of the JSON. Thus, all types of data are pushed to the preprocessing layer in JSON / XML / binary stream format.

[0031] 2. Supported protocols: HTTP / HTTPS, MQTT, OPCUA, Modbus, etc.

[0032] 3. Raw data is pushed to the preprocessing layer in JSON / XML / binary stream format.

[0033] II. Data Preprocessing Layer → Data Interaction Layer Interaction method: 1. Data cleaning: Filtering out missing / outlier values ​​(such as drift data from chemical sensors).

[0034] As a relational database, MySQL has a strict table structure. When dealing with missing values, which usually appear as NULL, the cleaning operation is mainly based on SQL. For example, SELECT FROM table_name WHERE column_name IS NULL; When dealing with outliers, filtering and deletion are adopted. For example, DELETE FROM table_name WHERE column_name < (SELECT AVG(column_name) - 3 STDDEV(column_name) FROM table_name) OR column_name > (SELECT AVG(column_name) + 3 STDDEV(column_name) FROM table_name) WHERE Numeric_column IS NULL; MongoDB is a document database with a flexible structure. Cleaning mainly uses the aggregation framework and query operations. The manifestations of dealing with missing values are that the field does not exist, the field value is null, an empty string "", or an empty array [].

[0035] 2. Format conversion: Unstructured data (such as logs) → Structured data (such as database tables).

[0036] Write regular expressions to define the extracted information, which is suitable for data with relatively fixed formats and clear patterns. It is fast, low-cost, and accurate in results.

[0037] 3. Standardized output: Unified into Avro / Protobuf format.

[0038] 1) First, add the necessary dependencies in pom.xml: <!-- Avro dependency --> <dependency> <groupid> org.apache.avro< / groupid> <artifactid> euro< / artifactid> <version> 1.11.3< / version> < / dependency> <dependency> <groupid> org.apache.avro< / groupid> <artifactid> avro-maven-plugin< / artifactid> <version> 1.11.3< / version> < / dependency> <!-- Protobuf dependency --> <dependency> <groupid> com.google.protobuf< / groupid> <artifactid> protobuf-java< / artifactid> <version> 3.25.1< / version> < / dependency> <dependency> <groupid> com.google.protobuf< / groupid> <artifactid> protobuf-java-util< / artifactid> <version> 3.25.1< / version> < / dependency> .

[0039] 2) Define the data Schema Create a Schema file: user.avsc in the src / main / avro / directory: { "type": "record", "name": "User", "namespace": "com.example.avro", "fields": [ { "name": "id", "type": "int" }, { "name": "name", "type": "string" }, { "name": "email", "type": "string" }, { "name": "age", type: ["int", "null"] }, { "name": "createdAt", "type": { "type": "long", "logicalType": "timestamp-millis" } } ] }

[0040] 3) Avro serialization service.

[0041] 4) Protobuf serialization service.

[0042] 5) Unified format conversion controller.

[0043] 6) Finally, perform batch conversion.

[0044] III. Core Logic of the Data Interaction Layer (SpringBoot) Key components: 1. RESTful API: POST / api / v1 / data receives pre-processed data.

[0045] 2. WebSocket: Real-time push of alarm information (such as temperature exceeding the standard).

[0046] 3. Security control: JWT authentication + AES-256 message encryption.

[0047] IV. Data Storage Layer → Application Service Layer Interaction Design: 1. Write path: Kafka streaming write → Flink real-time computation → store in HBase (historical data) + Redis (hot data).

[0048] 2. Read path: GraphQL interface queries on demand.

[0049] 2. Experimental Results: Data real-time performance: Experiments show that the system can complete data acquisition, transmission and fusion within milliseconds, meeting the real-time requirements.

[0050] Data compatibility: The system successfully processed data in various formats and protocols, verifying its good compatibility.

[0051] System stability: Under high concurrency scenarios, the system can still maintain stable operation without data loss or transmission errors.

[0052] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0053] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A multi-source heterogeneous data fusion and real-time interaction system based on Spring Boot, characterized in that, include: Data Acquisition Layer: Responsible for collecting data from various information systems, supporting multiple data formats and protocols; Data preprocessing layer: Cleans, transforms, and standardizes the collected data to ensure consistent data format; Data interaction layer: Based on the Spring Boot framework, a RESTful API interface is built to realize real-time data transmission and interaction; Data storage layer: A distributed database is used to store the merged data, ensuring data security and scalability; Application service layer: Provides application services for data visualization, analysis and early warning, supporting management decision-making; The system collects raw data from various information systems through the data acquisition layer. After being cleaned and transformed by the data preprocessing layer, the data is transmitted in real time to the data interaction layer through the Spring Boot interface service. The data interaction layer is responsible for receiving, parsing and forwarding the data, ensuring seamless data integration between different systems. The merged data is stored in a distributed database for application service layer to call and analyze.

2. The method for multi-source heterogeneous data fusion and real-time interaction system based on Spring Boot as described in claim 1, characterized in that, Includes the following steps: Step 1: Data Acquisition: The data acquisition layer connects to different data sources through the adapter pattern, and the raw data is pushed to the preprocessing layer in the form of JSON / XML / binary stream; Step 2: Data preprocessing: including Data cleaning: filtering out missing / outlier values; Format conversion: Converting unstructured data into structured data; Standardized output: uniformly in Avro / Protobuf format; Step 3: Data Interaction: including RESTful API: The application receives pre-processed data; WebSocket: Real-time push of alarm information; Security controls: JWT authentication + AES-256 message encryption; Step 4: Data storage: Kafka streaming write → Flink real-time computation → storage of historical data in HBase + hot data in Redis; Step 5: Data application: GraphQL interface for on-demand querying.

3. The method for a multi-source heterogeneous data fusion and real-time interaction system based on Spring Boot according to claim 2, characterized in that, The specific process of converting the raw data into JSON / XML / binary stream format in step 1 is as follows: For structured data in MySQL, the process involves retrieving data through queries: executing an SQL query (SELECT * FROM table), converting each returned row into a JSON object, with each column as a property of the object, and combining all the JSON objects from all rows into a JSON array; for semi-structured data in MongoDB, the process reads documents directly from MongoDB and serializes them into JSON strings; and for binary data, the process encodes the entire binary file (such as an image) into a Base64 string and then uses this string as a property in the JSON object.

4. The method for a multi-source heterogeneous data fusion and real-time interaction system based on Spring Boot according to claim 2, characterized in that, The specific process for filtering missing / outlier values ​​in step 2 is as follows: As a relational database, MySQL has a strict table structure and handles missing values, which are usually represented by NULL. Cleanup operations are based on SQL, and outlier values ​​are handled by filtering and deletion. Cleanup uses an aggregation framework and query operations. Missing values ​​are represented by fields not existing, fields with null values, empty strings "", or empty arrays [].

5. The method for a multi-source heterogeneous data fusion and real-time interaction system based on Spring Boot according to claim 2, characterized in that, The specific process of converting unstructured data into structured data in step 2 is as follows: write regular expressions to define the information to be extracted.

6. The method for a multi-source heterogeneous data fusion and real-time interaction system based on Spring Boot according to claim 2, characterized in that, The specific process of standardizing the output in step 2 is as follows: 2.1 First, add the necessary dependencies to pom.xml. 2.2 Define the data schema: Create a schema file in the src / main / avro / directory; 2.3 Avro serialization service, 2.4 Protobuf serialization service, 2.5 Unified Format Conversion Controller 2.6 Finally, perform batch conversion.

Images