Target-oriented multi-source heterogeneous service system and integration method

Abstract

The invention provides a target-oriented multi-source heterogeneous service system and method, and the system comprises a data integration layer which is used for extracting data from a source system, and carrying out the conversion, mapping and cleaning of the data; the target guiding layer is used for fusing the multi-source data according to the operation target of the production line; the data governance layer is used for data cleaning, duplicate removal and consistency check; data security management guarantees the security of data in transmission and storage through encryption, authentication and authority control; the application service layer provides query, visualization and analysis of integrated data; the data storage and database is used for storing the data in a relational database, an NoSQL database or a time sequence database; the rule engine system comprises a rule storage layer, a rule execution layer and a rule management layer. According to the invention, target-oriented multi-source heterogeneous service system data integration can be realized.

Description

Technical Field

[0001] The present invention relates to the field of data integration, and in particular to a goal-oriented multi-source heterogeneous business system and an integration method. Background Art

[0002] In the new round of technological transformation and development, the rapid advancement of information technology and big data has driven the complexity and diversification of business systems across various industries. Traditional business systems are often independent due to differences in data storage formats, transmission protocols, and interface standards, creating "data silos" that severely hinder data sharing and efficient utilization. Especially in the increasingly complex business environment of enterprises, achieving data interoperability between disparate systems has become a core challenge in enterprise digital transformation.

[0003] As business scenarios become more diverse and dynamic, data sources are becoming increasingly complex, encompassing structured, semi-structured, and unstructured data, distributed across diverse systems like ERP and CRM, as well as third-party platforms. Traditional integration approaches often rely on static adapters or a single platform. However, facing dynamically changing requirements and heterogeneous environments, these approaches lack flexibility, require long development cycles, and exhibit poor scalability. Summary of the Invention

[0004] To address the above issues, the present invention proposes a goal-oriented multi-source heterogeneous business system and integration method, which breaks down data silos through efficient data abstraction, intelligent scheduling, and dynamic adaptation, providing solid support for enterprise intelligent decision-making and business innovation.

[0005] A target-oriented multi-source heterogeneous business system according to the present invention includes: The data integration layer extracts data from various source systems and transforms, maps, and cleanses it; The goal-oriented layer integrates multi-source data according to the production line's operating goals; The data governance layer ensures the accuracy and integrity of integrated data through data cleansing, deduplication, and consistency checks. Data security management ensures the security of data during transmission and storage through encryption, authentication, and permission control. Application service layer, which provides query, visualization, and analysis of integrated data; Data storage and databases: storing data in relational databases, NoSQL databases, or time series databases to support subsequent real-time queries and analysis; The rule engine system includes a rule storage layer, a rule execution layer, and a rule management layer. The rule representation technology uses RuleML to represent business rules. The dynamic rule generation mechanism is based on a machine learning algorithm, automatically discovering rule patterns from historical data. The rule conflict resolution strategy adopts a mechanism based on priority and specificity, and uses a Git-like version control mechanism to manage rule changes.

[0006] Furthermore, the data integration layer includes: Data acquisition module, which extracts data from different systems through various communication protocols; The data mapping module converts data into a unified data model based on the goals and rules provided by the goal-oriented layer, so that the data format and semantics can adapt to the requirements of downstream systems; The data cleaning module processes the collected data, removes redundant and abnormal data, and ensures the integrity and accuracy of the data; The multi-source data abstraction layer maps the parsed data from different sources into a unified data model, and achieves standardized representation of the data by building a unified data model.

[0007] The present invention also provides a method for integrating a multi-source heterogeneous business system data integration system based on a target orientation, comprising the steps of: Step 1: Match data request with protocol; Step 2: Data acquisition and conversion; Step 3: Convert data into historical data; Step 4: Data adaptation and transmission; Step 5: Data storage and processing; Step 6: Perform real-time data processing and dynamic adjustment.

[0008] Furthermore, step one includes: First, a data request is sent to the source system through the adaptation protocol, specifying the type and format of the data to be obtained. This request message includes a description of the required data, the data protocol requirements, and the selected transmission method. The system generates a request message to ensure that the requested data is consistent with the protocol requirements of the source system. The system automatically identifies and selects the most suitable protocol adapter based on the protocol type of the source system. If the source system supports multiple protocols, the system will select the most appropriate protocol through preset rules or based on business needs.

[0009] Furthermore, step 2 includes: Obtain raw data from the source system according to the selected protocol; Through the adaptation protocol, data is converted into a format acceptable to the target system; during the conversion, the data fields in the source system are mapped so that the semantics of the data can be correctly understood in the target system.

[0010] Furthermore, step three includes: converting real-time data into historical data through a rule engine system.

[0011] Furthermore, step four includes: After the data format and protocol are converted, the adaptation protocol enables the data to be smoothly transmitted to the target system. The adaptation protocol includes data format conversion, data transmission security and protocol selection; Data is transmitted to the target system through the network. During the transmission process, the adaptation protocol ensures the integrity and correctness of the data. After the target system receives the data, it processes and stores it.

[0012] Furthermore, step five includes: The target system stores the data in the corresponding database according to the type and purpose of the data; The stored data is queried, analyzed and displayed through the interface provided by the application service layer.

[0013] Furthermore, step six includes: The system processes data in real time and displays it through dashboards and charts; The rule engine system automatically adjusts data processing strategies according to changes in business needs and goals. The rule engine system dynamically generates new rules based on real-time data and business needs to ensure that the data integration process is always consistent with the goals.

[0014] Beneficial effects: Compared with the prior art, the present invention has the following advantages: 1. In light of the characteristics of multi-source, heterogeneous business systems and the data integration requirements, a goal-oriented approach was employed to define the overall architecture and core functions of data integration technology. This architecture guided each step of data integration through goal-setting, providing a clear direction and systematic solution for technology implementation.

[0015] 2. Leveraging ETL technology and an ESB architecture, we address issues such as format differences and inconsistent communication protocols across heterogeneous data sources, enabling efficient extraction, dynamic conversion, and reliable transmission of complex data. Dynamic data mapping rules generated by the rules engine system adapt to the business needs of diverse scenarios, significantly improving the efficiency and accuracy of data integration.

[0016] 3. Through the integrated application service layer, it provides rich data query, analysis, and visualization capabilities, enabling data sharing and collaboration across business systems. Users can monitor and manage key data in the system in real time, optimize business processes, and make intelligent business decisions based on integrated data.

[0017] 4. This invention provides comprehensive data quality management, data security protection, and process monitoring capabilities through the data governance layer, ensuring the accuracy, security, and traceability of data during the integration process. This effectively safeguards data credibility and system stability, especially in complex business environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 It is a system architecture diagram of the multi-source heterogeneous business system data integration technology according to an embodiment of the present invention.

[0019] Figure 2 It is an operational flow chart of multi-source heterogeneous business system data integration technology. DETAILED DESCRIPTION

[0020] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other.

[0021] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0022] The present invention relates to a goal-oriented multi-source heterogeneous business system data integration technology, which aims to guide the extraction, conversion, transmission and integration of multi-source heterogeneous data through clear data integration goals, and ultimately achieve data sharing and collaboration between different business systems. Multi-source heterogeneous business systems generally include ERP, MES, CRM and other systems. There are differences in architecture, data format, protocol, etc. between these systems, which makes data exchange difficult, forms data islands, and increases the complexity of data integration. To overcome this problem, the present invention ensures the unified processing and flow of heterogeneous data through a goal-oriented integration method, an intelligent rule engine system and a data mapping module.

[0023] The goal-oriented layer is the starting point of this invention. It employs an ontology-based goal modeling approach to decompose business goals into a set of quantifiable sub-goals. The system uses OWL to define the goal ontology model. Each business goal is represented as a goal class, which includes attributes such as goal ID, priority, dependencies, and quantifiable indicators. The system uses a bidirectional mapping algorithm to construct a goal-data mapping matrix, clearly defining the relevance of each data element to the business goal. The goal-driven data conversion engine implements the data processing flow based on a directed acyclic graph, dynamically adjusting the node execution order and parameters based on active business goals. The goal adaptation feedback mechanism utilizes closed-loop control theory to automatically adjust processing parameters by monitoring the deviation between processing results and expected goals in real time. The data integration layer is the core module of this technology, responsible for processing and integrating multi-source heterogeneous data. The system collects multiple data elements, including sound, light, electricity, and images, and transmits and converts them through adaptation protocols, ultimately integrating the data into a standardized format. For example, sound data is collected by a microphone or other sensor and converted into digital signals using an analog-to-digital converter (ADC). Image data is collected by digital cameras and other devices and converted into recognizable machine data through image processing algorithms (such as denoising and edge detection). Electrical signals (such as temperature, current, and humidity) are converted into digital format using appropriate sensors for further processing.

[0024] The rule engine system adopts a three-tier architecture, consisting of a rule storage layer, a rule execution layer, and a rule management layer. RuleML is used to represent business rules. Dynamic rule generation is based on machine learning algorithms, automatically discovering rule patterns from historical data. Rule conflict resolution utilizes a priority- and specificity-based mechanism, and rule changes are managed using a Git-like version control mechanism.

[0025] During the data conversion process, the raw data in the MES needs to be converted into the business data required by the ERP system based on energy consumption coefficients and cost algorithms. This process requires not only calculation and induction, but also ensuring consistency in data format and semantics. Through an intelligent rule engine system, the system can dynamically adjust the data processing process when the goals change. For example, in some scenarios, the goal may be adjusted from "cost accounting" to "production efficiency improvement." The data integration framework will automatically adjust the data processing strategy based on the new goal. This goal-oriented approach ensures the accuracy and consistency of each data conversion step and can be optimized in real time according to needs.

[0026] Step 1: Matching data request with protocol During the data request phase, the system first sends a data request to the source system (such as a CNC system or MES system) using an adaptation protocol, specifying the type and format of the requested data. The system uses a signature-based protocol identification algorithm to automatically identify the protocol types (such as Modbus and OPC UA) supported by the source system by analyzing packet headers and transmission characteristics. The system maintains a protocol signature library containing feature vectors for common industrial and enterprise protocols. This request message includes a description of the requested data, the data protocol requirements, and the selected transmission method. The system automatically generates a request message to ensure that the requested data is consistent with the protocol requirements of the source system. A template engine is used to generate request messages that conform to the target protocol specifications. The system stores message templates for various protocols, such as Modbus request templates and OPC UA read service request templates, and dynamically populates parameters based on requirements. During the protocol matching process, the system automatically identifies and selects the most suitable protocol adapter based on the source system's protocol type (such as Modbus, OPC UA, or REST API). The system maintains a protocol adapter performance matrix that records the historical performance of each adapter under different conditions. If the source system supports multiple protocols, the system selects the most appropriate protocol based on pre-set rules or business requirements. For example, if the source system supports both Modbus and OPC UA protocols, the system automatically selects the protocol based on the requirements of the target system or preset rules to ensure smooth data exchange.

[0027] Step 2: Data acquisition and conversion During the data acquisition phase, the system retrieves raw data from the source system according to the selected protocol. For example, a CNC system might provide real-time processing data, while an MES system provides data during the production process. Based on the protocol specifications, the system initiates access to the data source and retrieves the required data.

[0028] During the data format conversion phase, the acquired data usually does not conform to the standard format of the target system, so data format conversion is required. The data format conversion engine is based on XSLT transformation technology and defines conversion templates from various source formats to standard formats. The system maintains a format conversion mapping library, which contains conversion rules from common enterprise system data formats (such as SAP IDocs, Oracle table structures, etc.) to unified JSON / XML formats. Through the adaptation protocol, the data is converted into a format acceptable to the target system. For example, converting from the Modbus protocol to formats such as JSON and XML, or converting binary data into a parsable structured data format. Semantic mapping is a key step in data conversion. The system maps the data fields in the source system to ensure that the semantics of the data can be correctly understood in the target system. For example, the "processing temperature" field of the CNC system may need to be mapped to the "equipment status" field in the ERP system to ensure that the target system can correctly process and use this data.

[0029] Step 3: Convert data into historical data During the real-time data conversion phase, the source system may contain real-time or process data, while the target system typically requires historical data. In this case, the system uses a rules engine to convert real-time data into historical data. For example, real-time production process data (such as temperature and speed) can be calculated to generate averages, cumulative values, or other statistical indicators, thereby generating historical data that facilitates analysis and decision-making in the target system.

[0030] The data supplementation and correction algorithm uses linear interpolation and exponential smoothing algorithms to handle missing and anomalies in data. For time series data, the system uses the ARIMA model to predict missing values ​​and uses median filtering to correct detected anomalies. Data calculation and synthesis are important links in data transformation. The system performs complex data calculations through an associated data calculation framework based on a rule engine system. The system maintains a business formula library that contains calculation rules for different fields (such as cost calculation, energy consumption calculation), and supports dynamic adjustment of parameters. For example, when converting production data into financial data, the system will calculate according to the cost formula and energy consumption formula, and summarize them into report data for management personnel to use. This process ensures the high reliability of data in the target system when making decisions. Step 4: Data adaptation and transmission During the adaptation protocol selection phase, after the data format and protocol are converted, the adaptation protocol ensures smooth data transmission to the target system. This adaptation protocol not only includes data format conversion but also considers data transmission security (such as encryption and authentication) and protocol selection (such as HTTP or WebSocket). The system automatically selects the optimal protocol based on the target system's requirements to ensure efficient data transmission.

[0031] During the data transmission phase, data is transmitted over the network to the target system. During transmission, the adapted protocol ensures data integrity and accuracy, preventing loss or corruption. After receiving the data, the target system processes and stores it, providing support for subsequent operations. This process includes data verification, encryption, and compression to ensure secure and efficient transmission.

[0032] Step 5: Data storage and processing During the data storage phase, the target system stores the data in a corresponding database (such as a relational database or NoSQL database) based on the data type and purpose. The stored data will be used for subsequent data processing and analysis.

[0033] During the data analysis and processing phase, stored data can be queried, analyzed, and displayed through interfaces provided by the application service layer. For example, the system can calculate production costs and energy consumption in real time or generate management reports to support decision-making. Through data visualization, managers can view production and business data in real time on a dashboard, providing strong support for decision-making.

[0034] Step 6: Real-time data processing and dynamic adjustment During the real-time data stream processing phase, for business scenarios with high real-time requirements, such as production process monitoring and equipment status monitoring, the system can process data in real time and present it to management personnel through dashboards and charts. The system can detect data changes in real time and automatically adjust processing strategies based on pre-set business objectives (such as improving production efficiency and optimizing energy consumption).

[0035] The dynamic adjustment of the intelligent rule engine system is a key feature of this invention. During the data integration process, the requirements of the target system may change (for example, from "cost accounting" to "production efficiency improvement"). In this case, the rule engine system can automatically adjust the data processing strategy based on the changing business needs and goals, ensuring that the system can flexibly respond to new business requirements. The rule engine system dynamically generates new rules based on real-time data and business needs, ensuring that the data integration process remains consistent with the goals.

[0036] The adaptation mechanism is a core component of this invention, ensuring seamless data integration between the source and target systems. The adaptation protocol not only involves protocol conversion but also involves the detailed process of data request and response, ensuring the integrity and consistency of data as it flows from the source system to the target system.

[0037] The system uses automated tools to identify the protocol types supported by the source system and selects the appropriate adapter protocol based on the requirements of the target system. For example, when transmitting data between a CNC system and an ERP system, if the CNC system supports the OPC UA protocol, an OPC UA adapter is selected; if not, the Modbus protocol is selected.

[0038] When data formats do not match, the adaptation protocol converts the data according to the target system's format requirements. For example, it converts temperature data from the source system from Modbus format to JSON format and transmits it to the target system via a web service.

[0039] During data transmission, the system verifies the data (such as data integrity, validity, and compliance) to ensure data security during transmission. At the same time, the system uses the adaptation protocol to perform data encryption and authentication operations to ensure data security.

[0040] The application service layer provides query, visualization, and analysis services for integrated data. This layer allows users to query data from diverse business systems and conduct unified, real-time data analysis. Through charts and dashboards, the system intuitively presents complex production and business data to managers, enabling them to make timely decisions. This layer also supports web, mobile, and API access, ensuring widespread data sharing and utilization.

[0041] The data governance layer ensures data quality, security, and traceability during the data integration process. Data cleansing, deduplication, and consistency checks ensure the accuracy and integrity of integrated data. Data security management ensures data security during transmission and storage through encryption, authentication, and access control. Through real-time monitoring and logging, the data governance layer ensures transparency of the integration process and meets compliance requirements.

[0042] The innovation of the present invention lies in the introduction of a goal-oriented data integration method and an intelligent rule engine system. Through the goal-oriented method, the system adopts ontology-based target modeling and bidirectional mapping algorithm to clarify each data conversion target and optimize and adjust it during the entire data flow process. The intelligent rule engine system adopts a three-tier architecture design, including a rule storage layer, a rule execution layer and a rule management layer. It can not only support the definition of static rules, but also realize dynamic rule generation through machine learning algorithms, automatically discover rule patterns from historical data, and greatly improve the flexibility and scalability of data integration. The combination of the adaptation mechanism and the rule engine system ensures seamless integration between the source system and the target system, so that heterogeneous data sources can be efficiently processed and meet the diverse needs in complex business scenarios. The technical solution of the present invention is suitable for data integration of multiple business systems such as ERP, CRM, MES, SCM, etc. of an enterprise. In a typical application scenario, the inventory data in the ERP system and the production data in the MES system are integrated in real time and transmitted to other business platforms through the service bus to ensure real-time updating and efficient management of data. For example, in the manufacturing industry, the operating hours of production equipment and workers' operation data will be recorded by the MES system. After being converted into a goal-oriented method, the system can calculate production costs and energy consumption in real time, and integrate this information into the ERP system to provide decision support for management.

[0043] This paper proposes a goal-oriented data integration technology for multi-source heterogeneous business systems. Combining ETL technology, an intelligent rule engine system, a service bus architecture, and data governance mechanisms, this technology creates a flexible, reliable, and efficient data integration framework. This framework efficiently integrates multi-source heterogeneous data, meeting the diverse needs of complex business scenarios and ultimately improving the value and application efficiency of data.

[0044] like Figure 1As shown, the goal-oriented multi-source heterogeneous business system data integration technology of the present invention includes five levels: data integration layer, goal-oriented layer, data governance layer, application service layer and data storage and database.

[0045] like Figure 1 As shown in the figure, the data integration layer is primarily responsible for extracting data from various source systems and performing transformation, mapping, and cleansing. The data integration layer includes a data acquisition module, a data mapping and cleansing module, a multi-source data abstraction layer, and a unified interface layer. The data acquisition module extracts data from diverse systems such as enterprise resource planning (ERP), manufacturing execution systems (MES), and customer relationship management (CRM) systems through various communication protocols (such as APIs, Modbus, and RESTful APIs). The data transformation module converts data into a unified data model based on the goals and rules provided by the goal-oriented layer, ensuring that the data format and semantics meet the requirements of downstream systems. The data cleansing and validation module processes the collected data, removing redundant and abnormal data to ensure data integrity and accuracy.

[0046] like Figure 1 As shown in Figure 2, the multi-source data abstraction layer maps parsed data from different sources into a unified data model. By building a unified data model (UDM), a standardized representation of stamping equipment operating parameters, order progress, and material status is achieved, laying the foundation for subsequent processing.

[0047] like Figure 1 As shown in Figure 1, the data mapping and cleaning module processes the collected data. For example, it maps the operating parameters recorded by different devices into a unified standard parameter table and cleans null values ​​and abnormal data in sensor data to ensure data integrity and accuracy.

[0048] like Figure 1 As shown in the figure, the goal-oriented layer integrates multi-source data based on the stamping line's operational goals, such as improving equipment utilization or optimizing material scheduling. For example, it adjusts equipment operating parameters based on order requirements and optimizes production plans based on inventory status. It also defines data flow target rules and guides data conversion and processing.

[0049] like Figure 1 As shown in Figure 1, the application service layer provides decision support, visualization, and real-time query through the query module, data analysis module, and data visualization module. It provides real-time data query and visualization analysis across systems to support management decisions.

[0050] like Figure 1 As shown in the figure, the data governance layer is connected with the data integration layer and application service layer through the data cleaning and consistency check module, security management module, and monitoring and audit module to ensure data quality, compliance and security.

[0051] like Figure 1 As shown, data storage and databases store data in relational databases, NoSQL databases, or time series databases through database storage modules to support subsequent real-time queries and analysis.

[0052] like Figure 2 As shown, the operation process of the present invention starts from the data source parsing module, which first collects multi-source data from the stamping production line, including real-time parameters collected by equipment sensors, order status recorded by the MES system, and material information in the ERP system.

[0053] like Figure 2 As shown in Figure 2, the multi-source data abstraction layer standardizes data from different sources. For example, it matches the operating data of stamping equipment with order requirements to generate unified production status information.

[0054] like Figure 2 As shown, the data mapping and cleaning module maps and cleans collected device data. For example, different devices may record the same parameters but in different units. The system standardizes these to standard units. It also cleans any abnormal data collected, such as removing duplicate records or correcting erroneous values.

[0055] like Figure 2 As shown in Figure 2, the goal-oriented layer dynamically adjusts the data fusion strategy based on the optimization goals of the stamping production line. For example, when shortening the production cycle, it prioritizes the allocation of materials with sufficient inventory and adjusts equipment operating parameters to improve production efficiency.

[0056] like Figure 2 As shown, the unified interface layer provides the integrated data to the production scheduling system or real-time monitoring system through APIs or message queues. For example, it can push the operating status of stamping equipment, order progress, and material status in real time, allowing schedulers to keep abreast of production conditions and make optimization adjustments.

[0057] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A goal-oriented multi-source heterogeneous business system, characterized by: include: The data integration layer extracts data from various source systems and transforms, maps, and cleanses it; The goal-oriented layer integrates multi-source data according to the production line's operating goals; The data governance layer ensures the accuracy and integrity of integrated data through data cleaning, deduplication, and consistency checks; Data security management ensures the security of data during transmission and storage through encryption, authentication and permission control; Application service layer, which provides query, visualization, and analysis of integrated data; Data storage and databases: storing data in relational databases, NoSQL databases, or time series databases to support subsequent real-time queries and analysis; The rule engine system includes a rule storage layer, a rule execution layer, and a rule management layer. The rule representation technology uses RuleML to represent business rules. The dynamic rule generation mechanism is based on a machine learning algorithm, automatically discovering rule patterns from historical data. The rule conflict resolution strategy adopts a mechanism based on priority and specificity, and uses a Git-like version control mechanism to manage rule changes.

2. The target-oriented multi-source heterogeneous business system according to claim 1, characterized in that: The data integration layer includes: Data acquisition module, which extracts data from different systems through various communication protocols; The data mapping module converts data into a unified data model based on the goals and rules provided by the goal-oriented layer, so that the data format and semantics can adapt to the requirements of downstream systems; The data cleaning module processes the collected data, removes redundant and abnormal data, and ensures the integrity and accuracy of the data; The multi-source data abstraction layer maps the parsed data from different sources into a unified data model, and achieves standardized representation of the data by building a unified data model.

3. The target-oriented multi-source heterogeneous business system according to claim 1, characterized in that: The goal-oriented layer adopts an ontology-based goal modeling method to decompose business goals into a set of quantifiable sub-goals; the system uses OWL to define the goal ontology model, and each business goal is represented as a goal class, which includes goal ID, priority, dependency and quantitative indicator attributes; the system uses a bidirectional mapping algorithm to construct a goal-data mapping matrix to clarify the correlation between each data element and the business goal; the goal-driven data conversion engine implements the data processing flow based on a directed acyclic graph, and dynamically adjusts the node execution order and parameters according to active business goals; the goal adaptation feedback mechanism adopts closed-loop control theory, and automatically adjusts the processing parameters by real-time monitoring of the deviation between the processing results and the expected goals.

4. A data integration method based on the target-oriented multi-source heterogeneous business system according to claim 1, characterized in that: Including steps: Step 1: Match data request with protocol; Step 2: Data acquisition and conversion; Step 3: Convert data into historical data; Step 4: Data adaptation and transmission; Step 5: Data storage and processing; Step 6: Perform real-time data processing and dynamic adjustment.

5. The method according to claim 4, characterized in that Step one includes: Send data requests to the source system through the adaptation protocol, clarify the data type and format to be obtained, and use a signature-based protocol identification algorithm to automatically identify the protocol type supported by the source system by analyzing the data packet header and transmission characteristics; maintain a protocol feature library containing feature vectors of industrial and enterprise protocols; this request message includes a description of the required data, data protocol requirements, and the selection of transmission methods; generate a request message to ensure that the requested data is consistent with the protocol requirements of the source system, and generate a request message that complies with the target protocol specifications based on the template engine. The system stores message templates for various protocols and dynamically fills in parameters according to needs.

6. The method according to claim 4, characterized in that Step 2 includes: Obtain raw data from the source system according to the selected protocol; Through the adaptation protocol, data is converted into a format acceptable to the target system; during the conversion, the data fields in the source system are mapped so that the semantics of the data can be correctly understood in the target system.

7. The method according to claim 4, characterized in that Step three includes: converting real-time data into historical data through the rule engine system, and using linear interpolation and exponential smoothing algorithms to supplement and correct missing and anomalies in the data. For time series data, the system uses the ARIMA model to predict missing values ​​and uses median filtering to correct detected anomalies. Complex data calculations are performed through the associated data calculation framework based on the rule engine system, maintaining a business formula library that includes calculation rules for different fields and supports dynamic parameter adjustment.

8. The method according to claim 4, characterized in that Step 4 includes: After the data format and protocol are converted, the adaptation protocol enables the data to be smoothly transmitted to the target system. The adaptation protocol includes data format conversion, data transmission security and protocol selection; Data is transmitted to the target system through the network. During the transmission process, the adaptation protocol ensures the integrity and correctness of the data. After the target system receives the data, it processes and stores it.

9. The method according to claim 4, characterized in that Step five includes: The target system stores the data in the corresponding database according to the type and purpose of the data; The stored data is queried, analyzed and displayed through the interface provided by the application service layer.

10. The method according to claim 4, characterized in that Step six includes: The system processes data in real time and displays it through dashboards and charts; The rule engine system automatically adjusts data processing strategies according to changes in business needs and goals. The rule engine system dynamically generates new rules based on real-time data and business needs to ensure that the data integration process is always consistent with the goals.

Images