Enterprise data integration method, device and equipment based on unified data access and adaptation, and medium

Abstract

The application discloses an enterprise data integration method and device based on unified data access and adaptation, equipment and medium, relates to the field of computer, and comprises the following steps: obtaining a target configuration form for pushing or pulling data sent by a user end at a preset interface, parsing the target configuration form through a preset interface to obtain a target data access mode, a target data receiving type and target configuration content required by the user end; according to the target data access mode and the target data receiving type obtained through parsing, matching a corresponding target adapter by using an adapter scheduling engine, establishing a connection with the user end, and obtaining corresponding initial data based on the target configuration content through the target adapter; performing data processing and format conversion operations on the initial data to generate processed data meeting preset data integration conditions, and integrating the processed data into a preset database of an enterprise data integration system. The two heterogeneous access modes of pushing and pulling are unified, abstracted, centrally configured and intelligently adapted.

Description

Technical Field

[0001] This invention relates to the field of computers, and in particular to enterprise data integration methods, apparatus, equipment and media based on unified data access and adaptation. Background Technology

[0002] Enterprises face significant challenges in building data integration platforms due to the diversification of data sources and the heterogeneity of access methods. Specifically, data access typically falls into two fundamentally different modes: "push" and "pull." In the "push" mode, the data provider acts as the client, and the platform, acting as the server, needs to provide endpoints such as TCP (Transmission Control Protocol) and MQTT (Message Queuing Telemetry Transport) for writing.

[0003] In the "pull" mode, the platform needs to act as a client and actively connect to the TCP, MQTT and other services provided by the data provider to consume the data.

[0004] Currently, these two models are typically implemented independently and in isolation: Technology stack fragmentation: Two different sets of code and configuration processes need to be developed to handle server-side listening and client-side connections respectively, resulting in system complexity and high development and maintenance costs.

[0005] Fragmented management entry points: Operation and maintenance personnel need to manage push access points and pull tasks separately in different functional modules or configuration interfaces, lacking a unified management view and operation experience.

[0006] High technical threshold: Users must have a deep understanding of the differences between the roles of clients and servers in network communication and the details of various protocols in order to configure and maintain them correctly, which greatly increases the difficulty of use and the risk of errors.

[0007] In summary, how to unify, centrally configure, and intelligently adapt the two heterogeneous access modes of "push" and "pull" is an urgent problem to be solved. Summary of the Invention

[0008] In view of this, the purpose of this invention is to provide an enterprise data integration method, apparatus, device, and medium based on unified data access and adaptation, capable of unifying, centrally configuring, and intelligently adapting the two heterogeneous access modes of "push" and "pull". The specific solution is as follows: Firstly, this application provides an enterprise data integration method based on unified data access and adaptation, applied to an enterprise data integration system, including: The system retrieves the target configuration form sent by the user terminal for pushing or pulling data through a preset interface, and parses the target configuration form through a preset interface to obtain the target data access method, target data reception type and corresponding target configuration content required by the user terminal. Based on the target data access method and target data reception type obtained from the parsing, the adapter scheduling engine is used to match the corresponding target adapter and establish a connection with the user terminal so as to obtain the corresponding initial data through the target adapter based on the target configuration content; The initial data is processed and converted to generate processed data that meets the preset data integration conditions, and the processed data is integrated into the preset database of the enterprise data integration system.

[0009] Optionally, the step of matching the corresponding target adapter using the adapter scheduling engine based on the parsed target data access method and target data reception type includes: Obtain a preset mapping table; the preset mapping table is a mapping table used to match the corresponding adapter according to two dimensions: access method and reception type; Based on the target data reception type and the preset mapping table, determine the corresponding target communication protocol type; Determine the corresponding data flow direction based on the target data access method; Using the adapter scheduling engine, a corresponding target adapter is matched based on the target communication protocol type, the data flow direction, and the preset mapping table.

[0010] Optionally, the method further includes: Based on the user-side protocol, add requirements to obtain new communication protocol types; Create a new target adapter corresponding to the new communication protocol type; The mapping relationship between the new communication protocol type and the corresponding new target adapter is added to the preset mapping table to update the preset mapping table, and the corresponding access data policy and data retrieval policy are determined based on the new communication protocol type.

[0011] Optionally, the target data access method includes a data push method or a data pull method. The data push method is that the enterprise data integration system receives data sent by the user terminal, and the data pull method is that the enterprise data integration system actively pulls data from the user terminal.

[0012] Optionally, establishing a connection with the user terminal to obtain corresponding initial data based on the target configuration content through the target adapter includes: If the target data access method is data push, the adapter scheduling engine is used to call and initialize the preset push mode adapter, and a connection request is sent to the user terminal. After connecting to the user terminal, the access data policy corresponding to the preset access mode adapter is determined based on the target configuration content. Based on the access data policy, the first initial data sent by the user terminal is listened to and received through the preset port of the target adapter.

[0013] Optionally, establishing a connection with the user terminal to obtain corresponding initial data based on the target configuration content through the target adapter includes: If the target data access method is a data retrieval method, the adapter scheduling engine is used to call and initialize the preset retrieval mode adapter, and a connection request is sent to the user terminal. Based on the target configuration content, the data retrieval strategy corresponding to the preset retrieval mode adapter is determined, so as to obtain the second initial data according to the preset data source address in the target configuration form based on the data retrieval strategy.

[0014] Optionally, the step of performing data processing and format conversion operations on the initial data to generate processed data that meets preset data integration conditions includes: The initial data is imported into a preset data processing pipeline to perform format standardization conversion, abnormal data cleaning, and data compliance verification on the initial data, generating processed data that meets preset business call conditions. The processed data is then persistently stored and integrated into a preset database so that the data in the preset database can be delivered to the corresponding enterprise business for application.

[0015] Secondly, this application provides an enterprise data integration device based on unified data access and adaptation, applied to an enterprise data integration system, comprising: The form parsing module is used to obtain the target configuration form sent by the user terminal for pushing or pulling data on a preset interface, and to parse the target configuration form through a preset interface to obtain the target data access method, target data receiving type and corresponding target configuration content required by the user terminal. The data determination module is used to match the corresponding target adapter using the adapter scheduling engine according to the parsed target data access method and target data reception type, and establish a connection with the user terminal so as to obtain the corresponding initial data through the target adapter based on the target configuration content; The data management module is used to perform data processing and format conversion operations on the initial data, generate processed data that meets the preset data integration conditions, and integrate the processed data into the preset database of the enterprise data integration system.

[0016] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the enterprise data integration method based on unified data access and adaptation as described above.

[0017] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the enterprise data integration method based on unified data access and adaptation as described above.

[0018] In summary, this application first obtains a target configuration form sent by the user terminal for pushing or pulling data on a preset interface. It then parses the target configuration form through a preset interface to obtain the target data access method, target data reception type, and corresponding target configuration content required by the user terminal. Based on the parsed target data access method and target data reception type, it uses an adapter scheduling engine to match the corresponding target adapter and establishes a connection with the user terminal. The target adapter then obtains the corresponding initial data based on the target configuration content. The initial data undergoes data processing and format conversion operations to generate processed data that meets preset data integration conditions, and this processed data is integrated into the preset database of the enterprise data integration system. As described above, the application first obtains a target configuration form from the user terminal for pushing or pulling data on a preset interface. It then parses the form through a preset interface to obtain the data access method, data reception type, and corresponding configuration content required by the user terminal. Based on the parsing results, it uses an adapter scheduling engine to match a target adapter and establish a connection with the user terminal. The adapter obtains initial data according to the configuration content, and then processes and converts the initial data to generate data that meets preset integration conditions and integrates it into the preset database of the enterprise data integration system. This simplifies the configuration and management process for multi-mode data access, reduces technical complexity, and improves the usability and maintainability of the data integration platform. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 This application discloses an architecture diagram of an enterprise data integration system based on unified data access and adaptation. Figure 2This application discloses a flowchart of an enterprise data integration method based on unified data access and adaptation. Figure 3 This application discloses a specific flowchart of an enterprise data integration method based on unified data access and adaptation. Figure 4 This is a schematic diagram of an enterprise data integration device based on unified data access and adaptation disclosed in this application; Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Currently, enterprises face significant challenges in building data integration platforms due to the diversification of data sources and the heterogeneity of access methods. Specifically, data access is typically divided into two fundamentally different modes: "push" and "pull." In the "push" mode, the data provider acts as the client, and the platform, as the server, provides endpoints such as TCP and MQTT for writing. In the "pull" mode, the platform acts as the client, actively connecting to the TCP and MQTT services provided by the data provider to consume data. The implementation of these two modes is usually independent and fragmented: Fragmented technology stacks: Two different sets of code and configuration processes need to be developed to handle server-side listening and client-side connections separately, leading to system complexity and high development and maintenance costs; Dispersed management entry points: Operations personnel need to manage push access points and pull tasks separately in different functional modules or configuration interfaces, lacking a unified management view and operational experience; High technical barriers: Users must have a deep understanding of the differences between the roles of clients and servers in network communication and the details of various protocols to correctly configure and maintain the system, greatly increasing the difficulty of use and the risk of errors. To address the aforementioned technical issues, this application discloses an enterprise data integration method, apparatus, device, and medium based on unified data access and adaptation, which can unify, centrally configure, and intelligently adapt the two heterogeneous access modes of "push" and "pull".

[0023] The system architecture of this invention is as follows: Figure 1The interface is divided into four logical layers from top to bottom: a unified data access configuration interface, a configuration parsing and routing module, an adapter scheduling engine, and a protocol processing layer. The unified data access configuration interface is the sole entry point for user configuration. Its core innovation is providing a single-choice field for "Data Access Method," with options for "Push" and "Pull," unifying the functionality that previously required navigating to different modules into a single form. Users also need to fill in the data source name, select the communication protocol, and other necessary connection parameters. The configuration parsing and routing module receives the complete configuration form from the front end. Parsing the "Access Method": Its core function is to identify whether the user has selected "Push" or "Pull" mode. Generating a unified configuration model: It encapsulates all form parameters into a standardized configuration data object, which explicitly contains the `accessMode="push"` or `accessMode="pull"` field. The adapter scheduling engine is responsible for determining the execution path based on the configuration model. Internally, it maintains a scheduling strategy, which can be implemented using simple if-else logic or a more elegant factory pattern. The workflow is as follows: the engine reads the `accessMode` field value from the configuration model. If the value is "Push", the "Push Adapter" is instantiated and called; if the value is "Pull", the "Pull Adapter" is instantiated and called. Protocol Processing Layer: This is the actual execution layer, called by different adapters. The Push Adapter calls the protocol processing layer to start listening (Bind) on a specific TCP port, MQTT topic, or Kafka topic as a Server. The Pull Adapter calls the protocol processing layer to initiate a connection (Connect) to a remote TCP service, MQTT broker, or Kafka cluster as a Client.

[0024] See Figure 2 As shown, this embodiment of the invention discloses an enterprise data integration method based on unified data access and adaptation, applied to an enterprise data integration system, including: Step S11: Obtain the target configuration form sent by the user terminal for pushing or pulling data on the preset interface, and parse the target configuration form through the preset interface to obtain the target data access method, target data receiving type and corresponding target configuration content required by the user terminal.

[0025] In this embodiment, the system first receives a target configuration form submitted by the user in the preset interface. This form is specifically used to define the relevant parameters for subsequent data push or pull operations. This form carries the user's specific requirements for data interaction, such as the desired data access method, the type of data expected to be received, and the specific configuration rules to be followed. A unified data access configuration model (DataAccessConfig) is created, the core of which is a "data access method" field (receiveMode). Subsequently, the interface of the data access configuration model is called to parse this target configuration form. This interface can identify each field in the form and extract key information. Through the parsing process, the target data access method (receiveMode) required by the user is clarified, which is used to identify the direction of data flow. It should be noted that the target data access method includes a data push method or a data pull method. The data push method is when the enterprise data integration system receives data sent by the user, and the data pull method is when the enterprise data integration system actively pulls data from the user. Simultaneously, the target data reception type (receiveType) can be determined. The target data reception type is an enumeration value "Kafka", "MQTT", or "TCP", used to identify the underlying communication protocol, such as structured data, unstructured data, or messages in a specific format. Furthermore, the parsing result also includes the corresponding target configuration content (configContent), used to store connection parameters with different structures corresponding to receiveType. This involves specific settings such as data source address, authentication credentials, transport protocol parameters, and data filtering conditions, thus fully reconstructing all the user's requirements for data interaction behavior and providing accurate basis for subsequent data processing. It can also obtain transferClusterId and transferTopic, used to define the target destination after data access. It is understandable that the receiveMode field is not only descriptive information at the business level but also plays the role of the sole trigger source for the system's routing decision. When the system starts the access task, it only needs to read the value of this field to determine the direction of the entire subsequent technical process, without requiring the user to provide any additional mode judgment information. Since the configContent configuration content uses JSON format, its internal structure can dynamically change according to different receiveTypes. This design allows a fixed configuration model to adapt to the different parameter requirements of various protocols.For example, when `receiveType` is `KAFKA`, the `configContent` configuration includes fields such as `bootstrapServers` cluster address, `groupId` consumer group ID, and `autoOffsetReset` offset reset policy; when `receiveType` is `TCP`, `configContent` includes fields such as `port` server listening port number, `host` server listening address, `charset` character encoding format, and `maxConnections` maximum concurrent connections. The property definitions for `DataAccessConfig` are as follows: / Data access configuration entity Unified encapsulation of the source, target, and runtime configuration of data access tasks / public class DataAccessConfig { / Primary Key ID / private Long id; / Tenant ID / private Long tenantId; / Data access name / private String name; / Status: 1-Not enabled 2-Enabled / private Integer status; / Data access methods: 1-Push 2-Pull / private Integer receiveMode; / Data reception types: 1-Kafka, 2-MQTT, 3-TCP / private Integer receiveType; / Protocol configuration parameters (JSON format) / private String configContent; / Target Cluster ID / private Long transferClusterId; / Target Topic / private String transferTopic; / / Omit getter and setter methods } Step S12: Based on the parsed target data access method and target data reception type, use the adapter scheduling engine to match the corresponding target adapter and establish a connection with the user terminal, so as to obtain the corresponding initial data through the target adapter based on the target configuration content.

[0026] In this embodiment, after parsing the target data access method and the target data reception type, a preset mapping table is obtained. This preset mapping table is used to match corresponding adapters based on the two dimensions of access method and reception type. Based on the target data reception type and the preset mapping table, the corresponding target communication protocol type is determined. Based on the target data access method, the corresponding data flow direction is determined. Using an adapter scheduling engine, the corresponding target adapter is matched based on the target communication protocol type, the data flow direction, and the preset mapping table. Specifically, a dual mapping (Map...) system is designed. <DataAccessTypeEnum, Map<DataAccessModeEnum,DataAccessConfigHandler> The factory of >) is the preset mapping table. The first layer Key: DataAccessTypeEnum corresponds to the data receiving type and different communication protocols; the second layer Key: DataAccessModeEnum corresponds to the data access method (PUSH, PULL) and the data flow direction; the value: DataAccessConfigHandler is a specific processor instance, responsible for executing the actual access logic. When the system needs to obtain a processor, the adapter scheduling engine can accurately locate it through two Map lookup operations: the first lookup: processorMap.get(type) → obtain the mode-processor mapping table corresponding to the protocol. The second lookup: modeMap.get(mode) → obtain the specific processor instance. The time complexity of this design is O(1), and the lookup performance remains constant regardless of how many protocols and modes the system supports. If there is a unique adapter in the mapping table that matches both the protocol type and the flow direction, it is directly identified as the target adapter; if there are multiple candidates, it is further combined with other parameters in the configuration content for fine-grained matching. Finally, the engine returns the successfully matched adapter instance for use in the subsequent data push and pull process. The corresponding code is as follows: @Component public class DataAccessProcessorFactory { private static final Map <DataAccessTypeEnum, Map<DataAccessModeEnum,DataAccessConfigHandler> > processorMap = new HashMap<>(); public static DataAccessConfigHandler getDataAccessProcessor(DataAccessTypeEnum type, DataAccessModeEnum mode) { Map<DataAccessModeEnum, DataAccessConfigHandler> modeMap =processorMap.get(type); if (modeMap == null) { throw new ServiceException(ErrorEnum.UNKNOWN_DATA_ACCESS_TYPE); } DataAccessConfigHandler processor = modeMap.get(mode); if (processor == null) { / / Exception throwing logic for unknown patterns can be added here. throw new ServiceException(ErrorEnum.UNKNOWN_DATA_ACCESS_MODE); } Return processor; } } In one specific implementation, if the target data access method is a data push method, the adapter scheduling engine calls and initializes a preset push mode adapter, and sends a connection request to the user terminal. After connecting to the user terminal, the access data policy corresponding to the preset access mode adapter is determined based on the target configuration content. Based on the access data policy, the adapter listens for and receives the first initial data sent by the user terminal through a preset port. Specifically, when the target data access method is determined to be a data push method, the adapter scheduling engine calls and initializes a pre-configured push mode adapter. This adapter first actively sends a connection request to the user terminal to establish a reliable data transmission channel. Once the connection is successfully established, the system determines the access data policy that the push mode adapter should follow based on the previously parsed target configuration content. This policy may involve data reception format specifications, verification rules, or caching mechanisms. Subsequently, the target adapter opens its preset communication port and enters a continuous listening state, waiting for the user terminal to actively push data. When the user terminal sends the first initial data according to the agreed policy, the adapter receives this data in real time through the port, thereby completing the data flow process from the user terminal to the system. The entire process ensures that data in push notification scenarios can be captured in a timely and accurate manner and enter subsequent processing stages. Specific workflow: 1. Receive the DataAccessConfig object; 2. Invoke the specific protocol server initiator through the policy pattern; 3. Process specific push business logic according to the specific protocol service; The corresponding code is shown below: public interface DataAccessConfigHandler { / / Verify configuration connectivity void validate(JSONObject configContent); / / Configuration logic for processing queries JSONObject processForQuery(JSONObject configContent); / / Configuration logic for handling additions or changes JSONObject processForSaveOrUpdate(JSONObject configContent, Long id,Long tenantId); / / Configuration logic when enabling or disabling void processForEnableOrDisable(DataAccessConfig dataAccessConfig,Integer status); } In another specific implementation, if the target data access method is a data pull method, the adapter scheduling engine calls and initializes a preset pull mode adapter and sends a connection request to the user terminal. Based on the target configuration content, the data pull strategy corresponding to the preset pull mode adapter is determined, and second initial data is obtained based on the preset data source address in the target configuration form according to the data pull strategy. Specifically, when the target data access method is determined to be a data pull method, the adapter scheduling engine calls and initializes a pre-configured pull mode adapter. This adapter first sends a connection request to the user terminal to establish a control channel for data interaction. After successful connection, the system determines the data pull strategy corresponding to the pull mode adapter based on the previously parsed target configuration content. This strategy may include parameters such as pull frequency, data page size, and timeout retry mechanism. Subsequently, the adapter extracts the preset data source address from the target configuration form, which is usually in the form of a URL, file path, or database connection string. Based on the data retrieval strategy, the adapter proactively initiates data requests to the specified address, such as via HTTP GET commands, JDBC queries, or FTP downloads, to retrieve secondary initial data from the user client or a designated data source. Throughout the process, the adapter handles protocol interactions, authentication, and data integrity verification, ensuring that the retrieved data conforms to the system's expected format, and ultimately hands the data over to subsequent modules for processing. Specific workflow: 1. Receive the DataAccessConfig object; 2. Invoke the specific protocol server initiator through the policy pattern; 3. Handle specific fetching business logic according to the specific protocol service.

[0027] The corresponding code is shown below: public interface DataAccessConfigHandler { / / Verify configuration connectivity void validate(JSONObject configContent); / / Configuration logic for processing queries JSONObject processForQuery(JSONObject configContent); / / Configuration logic for handling additions or changes JSONObject processForSaveOrUpdate(JSONObject configContent, Long id,Long tenantId); / / Configuration logic when enabling or disabling void processForEnableOrDisable(DataAccessConfig dataAccessConfig,Integer status); } Furthermore, when a new communication protocol needs to be added, the system first obtains the new communication protocol type based on the user's protocol addition request; creates a corresponding new target adapter based on the new communication protocol type; adds the mapping relationship between the new communication protocol type and the corresponding new target adapter to the preset mapping table to update the preset mapping table; and determines the corresponding access data policy and data retrieval policy based on the new communication protocol type. Specifically, when the system faces a new communication protocol expansion requirement, it first obtains the new communication protocol type to be added based on the protocol addition request proposed by the user. For this new communication protocol type, a corresponding new target adapter is developed and created. This adapter needs to implement core functions such as data transmission and reception, connection management, and format conversion under this protocol. After the adapter is created, the mapping relationship between the new communication protocol type and the new target adapter is added to the preset mapping table, thereby updating the original mapping table structure so that the system can identify and match the newly added protocol type. Simultaneously, based on the new communication protocol type, the system also needs to determine its corresponding access data policy and data retrieval policy. The former defines how to receive and parse data when it is pushed to the system via the protocol, while the latter specifies the request method, parameter encapsulation, and response processing rules that the system should follow when actively retrieving data via the protocol. After completing the above steps, the system can, at runtime, use the adapter scheduling engine to find the corresponding adapter from the updated mapping table based on the communication protocol type configured on the user end, and complete the data interaction according to the corresponding data policy.

[0028] Step S13: Perform data processing and format conversion operations on the initial data to generate processed data that meets the preset data integration conditions, and integrate the processed data into the preset database of the enterprise data integration system.

[0029] In this embodiment, the initial data is imported into a preset data processing pipeline to perform format standardization conversion, abnormal data cleaning, and data compliance verification. This generates processed data that meets preset business call conditions. The processed data is then persistently stored and integrated into a preset database, enabling delivery of data from the preset database to the corresponding enterprise business for application. Specifically, after entering the preset data processing pipeline, the initial data undergoes multiple processing stages. First, format standardization conversion is performed, transforming initial data from different sources with different structures or encodings into a unified standard format agreed upon by the system, such as a unified timestamp format, character encoding, or field naming conventions, ensuring consistency and readability of data expression. Next, abnormal data cleaning is performed, identifying and removing invalid values, duplicate records, values ​​exceeding reasonable ranges, or incomplete fields. Simultaneously, missing values ​​are filled or marked according to preset rules, thereby eliminating the impact of noisy data on subsequent processing. After cleaning, the system performs compliance verification, checking the legality of the data according to business and regulatory requirements. For example, it verifies whether sensitive fields are anonymized, whether the data range meets agreed thresholds, and whether required fields exist. Data that does not comply with the verification rules is intercepted or repaired. The data generated after the above processing meets the preset business call conditions. The system persists this data to disk or a distributed storage system and further integrates it into a preset database. This database, after structural optimization and index configuration, can efficiently respond to the query and call needs of various business lines within the enterprise, and ultimately delivers the data to the corresponding enterprise business systems for practical application.

[0030] As described above, this embodiment of the application obtains the target configuration form used by the user terminal for data push or retrieval through a preset interface, parses it through a preset interface to obtain the data access method, data reception type, and corresponding configuration content required by the user terminal, and then uses an adapter scheduling engine to match the target adapter and establish a connection with the user terminal based on the parsing results. Initial data is obtained through the adapter according to the configuration content, and then data processing and format conversion are performed on the initial data to generate data that meets the preset integration conditions and integrate it into the preset database of the enterprise data integration system. This simplifies the configuration and management process of multi-mode data access, reduces technical complexity, and improves the usability and maintainability of the data integration platform.

[0031] As can be seen from the previous embodiment, this application discloses an enterprise data integration method based on unified data access and adaptation, which can unify, centrally configure, and intelligently adapt the two heterogeneous access modes of "push" and "pull". Next, we will discuss one such method... Figure 3 The enterprise data integration method based on unified data access and adaptation is explained in detail.

[0032] Suppose a company needs to pull real-time device operation data from IoT devices in a factory and integrate it into its data platform system. The operations and maintenance personnel fill out the target configuration form for this data task on the system's preset configuration interface, selecting "Pull" as the data access method, "MQTT protocol" as the data receiving type, and entering the corresponding MQTTBroker address, client ID, subscription topic, and other configuration information. After parsing this target configuration form through the preset interface, the system clearly understands that the user wants to obtain device temperature and vibration data via pull, using the MQTT protocol, and the specific connection parameters. Next, the adapter scheduling engine automatically matches and activates the target adapter specifically responsible for MQTT pull based on this information, actively establishing a connection with the factory's MQTT Broker, subscribing to messages according to the configured topic, and starting to pull real-time initial device data. After obtaining this raw data, the system performs data cleaning, deduplication, and format standardization, such as converting JSON-formatted temperature and vibration data into a standardized data model and supplementing timestamps and device ID metadata. Finally, the processed data that meets the integration requirements is stored in the enterprise data integration system's preset database for subsequent analysis or monitoring platform use.

[0033] See Figure 4 As shown, this embodiment of the invention discloses an enterprise data integration device based on unified data access and adaptation, applied to an enterprise data integration system, comprising: The form parsing module 11 is used to obtain the target configuration form sent by the user terminal for pushing or pulling data on a preset interface, and to parse the target configuration form through a preset interface to obtain the target data access method, target data receiving type and corresponding target configuration content required by the user terminal. The data determination module 12 is used to match the corresponding target adapter using the adapter scheduling engine according to the parsed target data access method and target data reception type, and establish a connection with the user terminal so as to obtain the corresponding initial data through the target adapter based on the target configuration content; Data management module 13 is used to perform data processing and format conversion operations on the initial data, generate processed data that meets preset data integration conditions, and integrate the processed data into the preset database of the enterprise data integration system.

[0034] As described above, this application obtains the target configuration form used by the user terminal for data push or retrieval through a preset interface, parses it through a preset interface to obtain the data access method, data reception type, and corresponding configuration content required by the user terminal, and then uses an adapter scheduling engine to match the target adapter and establish a connection with the user terminal based on the parsing results. Initial data is then obtained through the adapter according to the configuration content, followed by data processing and format conversion to generate data that meets preset integration conditions and integrate it into the preset database of the enterprise data integration system. This simplifies the configuration and management process for multi-mode data access, reduces technical complexity, and improves the usability and maintainability of the data integration platform.

[0035] In some specific implementations, the data determination module 12 may specifically include: A mapping table acquisition unit is used to acquire a preset mapping table; the preset mapping table is a mapping table used to match the corresponding adapter according to two dimensions: access method and reception type. The protocol type determination unit is used to determine the corresponding target communication protocol type based on the target data reception type and the preset mapping table; A direction determination unit is used to determine the corresponding data flow direction according to the target data access method; The adapter matching unit is used to match the corresponding target adapter using the adapter scheduling engine based on the target communication protocol type, the data flow direction and the preset mapping table.

[0036] In some specific implementations, the enterprise data integration device based on unified data access and adaptation may further include: The protocol type acquisition module is used to acquire new communication protocol types based on the user's protocol addition requirements; The adapter creation module is used to create a new target adapter corresponding to the new communication protocol type. The mapping table update module is used to add the mapping relationship between the new communication protocol type and the corresponding new target adapter to the preset mapping table to update the preset mapping table, and determine the corresponding access data policy and data retrieval policy based on the new communication protocol type.

[0037] In some specific implementations, the target data access method includes a data push method or a data pull method. The data push method is that the enterprise data integration system receives data sent by the user terminal, and the data pull method is that the enterprise data integration system actively pulls data from the user terminal.

[0038] In some specific implementations, the data determination module 12 may specifically include: The initial data receiving unit is configured to, if the target data access method is a data push method, use the adapter scheduling engine to call and initialize a preset push mode adapter, and send a connection request to the user terminal. After connecting to the user terminal, it determines the access data policy corresponding to the preset access mode adapter based on the target configuration content, and listens for and receives the first initial data sent by the user terminal through the preset port of the target adapter based on the access data policy.

[0039] In some specific implementations, the data determination module 12 may specifically include: The initial data acquisition unit is used to, if the target data access method is a data retrieval method, utilize the adapter scheduling engine to call and initialize a preset retrieval mode adapter, send a connection request to the user terminal, determine the data retrieval strategy corresponding to the preset retrieval mode adapter based on the target configuration content, and obtain second initial data based on the preset data source address in the target configuration form according to the data retrieval strategy.

[0040] In some specific implementations, the data management module 13 may specifically include: The data storage unit is used to import the initial data into a preset data processing pipeline to perform format standardization conversion, abnormal data cleaning and data compliance verification on the initial data, generate processed data that meets preset business call conditions, and integrate the processed data into a preset database after persistent storage, so that the data in the preset database can be delivered to the corresponding enterprise business for application.

[0041] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the enterprise data integration method based on unified data access and adaptation disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0042] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0043] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon can include an operating system 221, computer programs 222, etc., and the storage method can be temporary storage or permanent storage.

[0044] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing an enterprise data integration method based on unified data access and adaptation, which is executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0045] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed enterprise data integration method based on unified data access and adaptation. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0046] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0047] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0048] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0049] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0050] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for enterprise data integration based on unified data access and adaptation, characterized in that, Applications in enterprise data integration systems include: The system retrieves the target configuration form sent by the user terminal for pushing or pulling data through a preset interface, and parses the target configuration form through a preset interface to obtain the target data access method, target data reception type and corresponding target configuration content required by the user terminal. Based on the target data access method and target data reception type obtained from the parsing, the adapter scheduling engine is used to match the corresponding target adapter and establish a connection with the user terminal so as to obtain the corresponding initial data through the target adapter based on the target configuration content; The initial data is processed and converted to generate processed data that meets the preset data integration conditions, and the processed data is integrated into the preset database of the enterprise data integration system.

2. The method for enterprise data integration based on unified data access and adaptation according to claim 1, characterized in that, The step of matching the corresponding target adapter using the adapter scheduling engine based on the parsed target data access method and target data reception type includes: Obtain a preset mapping table; the preset mapping table is a mapping table used to match the corresponding adapter according to two dimensions: access method and reception type; Based on the target data reception type and the preset mapping table, determine the corresponding target communication protocol type; Determine the corresponding data flow direction based on the target data access method; Using the adapter scheduling engine, a corresponding target adapter is matched based on the target communication protocol type, the data flow direction, and the preset mapping table.

3. The method for enterprise data integration based on unified data access and adaptation according to claim 2, characterized in that, Also includes: Based on the user-side protocol, add requirements to obtain new communication protocol types; Create a new target adapter corresponding to the new communication protocol type; The mapping relationship between the new communication protocol type and the corresponding new target adapter is added to the preset mapping table to update the preset mapping table, and the corresponding access data policy and data retrieval policy are determined based on the new communication protocol type.

4. The method for enterprise data integration based on unified data access and adaptation according to claim 1, characterized in that, The target data access method includes a data push method or a data pull method. The data push method is that the enterprise data integration system receives data sent by the user terminal, and the data pull method is that the enterprise data integration system actively pulls data from the user terminal.

5. The method for enterprise data integration based on unified data access and adaptation according to claim 4, characterized in that, The step of establishing a connection with the user terminal to obtain corresponding initial data based on the target configuration content through the target adapter includes: If the target data access method is data push, the adapter scheduling engine is used to call and initialize the preset push mode adapter, and a connection request is sent to the user terminal. After connecting to the user terminal, the access data policy corresponding to the preset access mode adapter is determined based on the target configuration content. Based on the access data policy, the first initial data sent by the user terminal is listened to and received through the preset port of the target adapter.

6. The method for enterprise data integration based on unified data access and adaptation according to claim 4, characterized in that, The step of establishing a connection with the user terminal to obtain corresponding initial data based on the target configuration content through the target adapter includes: If the target data access method is a data retrieval method, the adapter scheduling engine is used to call and initialize the preset retrieval mode adapter, and a connection request is sent to the user terminal. Based on the target configuration content, the data retrieval strategy corresponding to the preset retrieval mode adapter is determined, so as to obtain the second initial data according to the preset data source address in the target configuration form based on the data retrieval strategy.

7. The unified data access and adaptation based enterprise data integration method of claim 1, wherein, The step of performing data processing and format conversion operations on the initial data to generate processed data that meets preset data integration conditions includes: The initial data is imported into a preset data processing pipeline to perform format standardization conversion, abnormal data cleaning, and data compliance verification on the initial data, generating processed data that meets preset business call conditions. The processed data is then persistently stored and integrated into a preset database so that the data in the preset database can be delivered to the corresponding enterprise business for application.

8. An enterprise data integration device based on unified data access and adaptation, characterized in that, Applications in enterprise data integration systems include: The form parsing module is used to obtain the target configuration form sent by the user terminal for pushing or pulling data on a preset interface, and to parse the target configuration form through a preset interface to obtain the target data access method, target data receiving type and corresponding target configuration content required by the user terminal. The data determination module is used to match the corresponding target adapter using the adapter scheduling engine according to the parsed target data access method and target data reception type, and establish a connection with the user terminal so as to obtain the corresponding initial data through the target adapter based on the target configuration content; The data management module is used to perform data processing and format conversion operations on the initial data, generate processed data that meets the preset data integration conditions, and integrate the processed data into the preset database of the enterprise data integration system.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the enterprise data integration method based on unified data access and adaptation as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the enterprise data integration method based on unified data access and adaptation as described in any one of claims 1 to 7.

Images