A data integration system and method

Through data integration systems and methods, data calls, local services and remote callback components are used to process multiple data interfaces and types, and data loss and real-time problems in the existing technology are solved, cross-platform data source acquisition and data compensation are realized, and the Development work has improved data processing efficiency.

CN113190513BActive Publication Date: 2024-12-03GLODON CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202110529883.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-05-14
Publication Date
2024-12-03
Estimated Expiration
2041-05-14

AI Technical Summary

Technical Problem

In the face of multi-data streams, existing data integration tools have caused data loss problems due to batch processing methods, especially when changing data capture, it is difficult to achieve real-time cross-platform data source acquisition and data exchange.

Method used

Provide a data integration system and method, including data call components, local service components and remote callback components, obtain data through the protocol of the data interface, and convert and process data formats. The integrated management module is used for data compensation and permission management, simplifying The work of developers meets the needs of multi-data interfaces, multi-data types and inconsistent data standards.

Benefits of technology

Real-time data exchange and cross-platform data source collection are realized, data loss is avoided, developers' workflow is simplified, data compensation is supported, and data processing efficiency and standardization are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113190513B_ABST
    Figure CN113190513B_ABST
Patent Text Reader

Abstract

The present invention discloses a data integration system and method, which includes: receiving a data integration request; determining a data integration type based on the data integration request; determining a corresponding target format conversion rule in an integrated preset format conversion rule according to the data integration type; and determining the return data of the data integration request according to the data integration type and the target format conversion rule. The present invention meets the business requirements of numerous data interfaces, multiple data types, inconsistent data standards, real-time data exchange, data extraction and push, etc., realizes cross-platform data source collection, and overcomes the problem of data loss; simplifies the work of developers, so that developers do not need to deal with various operations such as data model conversion and log recording, do not need to understand the underlying protocol of the data interface, and only need to focus on the processing of business data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer software technology, and in particular to a data integration system and method. Background Art

[0002] Data integration mainly refers to the process of re-centralizing and unifying the management of business data based on the enterprise's decentralized information systems. The data integration architecture between institutions is an architecture established for multiple cross-enterprises and units to implement one or several business activities. It has the characteristics of complex data networks, high data security requirements, and strong real-time data exchange. It must meet business requirements such as numerous data interfaces, multiple data types, inconsistent data standards, real-time data exchange, and data extraction and push. Existing batch processing tools for data integration need to face more and more data streams. Especially when it comes to changing data, the batch processing method of existing batch processing tools will directly overwrite some information, resulting in data loss. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a data integration system and method to solve the problem of data loss caused by batch processing in the prior art when facing more and more data streams.

[0004] According to a first aspect, an embodiment of the present invention provides a data integration system, including: a data calling component, a local service component and a remote callback component.

[0005] The data calling component is used to obtain corresponding data through the protocol of the data interface;

[0006] The local service component is used to implement data format conversion of service data and corresponding data processing;

[0007] The remote callback component is used to convert the data format of the data obtained by the data calling component and perform corresponding data processing.

[0008] The data integration system provided by the embodiment of the present invention meets business requirements such as numerous data interfaces, multiple data types, inconsistent data standards, real-time data exchange, data extraction and push, realizes cross-platform data source collection, and overcomes the problem of data loss; simplifies the work of developers, so that developers do not need to deal with various operations such as data model conversion and log recording, do not need to understand the underlying protocol of the data interface, and only need to focus on the processing of business data.

[0009] In combination with the first aspect, in a first implementation of the first aspect, the local service component includes: a service data conversion subcomponent, configured to perform format conversion of corresponding service data to implement corresponding processing of the service data;

[0010] The remote callback component includes: a callback data conversion subcomponent, which is used to perform format conversion of corresponding service data to achieve corresponding processing of the service data.

[0011] In combination with the first aspect, in the second implementation of the first aspect, the data integration system provided by this embodiment further includes: an integration management module,

[0012] The integrated management module is used to read the interface data and store the interface log to achieve data compensation for the interface data that failed to process the data;

[0013] The integrated management module is also used to manage user information and data permissions.

[0014] According to a second aspect, an embodiment of the present invention provides a data integration method, including:

[0015] Receive data integration requests;

[0016] determining a data integration type based on the data integration request;

[0017] Determine the corresponding target format conversion rule in the preset format conversion rules of the integration according to the data integration type;

[0018] The return data of the data integration request is determined according to the data integration type and the target format conversion rule.

[0019] The data integration method provided by the embodiment of the present invention integrates preset format conversion rules through real-time monitoring of interfaces, and determines the corresponding target format conversion rules in the integrated preset format conversion rules according to the data integration type to determine the return data of the data integration request; so as to meet the business needs of numerous data interfaces, multiple data types, inconsistent data standards, real-time data exchange, data extraction and push, etc., to achieve cross-platform data source collection, and overcome the problem of data loss; simplify the work of developers, so that developers do not need to deal with various operations such as data model conversion and logging, do not need to understand the underlying protocol of the data interface, and only need to focus on the processing of business data.

[0020] In combination with the second aspect, in a first implementation of the second aspect, when the data integration request is a service interface request, the data integration type is local service processing, and determining the return data of the data integration request according to the data integration type and the target format conversion rule includes:

[0021] Matching the service interface in the data integration request in the interface registration list, and judging whether the service interface provides service according to the status parameter of the service interface;

[0022] When the service interface provides a service, the service interface request is parsed using the local service process and the target format conversion rule corresponding to the local service process to determine the return data of the service interface request.

[0023] The data integration method provided by an embodiment of the present invention, when the data integration request is a service interface request, determines whether the service is provided through the interface status parameters in the registered interface list, parses the service interface request to determine the return data of the service interface request, thereby ensuring the simplicity and accuracy of the data processing process.

[0024] In combination with the first implementation of the second aspect, in the second implementation of the second aspect, the using the local service processing and the target format conversion rule corresponding to the local service processing to parse the service interface request and determine the return data of the service interface request includes:

[0025] Analyze the service interface request using the target format conversion rule corresponding to the local service processing to determine the data conversion method;

[0026] Perform corresponding business processing on the preset business data instance according to the local service processing and data conversion method to determine the interface data;

[0027] The interface data is converted into a data exchange format using a target format conversion rule corresponding to the local service process, and a conversion result of the data exchange format is determined as return data.

[0028] The data integration method provided in this embodiment parses the service interface request through local service processing to determine the data conversion method, facilitates the determination of interface data through the corresponding business instance, and converts the interface data into a data format recognizable by the requesting end and returns it; it ensures that developers only need to focus on the processing of business data, thereby simplifying the processing flow.

[0029] In combination with the second implementation manner of the second aspect, in the third implementation manner of the second aspect, performing corresponding business processing on the preset business data instance according to the local service processing and data conversion method further includes:

[0030] Perform corresponding business processing on the preset business data instance, and record the service interface where the preset error occurs in the log file;

[0031] In response to the data compensation operation, the service interface request is re-parsed based on the log file.

[0032] The data integration method provided by the embodiment of the present invention can perform data compensation after a data processing error occurs, support reloading and processing of data in different scenarios, and has a simple implementation process, thereby improving data processing efficiency.

[0033] In combination with the second aspect, in a fourth implementation of the second aspect, when the data integration request is a callback interface request, the data integration type is remote callback processing, and determining the return data of the data integration request according to the data integration type and the target format conversion rule includes:

[0034] Matching the callback interface in the data integration request in the interface registration list, and determining whether the callback interface is enabled according to the status parameter of the callback interface;

[0035] When the callback interface has been enabled, the callback interface request is parsed using the remote callback process and the target format conversion rule corresponding to the remote callback process to determine the return data of the callback interface request.

[0036] The data integration method provided by the embodiment of the present invention, when the data integration request is a remote callback request, determines whether to start through the interface status parameters in the registered interface list, parses the remote callback request to determine the return data of the remote callback request, thereby ensuring the simplicity and accuracy of the data processing process.

[0037] In combination with the fourth implementation of the second aspect, in the fifth implementation of the second aspect, the use of remote callback processing and a target format conversion rule corresponding to the remote callback processing to parse the callback interface request and determine the return data of the callback interface request includes:

[0038] Performing data exchange format conversion on the callback interface request by using remote callback processing and target format conversion rules corresponding to the remote callback processing to generate a network request;

[0039] Determine response data according to the network request and the interface protocol of the remote callback processing;

[0040] Record the response data into a log file, and parse the response data using a target format conversion rule corresponding to the remote callback processing;

[0041] The parsing result of the response data and the preset business data instance are subjected to corresponding business processing to determine the return data.

[0042] In combination with the fifth implementation of the second aspect, in the sixth implementation of the second aspect, the use of remote callback processing and a target format conversion rule corresponding to the remote callback processing to convert the data exchange format of the callback interface request to generate a network request includes:

[0043] Insert the attribute value in the preset business data instance corresponding to the remote callback processing into the preset XML template to determine the XML file;

[0044] The callback interface request is converted into a data exchange format using a target format conversion rule corresponding to remote callback processing, and the data model after the data exchange format conversion is combined with an XML file to generate a standard XML file to determine the network request.

[0045] The data integration method provided by the embodiment of the present invention converts the data format of remote callback processing into a standard XML file to determine the network request, facilitate the process of data processing and conversion, and improve the standardization and normativity of data processing.

[0046] In combination with the fifth implementation of the second aspect, in the seventh implementation of the second aspect, determining the response data according to the network request and the interface protocol of the remote callback processing includes:

[0047] Registering the preset interface protocol into the protocol factory class according to the network request;

[0048] The response data is determined in the protocol factory class according to the interface protocol of the remote callback processing.

[0049] In combination with the fifth implementation manner of the second aspect, in the eighth implementation manner of the second aspect, recording the response data into a log file includes:

[0050] When there is preset error log data in the log file, the network request is reconverted in response to a data compensation operation.

[0051] The data integration method provided by the embodiment of the present invention can perform data compensation after a data processing error occurs, support reloading and processing of data in different scenarios, and has a simple implementation process, thereby improving data processing efficiency.

[0052] According to a third aspect, an embodiment of the present invention provides a data integration device, including:

[0053] A data integration interface configuration module, used to manage the data interface and the data interface status, wherein the data interface includes a service interface and a callback interface;

[0054] The data integration interface configuration module is also used to configure parameters of the data interface and send an alarm message when the parameter configuration fails;

[0055] An interface processor management module is used to register and monitor the status of the interface processor;

[0056] The data compensation module is used to trigger the data interface to re-execute the parsing when the data integration request parsing fails.

[0057] According to the fourth aspect, an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the data integration method described in the second aspect or any one of the embodiments of the second aspect by executing the computer instructions.

[0058] According to the fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the data integration method described in the second aspect or any one of the embodiments of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 is a schematic diagram of a data integration system according to an embodiment of the present invention;

[0061] Figure 2 is a flow chart of a data integration method according to an embodiment of the present invention;

[0062] Figure 3 is a flow chart of determining return data of a service interface request according to a preferred embodiment of the present invention;

[0063] Figure 4 is a flow chart of determining return data of a callback interface request according to a preferred embodiment of the present invention;

[0064] Figure 5 is a schematic diagram of a data integration device according to an embodiment of the present invention;

[0065] Figure 6 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0067] With the development of enterprise informatization, messy, repeated and ambiguous data are coming one after another. Enterprises need a master data management system to unify the product information and customer information of the enterprise; they need a data center system to centrally exchange, distribute, schedule and manage the basic data of the enterprise. The necessity and urgency of data integration are self-evident. From the system deployment and business scope of data integration applications, there are mainly three types of architectures: one is a single system data integration architecture, one is an enterprise unified data integration architecture, and one is an inter-institutional data integration architecture. Among them, the inter-institutional data integration architecture is an architecture established for multiple cross-enterprises and units to realize one or several business activities. It has the characteristics of complex data network, high data security requirements, and strong real-time data exchange. The data integration platform is built between multiple enterprise units and must meet business requirements such as numerous data interfaces, multiple data types, inconsistent data standards, real-time data exchange, and data extraction and push.

[0068] The mainstream architecture of data integration is ETL, in which the process of ETL is Extraction-Transformation-Load. After the data source is extracted, it is first transformed, and then the result of the transformation is written to the destination. The biggest limitation of existing tools is that they are designed to "work in batches". This batch loading of data is not suitable when facing more and more data streams and other types of data sources, especially when the latest data needs to be provided as soon as possible. In addition, it is difficult for data batch processing and transformation tools to achieve cross-platform data source collection, especially when it comes to change data capture. When there is a problem with uploading batch data, it is necessary to track the problem, troubleshoot, and resubmit the job as soon as possible. However, this batch processing method may cause data loss because there is more than 24 hours of data in the data warehouse that is called and allocated by the API and may be overwritten.

[0069] In this embodiment, a data integration system is provided, such as Figure 1 As shown, it includes: a data calling component 010 (MyClient), a local service component 020 (AbstractService) and a remote callback component 030 (AbstractCallback), wherein the data calling component 010 is used to obtain corresponding data through the protocol of the data interface; the local service component 020 is used to realize the data format conversion of the service data and the corresponding data processing; the remote callback component 030 is used to convert the data format of the data obtained by the data calling component 010 and the corresponding data processing.

[0070] The data interface protocols include SOAP and REST. When the application is started, MyClient will be automatically registered in the ClientFactory factory, and AbstractCallback will obtain the WsClient instance from the factory according to the protocol used by the data interface. AbstractService encapsulates the data processing process of the local service interface; AbstractCallback encapsulates the data processing process of the remote interface call. When engineers implement the data interface, they only need to inherit the above abstract class and process the business data (POJO). Abstract class AbstractCallback <t>Encapsulates the callback interface processing. AbstractCallback implements synchronous or asynchronous calls through WsClient.

[0071] Specifically, the local service component 020 includes: a service data conversion subcomponent, which is used to perform format conversion of the corresponding service data to achieve corresponding processing of the service data; the remote callback component 030 includes: a callback data conversion subcomponent, which is used to perform format conversion of the corresponding service data to achieve corresponding processing of the service data. In this embodiment, AbstractService defines four subclasses according to the data conversion method: AbstractJsonService, AbstractPojoService, AbstractXmlService and AbstractStringService, which are used to implement specific data conversion operations (toDataModel abstract method). For the POJO data format, AbstractPojoService optimizes the data processing process of AbstractService. AbstractCallback defines three subclasses: AbstractJsonCallback, AbstractXmlCallback and AbstractStringCallback, which are used to implement specific data format conversion operations.

[0072] Specifically, the data integration system provided by this embodiment also includes: an integrated management module, wherein the integrated management module is used to read interface data and store interface logs to achieve data compensation for interface data that fails in data processing; the integrated management module is also used to manage user information and data permissions. Information such as signatures and permissions of user roles can be verified and managed.

[0073] According to an embodiment of the present invention, a data integration method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps can be executed or described in an order different from that shown here.

[0074] The data integration system provided by the embodiment of the present invention can realize data extraction, conversion, push or pull in the data integration process; data compensation, data permission management; real-time interface monitoring, meet the business needs of numerous data interfaces, multiple data types, inconsistent data standards, real-time data exchange, data extraction and push, realize cross-platform data source collection, and overcome the problem of data loss; simplify the work of developers, so that developers do not need to deal with various operations such as data model conversion and logging, do not need to understand the underlying protocol of the data interface, and only need to focus on the processing of business data.

[0075] In this embodiment, a data integration method is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 2 is a flow chart of a data integration method according to an embodiment of the present invention. Figure 2 , the process includes the following steps:

[0076] S11, receiving a data integration request. By real-time monitoring of the interface, a data integration request is received, wherein the data integration request may include a request sent by an external network when a local service interface provides a service, or a request sent by a local sub-process when a remote callback interface enables a callback.

[0077] S12: Determine a data integration type based on the data integration request.

[0078] In this embodiment, different data integration types can be determined according to different data integration requests. An example is used to illustrate that the data integration type is encapsulated through an abstract class. For example, the AbstractService abstract class encapsulates the data processing process of the local service interface; the AbstractCallback abstract class encapsulates the data processing process of the remote interface call. When engineers implement the data interface, they only need to inherit the corresponding abstract class according to the data integration request and then process the business data. It should be noted that this embodiment only illustrates the specific name and form of the abstract class, and the present invention is not limited to this.

[0079] S13, determining a corresponding target format conversion rule from the integrated preset format conversion rules according to the data integration type.

[0080] Select tools that can meet the data models of XML, JSON, POJO, etc. in this embodiment to implement the preset format conversion rules for data conversion; use the Digester tool to convert XML into Java objects. Digester is a component of Apache, apache commons-digester.jar, which can easily generate Java objects from XML files. Digester uses SAX to parse XML files at the bottom layer, so naturally, object conversion is driven by "events", that is, when a specific XML element is identified, a specific action will be performed, such as creating a specific Java object or calling a method of a specific object. The XML elements here are identified according to the matching pattern, and the relevant operations are defined by rules. During the conversion process, Digester maintains an object stack, which can be regarded as a workbench for object conversion, used to store Java objects generated during the conversion or temporarily created for the conversion. After a complete scan of the input XML file, the top element of the object stack is the target object. Since Digester shields the details of SAX parsing, the user only needs to focus on the conversion operation itself, which greatly simplifies the conversion operation.

[0081] XmlBuilder is a general tool that generates standard XML files based on templates. The definition of XmlBuilder template tool: XML template + data model = standard XML file; XmlBuilder is a tool that uses DOM4J technology to convert data models into XML files. The data models supported by XmlBuilder include: POJO, Map, collection, array, etc. Although general template engines such as FreeMarker can create XML, they do not implement the standard specifications of XML. Among them, the language of the XmlBuilder template tool: ${attribute name}, insert the XML template with the corresponding attribute value in the data instance. If the member attribute value of the object instance is an array or collection type, the syntax for generating multiple child nodes (abs) is as follows:

[0082] <#forEach:abs collection="dataList">

[0083] ${ab.a}

[0084] < / #forEach:abs>

[0085] Among them, the tools for implementing the preset format conversion rules for data conversion also include the FastJson processing toolkit. Fastjson is a high-performance and fully functional JSON library written in Java. The Fastjson interface is simple and easy to use, and has been widely used in cache serialization, protocol interaction, Web output, Android client and other application scenarios to achieve mutual conversion between Java objects and JSON formats.

[0086] It should be noted that in this embodiment, an existing perfect tool for implementing preset format conversion rules for data conversion is selected. In practical applications, new tools can be selected according to actual needs to meet different format conversion rules, and this embodiment is not limited to this.

[0087] S14, determining the return data of the data integration request according to the data integration type and the target format conversion rule. In this embodiment, the abstract classes corresponding to different data integration types and the target format conversion rules corresponding to the data integration request determined above are used to process the business data of the corresponding business instance and determine the return data of the data integration request.

[0088] In this embodiment, before the data integration development framework calls this method, that is, before receiving the data integration request, the SignUtils tool class can be used to generate a signature for the data interface. Data interface signature authentication mechanism: The request Header contains _sign and _timpstamp parameters, and the system verifies that _sign is valid and determines that the request is legal.

[0089] Take the following request as an example to illustrate the three steps of generating _sign:

[0090] http: / / xxcom / api / car? type=x&brand=x&series=x

[0091] step1: construct _sign source string

[0092] Request composition name convention:

[0093] api_name: / api / car

[0094] query_string:type=x&brand=x&series=x

[0095] Use & to concatenate the parameter list of api_name and parameter key in the above request sorted by dictionary, where _timestamp is the current timestamp (seconds) used to determine the validity period of the signature (60s).

[0096] Step 2: Add the key private_key to the source string

[0097] This key is only known by both systems: private_key=x does not participate in the sorting (last).

[0098] Step 3: Encrypt the source string with md5 and capitalize it

[0099] toupper(MD5( / api / car&_timestamp=x&brand=x&series=x&type=x&private_key=x))

[0100] AbstractCallback subclasses use this signature strategy by default. The service interface can use @SignValidated for declarative signature verification. Only after the signature verification is passed can the method provided in this embodiment be called to obtain the corresponding return data, further ensuring the security and reliability of data processing.

[0101] The data integration method provided in this embodiment integrates preset format conversion rules through real-time monitoring of interfaces, and determines the corresponding target format conversion rules in the integrated preset format conversion rules according to the data integration type to determine the return data of the data integration request; so as to meet the business needs of numerous data interfaces, multiple data types, inconsistent data standards, real-time data exchange, data extraction and push, etc., to achieve cross-platform data source collection, and overcome the problem of data loss; simplify the work of developers, so that developers do not need to deal with various operations such as data model conversion and logging, do not need to understand the underlying protocol of the data interface, and only need to focus on the processing of business data.

[0102] In this embodiment, a data integration method is provided. Figure 3 is a flow chart of a data integration method according to an embodiment of the present invention. Figure 3 , the process includes the following steps:

[0103] S21, receiving a data integration request.

[0104] For details, please see Figure 2 S11 of the embodiment will not be described in detail here.

[0105] S22: Determine a data integration type based on the data integration request.

[0106] For details, please see Figure 2 S12 of the embodiment will not be described in detail here.

[0107] S23, determining a corresponding target format conversion rule from the integrated preset format conversion rules according to the data integration type.

[0108] For details, please see Figure 2 S12 of the embodiment will not be described in detail here.

[0109] S24, determining the return data of the data integration request according to the data integration type and the target format conversion rule.

[0110] Specifically, when the data integration request is a service interface request, the data integration type is local service processing, and the above S24 may include the following steps:

[0111] S241, matching the service interface in the data integration request in the interface registration list, and determining whether the service interface provides services according to the status parameters of the service interface.

[0112] When any interface is connected to the data integration development framework, the interface information needs to be registered in the interface registration list first, and each interface corresponds to its current running status. When the data integration development framework receives a service interface request, it uses the service interface ID corresponding to the service interface request to match it in the interface registration list. After matching the corresponding service interface, it determines whether the service interface provides services through the status parameters of the service interface. The status parameters of each interface in the interface registration list are changed according to the real-time monitoring of the interface, and the status parameters of the service interface include the state of providing services and the state of not providing services.

[0113] S242, when the service interface provides services, the service interface request is parsed using the local service processing and the target format conversion rules corresponding to the local service processing to determine the return data of the service interface request. Among them, the local service processing is the data processing process of the local service interface implemented by the abstract class AbstractService encapsulated by the data integration type. It should be noted that this embodiment only uses the above abstract class as an example for explanation. In actual applications, it can be adjusted according to actual needs, as long as the data processing process of the local service interface can be implemented, and the present invention is not limited to this.

[0114] In some optional implementations of this embodiment, the above S242 may include the following steps:

[0115] (1) Use the target format conversion rules corresponding to the local service processing to parse the service interface request and determine the data conversion method.

[0116] In this embodiment, in the process of determining the target format conversion rules corresponding to the local service processing, that is, the local service request received by the local service processing is a network request, and the network format is first converted into the data format required for actual business data processing, thereby completing the process of determining the data conversion method.

[0117] (2) Perform corresponding business processing on the preset business data instance according to the local service processing and data conversion method to determine the interface data.

[0118] Taking the above-mentioned abstract class AbstractService for implementing the data processing process of the local service interface as an example, AbstractService defines 4 subclasses according to the data conversion method: AbstractJsonService, AbstractPojoService, AbstractXmlService and AbstractStringService, which are used to implement specific data conversion operations (toDataModel abstract method). When the data format required for actual business data processing is the POJO data format, AbstractPojoService optimizes the data processing process of AbstractService. In this embodiment, after the local service processing and the data conversion method are determined, the preset business instance can be determined. For example, when it is for the POJO data format, the POJO instance needs to be encapsulated, and then the corresponding business processing is performed according to the business logic of the POJO instance, and finally the interface data is obtained. It should be noted that this embodiment is a subclass defined according to the needs of the actual data conversion method, and can be adjusted according to the needs of the actual data conversion method. This embodiment is not limited to this.

[0119] (3) The interface data is converted into a data exchange format using a target format conversion rule corresponding to the local service processing, and the conversion result of the data exchange format is determined as the return data.

[0120] For the local service processing process, after the interface data is determined as mentioned above, the data needs to be returned to the sender of the local service request, that is, the external network end. Therefore, it is necessary to use the corresponding target format conversion rules to convert the interface data into a network format, and return the conversion result of the data exchange format as a Web response to the external network end.

[0121] (4) Perform corresponding business processing on the preset business data instance, and record the service interface where the preset error occurs in the log file. In the process of performing business processing on the preset business data instance, the service interface where the error occurs is recorded in the log file, where the preset error may include: the local service request does not access the corresponding service interface, resulting in service failure; it may also be that after the local service request accesses the corresponding service interface, the service fails in the final data processing process, etc., and some other predetermined situations. It should be noted that the above-mentioned preset errors in this embodiment are set based on actual experience and can be adjusted in real time in actual applications, and the present invention is not limited to this.

[0122] And when recording the log file, you can also choose to record all processing situations, regardless of whether the processing is successful or not. During the data compensation monitoring process, the records in the log file are judged, and when a preset error occurs, an instruction requiring data compensation is issued.

[0123] (5) Re-parse the service interface request based on the log file in response to the data compensation operation. After receiving the data compensation instruction, the user performs the data compensation operation, and the data integration development framework of this embodiment responds to the data compensation operation and attempts to re-parse (the local service interface).

[0124] The data integration method provided in this embodiment integrates preset format conversion rules through real-time monitoring of interfaces, and determines the corresponding target format conversion rules in the integrated preset format conversion rules according to the data integration type to determine the return data of the data integration request; so as to meet the business needs of numerous data interfaces, multiple data types, inconsistent data standards, real-time data exchange, data extraction and push, etc., to achieve cross-platform data source collection, and overcome the problem of data loss; simplify the work of developers, so that developers do not need to deal with various operations such as data model conversion and logging, do not need to understand the underlying protocol of the data interface, and only need to focus on the processing of business data; ensure the simplicity and accuracy of the data processing process.

[0125] In this embodiment, a data integration method is provided. Figure 4 is a flow chart of a data integration method according to an embodiment of the present invention. Figure 4 , the process includes the following steps:

[0126] S31, receiving a data integration request.

[0127] For details, please see Figure 2 S11 of the embodiment will not be described in detail here.

[0128] S32: Determine a data integration type based on the data integration request.

[0129] For details, please see Figure 2 S12 of the embodiment will not be described in detail here.

[0130] S33, determining a corresponding target format conversion rule from the integrated preset format conversion rules according to the data integration type.

[0131] For details, please see Figure 2 S13 of the embodiment will not be described in detail here.

[0132] S34, determining the return data of the data integration request according to the data integration type and the target format conversion rule.

[0133] As an optional implementation of this embodiment, when the data integration request is a callback interface request, the data integration type is remote callback processing, and the above S34 may include the following steps:

[0134] S341, matching the callback interface in the data integration request in the interface registration list, and judging whether the callback interface is enabled according to the state parameter of the callback interface. The details have been described in the previous embodiment, and will not be repeated here. In this embodiment, it is only necessary to judge whether the callback interface is enabled.

[0135] S342, when the callback interface has been enabled, the callback interface request is parsed using the remote callback processing and the target format conversion rules corresponding to the remote callback processing to determine the return data of the callback interface request. Among them, the remote callback processing is the data processing process of the remote callback interface implemented by the abstract class AbstractCallback of the data integration type encapsulation. AbstractCallback defines three subclasses: AbstractJsonCallback, AbstractXmlCallback and AbstractStringCallback, which are used to implement specific data format conversion operations. It should be noted that this embodiment only uses the above abstract class as an example for explanation. In actual applications, it can be adjusted according to actual needs, as long as the data processing process of the remote callback interface can be implemented, and the present invention is not limited to this.

[0136] In some optional implementations of this embodiment, the above S342 may include the following steps:

[0137] (1) Use the remote callback processing and the target format conversion rules corresponding to the remote callback processing to convert the callback interface request into a data exchange format and generate a network request. When the remote callback processing receives a request sent by the sub-process of the data integration development framework, the sub-process request is first converted into the data format required by the external network end of the remote callback processing, and then a network request is generated.

[0138] a. Insert the attribute value in the preset business data instance corresponding to the remote callback processing into the preset XML template to determine the XML file.

[0139] b. Use the target format conversion rules corresponding to the remote callback processing to convert the data exchange format of the callback interface request, combine the data model after the data exchange format conversion with the XML file, and generate a standard XML file to determine the network request.

[0140] In this embodiment, if interface A needs to convert the collection List <pojo>In XML format, the callback interface of the data integration development framework will use XmlBuilder to push the List <pojo>Convert it into XML format, and then use WsClient to push it to the remote interface (external network end), where WsClient is an abstract class for encapsulating the Web external network end. The specific implementation process of XmlBuilder refers to the above embodiment.

[0141] (2) Determine the response data based on the network request and the interface protocol of the remote callback processing.

[0142] a. Register the preset interface protocol to the protocol factory class according to the network request. In java, general class objects are created through the new function, but some classes must be created through their getInstance method, which is called the factory class.

[0143] b. Determine the response data in the protocol factory class according to the interface protocol of the remote callback processing.

[0144] WsClient is an abstract class for encapsulating the external network end of the Web, supporting synchronous and asynchronous calls, and the asynchronous call uses the callback function AsyncCallback to process the data returned by the service interface. According to the protocol used by the data interface, this embodiment provides two subclasses, SoapClient and RestClient, which process SOAP and REST protocols respectively. It should be noted that in the development process of the data integration development framework, the strategy mode, factory mode and annotation are used to support custom WsClient and meet the requirements of the open and closed principle. Among them, the annotation @WsType is used to specify the protocol processed by MyClient, that is, the protocol used by the data interface (such as: SOAP, REST, etc.), and MyClient uses a custom protocol. When starting the application, MyClient will automatically register with the ClientFactory factory, and AbstractCallback will obtain an instance of WsClient from the factory according to the protocol used by the data interface, and then determine the response data.

[0145] (3) Record the response data into a log file, and parse the response data using the target format conversion rule corresponding to the remote callback processing. Convert the data format of the obtained response data according to the corresponding target format conversion rule to convert the data exchange format of the response data.

[0146] (4) Perform corresponding business processing on the parsing results of the response data and the preset business data instance to determine the return data. After determining the corresponding business data instance, such as a POJO instance, perform corresponding business processing on it, and then determine the parsing results of the response data and the business processing results as the return data and return them to the sub-process.

[0147] (5) When there is preset error log data in the log file, the network request is converted again in response to the data compensation operation. During the data compensation monitoring process, the records in the log file are judged. When the preset error log data appears, an instruction requiring data compensation is issued. The user performs the data compensation operation. In response to the data compensation operation, the data integration development framework of this embodiment attempts to re-push the request (remote callback interface) to convert the network request.

[0148] The method of using the data integration development framework in this embodiment is as follows:

[0149] The Maven project introduces the jar package required by the data integration development framework.

[0150] <dependency>

[0151] <groupid> com.glodon.data< / groupid>

[0152] <artifactid> glodon-data-core< / artifactid>

[0153] <version> 1.0.0< / version>

[0154] < / dependency>

[0155] Implement the data interface executor configuration class ExecutorConfigurer; data interface implementation, the implementation class must be a subclass of AbstractCallback or AbstractService.

[0156] The data integration method provided in this embodiment integrates preset format conversion rules through real-time monitoring of interfaces, and determines the corresponding target format conversion rules in the integrated preset format conversion rules according to the data integration type to determine the return data of the data integration request; so as to meet the business requirements of numerous data interfaces, multiple data types, inconsistent data standards, real-time data exchange, data extraction and push, etc., to achieve cross-platform data source collection, and overcome the problem of data loss; simplify the work of developers, so that developers do not need to deal with various operations such as data model conversion and logging, do not need to understand the underlying protocol of the data interface, and only need to focus on the processing of business data; ensure the simplicity and accuracy of the data processing process; by converting the data format of remote callback processing into a standard XML file to determine the network request, the data processing and conversion process is facilitated, the standardization and normativeness of data processing is improved, and after a data processing error, data compensation can be performed to support the reloading and processing of data in different scenarios, the implementation process is simple, and the data processing efficiency is improved.

[0157] In this embodiment, a data integration device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0158] This embodiment provides a data integration device, such as Figure 5 , including: data integration interface configuration module 1, used to manage data interface and data interface status, data interface includes service interface and callback interface; data integration interface configuration module is also used to configure parameters of data interface, and send alarm information when parameter configuration fails; interface processor management module 2, used to register and monitor the status of interface processor; data compensation module 3, used to trigger data interface to re-execute parsing when data integration request parsing fails. The data integration platform provided in this embodiment can add data interface (local service interface, remote callback interface) and configure interface parameters (REST, SOAP, etc.) on the data integration interface configuration page; can enable or disable data interface online; configure alarm Email, and receive alarm information once data interface fails; add interface processor (data interface executor) on the interface processor management page, and the IP and port of the executor can be manually or automatically registered to the data integration management platform, and the automatic registration detects the online status through the heartbeat mechanism.

[0159] The data integration device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0160] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0161] The embodiment of the present invention also provides an electronic device having the above Figure 5 data integration device.

[0162] See also Figure 6 , Figure 6 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present invention, such as Figure 6 , the electronic device may include: at least one processor 601, such as a CPU (Central Processing Unit), at least one communication interface 603, a memory 604, and at least one communication bus 602. The communication bus 602 is used to realize the connection and communication between these components. The communication interface 603 may include a display screen (Display) and a keyboard (Keyboard), and the optional communication interface 603 may also include a standard wired interface and a wireless interface. The memory 604 may be a high-speed RAM memory (Random Access Memory) or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 604 may optionally be at least one storage device located away from the aforementioned processor 601. The processor 601 may be combined with Figure 5 In the described device, the memory 604 stores an application program, and the processor 601 calls the program code stored in the memory 604 to execute any of the above method steps.

[0163] The communication bus 602 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The communication bus 602 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0164] Among them, the memory 604 may include a volatile memory (English: volatile memory), such as a random access memory (English: random-access memory, abbreviated: RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk drive (English: hard disk drive, abbreviated: HDD) or a solid-state drive (English: solid-state drive, abbreviated: SSD); the memory 604 may also include a combination of the above types of memory.

[0165] The processor 601 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP.

[0166] The processor 601 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0167] Optionally, the memory 604 is also used to store program instructions. The processor 601 can call the program instructions to implement the present application Figure 2 , 3 and the data integration method in 4 embodiments.

[0168] The embodiment of the present invention further provides a non-transitory computer storage medium, which stores computer executable instructions, and the computer executable instructions can execute the data integration method in any of the above method embodiments. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0169] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.< / pojo> < / pojo> < / t>

Claims

1. A data integration method, It is characterized in that include: receiving a data integration request, wherein the data integration request is a service interface request; Determine a data integration type based on the data integration request, the data integration type being local service processing; Determine the corresponding target format conversion rule in the preset format conversion rules of the integration according to the data integration type; Determine the return data of the data integration request according to the data integration type and the target format conversion rule; Return the conversion result of the data exchange format to the external network end as a Web response; Wherein, determining the return data of the data integration request according to the data integration type and the target format conversion rule includes: Matching the service interface in the data integration request in the interface registration list, and judging whether the service interface provides service according to the status parameter of the service interface; When the service interface provides a service, the service interface request is parsed using the local service process and the target format conversion rule corresponding to the local service process to determine the return data of the service interface request, wherein the local service process is the data processing of the local service interface implemented by the abstract class AbstractService encapsulated by the data integration type, and the local service request received by the local service process is a network request; The step of parsing the service interface request by using the local service process and the target format conversion rule corresponding to the local service process to determine the return data of the service interface request includes: Analyze the service interface request using the target format conversion rule corresponding to the local service processing to determine the data conversion method; Perform corresponding business processing on the preset business data instance according to the local service processing and data conversion method to determine the interface data; The interface data is converted into a data exchange format using a target format conversion rule corresponding to the local service process, the interface data is converted into a network format, and a conversion result of the data exchange format is determined as return data.

2. The method according to claim 1, It is characterized in that The performing corresponding business processing on the preset business data instance according to the local service processing and data conversion method also includes: Perform corresponding business processing on the preset business data instance, and record the service interface where the preset error occurs in the log file; In response to the data compensation operation, the service interface request is re-parsed based on the log file.

3. The method according to claim 1, It is characterized in that When the data integration request is a callback interface request, the data integration type is remote callback processing, and the determining the return data of the data integration request according to the data integration type and the target format conversion rule includes: Matching the callback interface in the data integration request in the interface registration list, and determining whether the callback interface is enabled according to the status parameter of the callback interface; When the callback interface is enabled, the callback interface request is parsed using the remote callback process and the target format conversion rule corresponding to the remote callback process to determine the return data of the callback interface request.

4. The method according to claim 3, It is characterized in that The method of parsing the callback interface request by using the remote callback process and the target format conversion rule corresponding to the remote callback process to determine the return data of the callback interface request includes: Performing data exchange format conversion on the callback interface request by using remote callback processing and target format conversion rules corresponding to the remote callback processing to generate a network request; Determine response data according to the network request and the interface protocol of the remote callback processing; Record the response data into a log file, and parse the response data using a target format conversion rule corresponding to the remote callback processing; The parsing result of the response data and the preset business data instance are subjected to corresponding business processing to determine the return data.

5. The method according to claim 4, It is characterized in that The method of converting the data exchange format of the callback interface request by using the remote callback processing and the target format conversion rule corresponding to the remote callback processing to generate a network request includes: Insert the attribute value in the preset business data instance corresponding to the remote callback processing into the preset XML template to determine the XML file; The callback interface request is converted into a data exchange format using a target format conversion rule corresponding to remote callback processing, and the data model after the data exchange format conversion is combined with an XML file to generate a standard XML file to determine the network request.

6. The method according to claim 4, It is characterized in that The determining of response data according to the network request and the interface protocol of the remote callback processing includes: Registering the preset interface protocol into the protocol factory class according to the network request; The response data is determined in the protocol factory class according to the interface protocol of the remote callback processing.

7. The method according to claim 4, It is characterized in that The recording of the response data into a log file includes: When there is preset error log data in the log file, the network request is reconverted in response to a data compensation operation.

8. A non-transitory computer-readable storage medium, It is characterized in that The non-transitory computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the data integration method according to any one of claims 1 to 7 is implemented.

9. An electronic device, It is characterized in that include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data integration method as described in any one of claims 1 to 7 by executing the computer instructions.

Citation Information

Patent Citations

  • Method and device for achieving data exchange based on data bus technology

    CN103338206A