XML-based low-code configurable system integration implementation method

Abstract

The invention discloses a low-code configurable system integration implementation method based on XML (Extensible Markup Language). The algorithm relates to the technical field of application software development and low code. According to the method, through cooperative work of a task scheduling layer, an XML model layer, a connection establishment layer, a data exchange layer, a data conversion layer and a business logic layer, integrated adaptation of multiple modes of SOAP and RESTFUL is achieved. The functions of data pushing, data pulling, process triggering, data mapping and a data analysis module between systems are highly abstracted into code models irrelevant to services, so that integration implementation personnel do not need to write codes, and the functions of data integration and application integration can be operated only by defining the service models on XML (Extensible Markup Language) files. The business model comprises configuration of a data source address, an operation token, data mapping and flow triggering. According to the method, through code packaging, an implementation engineer does not need to consider implementation details and more focuses on business logic implementation. The development efficiency of system integration can be greatly improved, and the integration online debugging time is shortened.

Description

Technical Field

[0001] The present invention relates to the fields of application development and low-code platforms. Background Art

[0002] With the rapid development of information technology, in the process of digital transformation, breaking information silos and realizing system interconnection among diverse and heterogeneous business systems have become important challenges faced by enterprises and organizations. Each independently built application system may face the task of data and application integration with multiple other systems. And usually, after data exchange, further business logic processing is required. Therefore, how to provide a low-code configurable technical solution to integrally solve the problems of data integration and application integration, improve the efficiency and quality of system integration, and shorten the development and debugging cycle is an important issue faced by those skilled in the art. Summary of the Invention

[0003] In the past, when low-code was used to achieve data integration among multiple systems, it usually focused on visual and simple drag-and-drop for docking configuration, ignoring the complex data cleaning and data loading processing requirements among different information systems in manufacturing enterprises. During the actual implementation process, it often fails to meet the requirements of actual system business collaboration. In many cases, developers have to perform secondary development on this basis or write code separately for processing. Against this background, the present invention proposes a low-code configurable multi-source heterogeneous data integration method, designs an XML configuration model, which can solve problems such as parent-child table association, attachment association, and special data processing during system data integration. Further, complex logics related to application integration such as process triggering and business status update can be configured and implemented. Applying the present invention can not only improve code utilization rate and reduce development costs, but also meet the implementation of complex business logics for multi-source data integration in manufacturing enterprises. Thus, it improves the efficiency and quality of data integration and meets the requirements of modern data management. The method includes the following steps: S1. Task Scheduling: Schedule and execute data integration tasks periodically at regular intervals. Specify the trigger time rule and use the Java Quartz trigger to start task execution.

[0004] The Java Quartz trigger CronTriggerBean is used for periodic task scheduling.

[0005] The steps include: First, create an instance object of CronTriggerBean and set relevant attributes.

[0006] Set the Cron expression. The Cron expression is used to specify the trigger time rule of the trigger.

[0007] Add the CronTriggerBean object to the Scheduler, and finally start the Scheduler to begin the execution of data integration processing.

[0008] The results of task scheduling are displayed on the module page. It includes at least: action command, Trigger name, Trigger group, next execution time, last execution time, Trigger status, Trigger type, start time, end time. The action command includes at least: execute, pause, and delete. It is used for administrators to manually operate scheduling tasks.

[0009] S2. Parse the XML model: Load the XML configuration model according to the configuration file name and perform configuration parameter parsing.

[0010] The entry configuration file for data integration processing includes at least: service processing class and configuration file name. The configuration file name loads the XML file in a specific directory and then executes the service processing class. There are at least two types of service processing classes: SOAP and restFul. The two methods are based on different protocols and are processed by different processing classes.

[0011] S3. Establish a connection: Then obtain the data access token according to the token address in the XML file. After having the token, access the service address of the data integration peer system, send a data access request, and establish a data connection.

[0012] A token needs to be obtained before establishing a connection. The token is also defined in the XML file. The token connection attributes in the configuration model include at least: token service URL address, service parameters, unique identification ID of the requester, authentication type. After splicing the html request header according to the above connection attributes, obtain the token.

[0013] The data access model in the configuration model is used for data processing. Its top-level parameter is the data integration step. That is, a data integration exchange can be processed step by step in sequence.

[0014] The access attributes of the integration step include at least: data processing step to integrate the data service URL address of the other party, unique data integration identification, data update date, paging size, request page number. After splicing the html request header according to the above connection attributes and carrying the token to send a request to the target address, after getting the return value 200, the data connection is established.

[0015] S4. Data exchange: Execute the push data or pull data task according to the data flow configuration. When executing the task, it is executed sequentially according to the data acquisition steps in the configuration model. Read the data to be pushed from the data source table, pack it in json format and then perform the push operation; for pulling data, it is stored in the intermediate table. The intermediate table stores the complete set of data required for this integration task. After the data push and pull are completed, the integration log is recorded.

[0016] The first case is data pulling including: Data is pushed or pulled according to the data exchange mode (dataflow) in the configuration module. The data pull conversion attributes in the configuration model include at least: intermediate table unique identifier, data field mapping FROM-TO model, enumeration value conversion model, data source primary key field, data cleaning SQL, data conversion SQL, data processing SQL. Data push attributes include at least: data source table, push field, data filtering SQL, default page size.

[0017] The data pull operation obtains the JSON format data set returned by the integration counterpart service. First, delete all the data in the intermediate table, and then insert all the pulled data into the intermediate table as the complete set of local data sources for this integration task. The data in the intermediate table is not converted and remains consistent with the integrated data source.

[0018] Pulling data requires processing clob fields, child tables, and attachments separately. Child tables are connected to parent records through the parent_id field. Attachments use external links to achieve local access.

[0019] Another way to pull data is to display a tree structure, which is displayed through the mapping relationship between XML-configured data items and node attributes.

[0020] The second case is data push, including: The data push operation requires the preparation of a data set. By filtering SQL, the data that meets the push conditions is extracted and packaged and pushed to the target address as required. After the push is successful, the target address will receive a return value of 200.

[0021] After the data is successfully pushed and pulled, the integration log table is recorded. The fields of the integration log table include at least: identification, task name, time, data flow, execution duration, and number of data items.

[0022] S5 Data conversion and loading: According to the data mapping relationship in the configuration model, the data in the intermediate table is collected into the target table. The target table is the business operation table in the system. According to the keyword comparison and update time comparison, the data set to be processed is obtained. After data cleaning and data conversion according to the mapping relationship and conversion rules in the XML configuration module, the target table is updated or inserted to complete the data loading.

[0023] For the entire set of local integrated data sources, the cleaned data set is obtained according to the data cleansing SQL in the XML configuration module, and then data conversion is performed one by one according to the data conversion rules to obtain the local operation data set.

[0024] Compare data item by item between the target table according to the keyword fields of the data source and the mapping relationship. The comparison is based on keywords and update time. Records that do not exist in the target table are inserted, and records that already exist in the target table and need to be updated are updated. Thus, the data loading is completed.

[0025] S6. Business logic collaborative processing: After the data loading is completed, perform associated business logic processing according to the configuration model in the XML. This includes initiating a process according to filtering conditions, submitting the process, changing the status of business identifiers, etc.

[0026] After the data loading is completed in the business operation table in the own system, further business logic processing is required. Initiate the corresponding process according to the process processing model in the xml file. The configuration of the process processing model at least includes: English name of the process (unique identifier), ID of the target storage table, filtering conditions for initiating the process, field mapping between the data source table and the target table, and initiating the process according to filtering conditions.

[0027] After the process is initiated, continue to process by submitting parameters according to the process. The parameters submitted by the process at least include: ID of the submitted node, name of the recipient role.

[0028] Correspondingly, after the process is initiated and submitted for processing, the status of the business identifier needs to be updated according to the configuration.

[0029] Due to the adoption of the above technical solutions, the technical progress achieved by the present invention is as follows: The present invention abstracts the processing code of system integration and develops an integration processing engine independent of business. This engine is stable and reliable, and supports multiple integration methods. Implementers do not need to master the development language, and only need to flatly configure business attributes in the XML file to achieve data integration and application integration functions. Such a tool allows implementers to only focus on business requirements without considering complex implementation details, greatly shortening the development cycle and reducing the implementation cost. It enables the full reuse of code and is conducive to the technical accumulation within the organization. Brief Description of the Drawings

[0030] Figure 1 It is a schematic diagram of the method flow of the present invention Figure 2 It is a schematic diagram of the system structure of the present invention: Figure 3 It is a processing flow chart of the integration core engine Figure 4 It is an example diagram of the data integration xml file of the present invention; Figure 5 It is an example diagram of the spring configuration file Figure 6 It is a screenshot of the task scheduling module interface: Figure 7Code for the data synchronization scheduling task class Detailed implementation mode

[0031] Taking the specific implementation process of data integration in a RESTful manner as an example, the data exchange method is to pull data. The present invention will be explained in detail with reference to the accompanying drawings.

[0032] First step, create an integrated data configuration package. There are 2 sub-packages under the package: config and sevice.impl. There is 1 file under the config package: spring-module.xml. There are: syncJobSchedule.java and launchJobSchedule.java, 2 files under the sevice.impl package.

[0033] spring-module.xml is used for the configuration of data synchronization processing classes, process initiation classes, integrated XML file names, and daily execution time cronExpression, and sets the task trigger time and processing class mapping.

[0034] The main execution method of the data synchronization processing class syncJobSchedule.java calls the pullRemoteData() method of the integrated service class. This method has been written in the integrated service class, and only the call statement needs to be written here.

[0035] The main execution method of the process initiation class launchJobSchedule.java calls the batchLaunchProcess() method of the integrated service class. This method has been written in the integrated service class, and only the call statement needs to be written here.

[0036] Second step, create an intermediate table. The fields of the intermediate table should correspond one by one to the data transmitted by the integration partner. In addition, an auto-incrementing id should be added as the unique identifier field. Attachments are stored in a character field in the form of an external link. If there are parent-child tables that need to be synchronized and created, the child table needs to have a parent record identifier field.

[0037] Third step, create an integration log table. The fields should at least include id, integration task name, execution start time, execution end time, number of processed data, and the address of the partner service. Except for the id field, the remaining fields are character types.

[0038] Third step, create an integrated XML file integrate.xml. It should at least include the following content <tokenservice>Label. Used to configure the token service, including at least the token service address, parameters, client ID, client password, connection method, authentication method, and token value path.

[0039] <dataconfig>Label. Used to configure connections and data mapping. At least includes data service address, data service identifier, data name, copy data type, target table, page number, page size per page, request parameters, and <fromto>Label set. <fromto>The label includes at least: fromId, toId, and the Chinese name of the field.

[0040] For data items that require conversion of enumerated items, such as when the data saved in the integrated system of the other party is the enumerated item id value and the data in our system needs to be converted into Chinese, <valueto>The label includes at least: from and to attributes <affiliatedsql>The label is an SQL that is automatically executed after data pulling. It can include multiple SQL statements and is used to supplement field values or perform status conversions.

[0041] <launchprocess>The label is the configuration for initiating and submitting the process after data loading is completed. It includes at least: English name of the process, process module ID, submission type, submission role ID, table ID, from node ID, to node ID, and <tablecopy>Label set.

[0042] Subordinate to <launchprocess>Of the label <tablecopy>The tag set is used for the configuration of data replication when initiating a process. It includes at least: source table id, target table id, source data item string, and target data item string. If nested parent-child associated table replication is required <tablecopy>Configuration of tags.

[0043] Through the above three-step configuration, the business configuration of a certain integration task can be completed. When the system automatically checks the configuration of task scheduling, if the time condition for initiating scheduling is met, the execution of task scheduling will be triggered. The system includes a task scheduling management module, which displays a list of all scheduling execution situations and has functions of manually executing, pausing, and deleting tasks.

[0044] Including: The above-described embodiments are only descriptions of the preferred embodiments of the present invention and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.< / tablecopy> < / tablecopy> < / launchprocess> < / tablecopy> < / launchprocess> < / affiliatedsql> < / valueto> < / fromto> < / fromto> < / dataconfig> < / tokenservice>

Claims

1. A method for implementing an XML-based low-code configurable system integration. It is characterized in that, It includes: Task scheduling layer: Periodically schedule and execute data integration tasks at regular intervals. Specify the trigger time rule and use the Java Quartz trigger to start task execution. XML model layer: Configure the token address and parameters, configure the integration steps, configure the data integration rules, configure the request parameters, configure the data mapping, configure the data conversion rules, configure the data update rules, configure the initiation process and the submission process rules. The integration task loads the XML configuration model according to the configuration file name and parses the configuration parameters. Connection establishment layer: Then obtain the data access token according to the token address in the XML file. After obtaining the token, access the service address of the data integration partner system, send a data access request, and establish a data connection. Data exchange layer: Configure and execute the task of pushing data or pulling data according to the data flow. When executing the task, it is executed sequentially according to the data acquisition steps in the configuration model. Read the data to be pushed from the data source table, pack it in json format and then perform the push operation; for pulling data, it is stored in the intermediate table. The intermediate table stores the complete set of data required for this integration task. After the data push or pull is completed, an integration log is recorded. Data conversion layer: According to the data mapping relationship in the configuration model, collect the data in the intermediate table into the target table. The target table is the business operation table in the local system. According to the keyword comparison and the update time comparison, obtain the dataset to be processed that needs to be processed. After data cleaning and data conversion according to the mapping relationship and conversion rules in the XML configuration module, perform data update or insertion operations on the target table to complete data loading. Business logic coordination layer: After the data loading is completed, perform associated business logic processing according to the configuration model in the XML. It includes initiating a process, submitting a process, changing the business identifier status, etc. according to the filtering conditions.

2. The method according to claim 1, wherein For the periodic task scheduling mentioned above, the Java Quartz trigger CronTriggerBean is used. The steps include: First, create an instance object of CronTriggerBean and set the relevant properties. Set the Cron expression. The Cron expression is used to specify the trigger time rule of the trigger. Add the CronTriggerBean object to the Scheduler, and finally start the Scheduler to start the data integration process. The results of the task scheduling are displayed on the module page. It includes at least: action command, Trigger name, Trigger group, next execution time, last execution time, Trigger status, Trigger type, start time, end time. The action command includes at least: execute, pause, and delete. This page is used for administrators to manually operate the scheduling tasks.

3. The method according to claim 1, wherein It includes: The entry configuration file for data integration processing includes at least: service processing class and configuration file name. The configuration file is stored in a specific directory. The task scheduling module loads the XML file and then executes the service processing class. There are at least two types of service processing classes: SOAP and restFul. The two methods are based on different protocols and are processed by different processing classes. Before establishing a connection, you need to obtain a token. The token is used to confirm the legitimacy of the user accessing the integrated data. The token address and parameters are also defined in the XML file. The token connection attributes in the configuration model include at least: token service URL address, service parameters, requester unique identification ID, and authentication type. According to the above connection attributes, the token is obtained after splicing the html request message header. The data access model in the configuration model is used for data exchange processing. Its top-level parameter is the data integration step. That is, a data integration exchange can be processed in sequence. The integration step can solve complex, interrelated multi-set data exchange logic. Operations can be performed strictly according to the legal data of data processing. The access attributes of the integration step include at least: data processing step id, integration partner data service URL address, data integration unique identifier, data update date, paging size, and request page number. After splicing the html request message header according to the above connection attributes, a request is sent to the target address with the token, and after the return value 200 is obtained, the data connection is established.

4. The method according to claim 1, wherein include: Data is pushed or pulled according to the data exchange mode (dataflow) in the configuration model. The data pull conversion attributes in the configuration model include at least: intermediate table unique identifier, data field mapping FROM-TO model, enumeration value conversion model, data source primary key field, data cleaning SQL, data conversion SQL, data processing SQL. Data push attributes include at least: data source table, push field, data filtering SQL, default page size. The data pull operation obtains the JSON format data set returned by the integration counterpart service. First, delete all the data in the intermediate table, and then insert all the pulled data into the intermediate table as the complete set of local data sources for this integration task. The data in the intermediate table is not converted and remains consistent with the integrated data source. Pulling data requires processing clob fields, child tables, and attachments separately. Child tables are connected to parent records through the parent_id field. Attachments use external links to achieve local access. Another way to pull data is to display a tree structure, which is displayed through the mapping relationship between data items configured in XML and node attributes. The data push operation requires the preparation of a data set. By filtering SQL, the data that meets the push conditions is extracted and packaged and pushed to the target address as required. After the push is successful, the target address will receive a return value of 200.

5. The method according to claim 1, characterized in that, include: After the data is successfully pushed and pulled, the integration log table is recorded. The fields of the integration log table include at least: identification, task name, time, data flow, execution duration, and number of data items.

6. The method according to claim 1, wherein include: For the entire set of local integrated data sources, the cleansed data set is obtained according to the data cleansing SQL in the XML configuration model, and then data conversion is performed one by one according to the data conversion rules to obtain the local operation data set. Compare data item by item with the target table based on the key fields and mapping relationships of the data source. The comparison is based on the key words and update time. Insert records that do not exist in the target table, and update existing records in the target table that need to be updated. This completes the data loading.

7. The method according to claim 1, characterized in that, include: After the data loading of the business operation table in the local system is completed, further business logic processing is required. Initiate the corresponding process according to the process processing model in the xml file. The configuration of the process processing model at least includes: the English name of the process (unique identifier), the target storage table ID, the filtering conditions for initiating the process, the field mapping between the data source table and the target table, initiating the process according to the filtering conditions, and data copying. After the process is initiated, continue to process the process submission by submitting parameters according to the process. The process submission parameters at least include: the submission node ID, the name of the recipient role. Correspondingly, after the process initiation and submission processing, the business identification status needs to be updated according to the configuration.

8. A server, characterized in that, Including: At least one memory and at least one processor, the memory stores computer-executable instructions, and the processor invokes the computer-executable instructions to execute the data integration method according to any one of claims 1-7.

9. A storage medium, characterized in that, The storage medium stores computer-executable instructions, and when the computer-executable instructions are executed, the data integration method according to any one of claims 1-7 is implemented.

10. A computer program product, characterized in that, Including computer-executable instructions, and when the computer-executable instructions are executed, the data integration method according to any one of claims 1-7 is implemented.

Images