Data integration platform based on low code configuration

Abstract

The invention discloses a data integration platform based on low code configuration. The data integration platform comprises an EDI module, a template engine module, a Mapping module, an AS2 module, an OFTP2 module and an SAP module. The EDI module comprises a ParseEdi module, a UnparseEdi module, a RecognizeEdi module, a GetControlNumber module, a ParseDaffedilSchema module, a ParseDaffedilSchema module, a UnparseDaffedilSchema module, an EDI module, an EDI module, an EDI module, an EDI module, an EDI module, an EDI module, an EDI module, an EDI module, an EDI module, an EDI module, an EDI module, an EDI module, an EDI module, an EDI module and an EDI module, the ParseEdi module is used for analyzing the X12 format file and the EDIFact format file to generate an xml format file or a Json format file; the UnparseEdi module is used for analyzing an XML (Extensible Markup Language) format file or a Json format file to generate an X12 format file and an EDIFact format file; according to the method, scripts are used, a user can easily achieve complex data conversion operation, the data are mapped to another format from one format, and meanwhile the integrity and confidentiality of the data are kept. Due to the design of the system, the system is suitable for various different industries and application scenes, and a powerful tool is provided for users to simplify the data exchange process and improve the working efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer-related technologies, and specifically to a data integration platform based on low-code configuration. Background Art

[0002] The open source product Apache Nifi is popular in the integration field, but it lacks some functions, the most common of which are as follows: 1. It cannot support standard EDI format and Json and XML format conversion; 2. It lacks the ability of template engine; 3. The means of data conversion are limited, inefficient and not flexible enough; 4. It cannot support AS2 (Applicability Statement 2) protocol; 5. It cannot support OFTP2 (Odette File Transfer Protocol 2) protocol; 6. It lacks the ability to integrate with SAP. Therefore, we provide a data integration platform based on low-code configuration. Summary of the invention

[0003] In order to solve the defects of the prior art, the present invention provides a data integration platform based on low-code configuration.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0005] The present invention discloses a data integration platform based on low-code configuration, including an EDI module, a template engine module, a Mapping module, an AS2 module, an OFTP2 module and a SAP module;

[0006] The EDI module includes ParseEdi module, UnparseEdi module, RecognizeEdi module, GetControlNumber module, ParseDaffodilSchema module and UnparseDaffodilSchema module; the ParseEdi module is used to parse X12 format files and EDIFact format files to generate XML format files or Json format files; the UnparseEdi module is used to parse XML format files or Json format files to generate X12 format files and EDIFact format files;

[0007] The RecognizeEdi module is used to identify the format of the input data file; the GetControlNumber module is used to generate EDI ControlNumber; the ParseDaffodilSchema module is used to parse the FlatFile format file to generate a Json format file or an XML format file; the UnparseDaffodilSchema module is used to parse the XML format file and the Json format file to generate a FlatFil format file; the Mapping module is used to generate the converted data according to the input and mapping script;

[0008] The AS2 module is used to match the AS2 protocol; the OFTP2 module is used to match; the SAP module is used to match the SAP Java Connector protocol;

[0009] The method for using the lightweight low-code integration platform system includes the following steps: first, the format file to be converted is set through the template engine module, and then the file format of the file to be converted is identified by using the RecognizeEdi module. After the file format is identified, the EDI module is called, and the ParseEdi module, UnparseEdi module, GetControlNumber module, ParseDaffodilSchema module and UnparseDaffodilSchema module in the EDI module are called according to the set conversion format to perform format conversion.

[0010] As a preferred technical solution of the present invention, the EDI module is further provided with a distribution module, which is used to set the format file for conversion according to the template engine module, and the distribution module selects a suitable module for format conversion.

[0011] As a preferred technical solution of the present invention, it also includes an input cache module and an output cache module, the input cache module is used to cache the input data file format, and the output cache module is used to cache the data after the format conversion.

[0012] As a preferred technical solution of the present invention, it also includes a return calling module, which calls the ParseEdi module, UnparseEdi module, GetControlNumber module, ParseDaffodilSchema module and UnparseDaffodilSchema module in the EDI module according to the output format file stored in the output cache module to perform format conversion on the output format file, to convert the output format into an intermediate file with a format consistent with the input format file, and then compares the content of the intermediate file with the content of the input file to determine whether the conversion is successful; when the content of the intermediate file is consistent with the content of the input file, the format conversion is successful; when the content of the intermediate file is different from the content of the input file, the format conversion fails, and re-conversion is performed until the content of the intermediate file is consistent with the content of the input file.

[0013] As a preferred technical solution of the present invention, the method of comparing the content of the intermediate file with the content of the input file to determine whether the conversion is successful is that when the content of the intermediate file is consistent with the content of the input file, the format conversion is successful; when the content of the intermediate file is different from the content of the input file, the format conversion fails and re-conversion is performed until the content of the intermediate file is consistent with the content of the input file.

[0014] As a preferred technical solution of the present invention, the method for comparing the content of the intermediate file with the content of the input file is to check whether the file attribute type and file size of the intermediate file and the input file are consistent. When the file attribute type and file size of the intermediate file and the input file are consistent, it indicates that the format conversion is successful, and when the file attribute type and file size of the intermediate file and the input file are inconsistent, it indicates that the format conversion has failed, and the conversion is performed again.

[0015] As a preferred technical solution of the present invention, a warning module is also included, and when the intermediate file and the file attribute type and file size are inconsistent, a prompt is given.

[0016] As a preferred technical solution of the present invention, it also includes an environment management module, which is used to provide environment configuration and management functions, and manage and configure the integration center cluster, connected third party and other systems at the front end.

[0017] As a preferred technical solution of the present invention, it also includes a data anomaly warning module. During the data processing process, if abnormal situations such as data accumulation and a large number of processing failures occur, it is necessary to use SLS log monitoring or ARMS alarm to monitor and alarm; and different alarm frequencies for different types of anomalies are achieved through log levels.

[0018] The beneficial effects of the present invention are:

[0019] 1. This data integration platform based on low-code configuration first sets the format file to be converted through the template engine module, and then uses the RecognizeEdi module to identify the file format of the file to be converted. When the file format is identified, the EDI module is called to call the ParseEdi module, UnparseEdi module, GetControlNumber module, ParseDaffodilSchema module and UnparseDaffodilSchema module in the EDI module according to the set conversion format to perform format conversion. In this way, multiple formats can be converted to obtain the required format file, and it can also support AS2 protocol, OFTP2 protocol and SAP Java Connector protocol.

[0020] 2. A return call module is also provided in the data integration platform based on low-code configuration. The return call module calls the ParseEdi module, UnparseEdi module, GetControlNumber module, ParseDaffodilSchema module and UnparseDaffodilSchema module in the EDI module according to the output format file stored in the output cache module to perform format conversion on the output format file, so as to convert the output format into an intermediate file of a format consistent with the input format file, and then compare the content of the intermediate file with the content of the input file to determine whether the conversion is successful; when the content of the intermediate file is consistent with the content of the input file, the format conversion is successful; when the content of the intermediate file is different from the content of the input file, the format conversion fails and is re-converted until the conversion is successful. The content of the intermediate file is consistent with the content of the input file. The method of comparing the content of the intermediate file with the content of the input file to determine whether the conversion is successful is that when the content of the intermediate file is consistent with the content of the input file, the format conversion is successful; when the content of the intermediate file is different from the content of the input file, the format conversion fails, and the conversion is performed again until the content of the intermediate file is consistent with the content of the input file. The method of comparing the content of the intermediate file with the content of the input file is to determine whether the file attribute type and file size of the intermediate file are consistent. When the file attribute type and file size of the intermediate file are consistent, it indicates that the format conversion is successful. When the file attribute type and file size of the intermediate file are inconsistent, it indicates that the format conversion fails, and the conversion is performed again. In this way, the file after format conversion is rotated back to ensure that the converted file will not be lost. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0022] In the attached picture:

[0023] Figure 1 It is a system block diagram of a data integration platform based on low-code configuration in the present invention.

[0024] In the figure: 1. EDI module; 2. Template engine module; 3. Mapping module; 4. AS2 module; 5. OFTP2 module; 6. SAP module; 7. ParseEdi module; 8. UnparseEdi module; 9. RecognizeEdi module; 10. GetControlNumber module; 11. ParseDaffodilSchema module; 12. UnparseDaffodilSchema module; 13. Allocation module; 14. Input cache module; 15. Output cache module; 16. Return call module; 17. Warning module; 18. Environment management module; 19. Data abnormality warning module. DETAILED DESCRIPTION

[0025] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0026] Example: Figure 1 As shown, the present invention is a data integration platform based on low-code configuration, including an EDI module 1, a template engine module 2, a Mapping module 3, an AS2 module 4, an OFTP2 module 5 and a SAP module 6;

[0027] The EDI module 6 includes a ParseEdi module 7, an UnparseEdi module 8, a RecognizeEdi module 9, a GetControlNumber module 10, a ParseDaffodilSchema module 11 and an UnparseDaffodilSchema module 12; the ParseEdi module 7 is used to parse an X12 format file or an EDIFact format file to generate an XML format file or a Json format file; the UnparseEdi module 8 is used to parse an XML format file or a Json format file to generate an X12 format file or an EDIFact format file;

[0028] The RecognizeEdi module 9 is used to identify the format of the input data file; the GetControlNumber module 10 is used to generate EDI ControlNumber; the ParseDaffodilSchema module 11 is used to parse the FlatFile format file to generate a Json format file or an XML format file; the UnparseDaffodilSchema module 12 is used to parse the XML format file and the Json format file to generate a FlatFil format file; the Mapping module 2 is used to generate the converted data according to the input and mapping script;

[0029] The AS2 module 3 is used to match the AS2 protocol; the OFTP2 module 4 is used to match; the SAP module is used to match the SAP Java Connector protocol 5;

[0030] The method for using the lightweight low-code integration platform system includes the following steps: first, the format file to be converted is set by the template engine module 2, and then the file format of the file to be converted is identified by the RecognizeEdi module 9. When the file format is identified, the EDI module 1 is called to call the ParseEdi module 7, the UnparseEdi module 8, the GetControlNumber module 10, the ParseDaffodilSchema module 11 and the UnparseDaffodilSchema module 12 in the EDI module 1 according to the set conversion format to perform format conversion. The present invention enables users to customize the data transmission process according to their specific needs. By using scripts, users can easily implement complex data conversion operations, mapping data from one format to another while maintaining the integrity and confidentiality of the data. The design of the system makes it suitable for a variety of different industries and application scenarios, providing users with a powerful tool to simplify the data exchange process and improve work efficiency.

[0031] Among them, the EDI module 1 is also provided with an allocation module 13, which is used to set the conversion format file according to the template engine module 2. The allocation module 13 selects a suitable module for format conversion to set the conversion between formats, and plays the role of format conversion definition.

[0032] It also includes an input cache module 14 and an output cache module 15. The input cache module 13 is used to cache the input data file format, and the output cache module 15 is used to cache the data after format conversion, which is convenient for later callback and comparison of the converted format file with the input file content to ensure that no content is lost during the format conversion process.

[0033] It also includes a return calling module 16, which calls the ParseEdi module 7, UnparseEdi module 8, GetControlNumber module 10, ParseDaffodilSchema module 11 and UnparseDaffodilSchema module 12 in the EDI module 1 according to the output format file stored in the output cache module 15 to perform format conversion on the output format file, to convert the output format into an intermediate file with a format consistent with the input format file, and then compares the content of the intermediate file with the content of the input file to determine whether the conversion is successful; when the content of the intermediate file is consistent with the content of the input file, the format conversion is successful; when the content of the intermediate file is different from the content of the input file, the format conversion fails, and re-conversion is performed until the content of the intermediate file is consistent with the content of the input file. The method of comparing the content of the intermediate file with the content of the input file to determine whether the conversion is successful is that when the content of the intermediate file is consistent with the content of the input file, the format conversion is successful; when the content of the intermediate file is different from the content of the input file, the format conversion fails and re-conversion is performed until the content of the intermediate file is consistent with the content of the input file.

[0034] Among them, the method for comparing the content of the intermediate file with the content of the input file is to check whether the file attribute type and file size of the intermediate file and the input file are consistent. When the file attribute type and file size of the intermediate file and the input file are consistent, it indicates that the format conversion is successful, and when the file attribute type and file size of the intermediate file and the input file are inconsistent, it indicates that the format conversion has failed, and the conversion is performed again.

[0035] The system also includes a warning module 17, which gives a prompt when the intermediate files and the file attribute types and file sizes are inconsistent, so that it is easy to know whether the conversion is successful in time and make corrections in time.

[0036] It also includes an environment management module 18, which is used to provide environment configuration and management functions, and manage and configure the integration center cluster, connected third-party systems and other systems at the front end.

[0037] It also includes a data anomaly warning module 19. During the data processing process, if abnormal situations such as data accumulation and a large number of processing failures occur, it is necessary to use SLS log monitoring or ARMS alarm to monitor and alarm; and different alarm frequencies for different types of abnormalities are achieved through log levels.

[0038] When working, this data integration platform based on low-code configuration first sets the format file to be converted through the template engine module 2, and then uses the RecognizeEdi module 9 to identify the file format of the file to be converted. When the file format is identified, the EDI module 1 is called to call the ParseEdi module 7, UnparseEdi module 8, GetControlNumber module 10, ParseDaffodilSchema module 11 and UnparseDaffodilSchema module 12 in the EDI module 1 according to the set conversion format to perform format conversion. The present invention enables users to customize the data transmission process according to their specific needs. By using scripts, users can easily implement complex data conversion operations, mapping data from one format to another while maintaining the integrity and confidentiality of the data. The design of the system makes it suitable for a variety of different industries and application scenarios, providing users with a powerful tool to simplify the data exchange process and improve work efficiency.

[0039] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data integration platform based on low-code configuration, characterized in that: It includes an EDI module (1), a template engine module (2), a Mapping module (3), an AS2 module (4), an OFTP2 module (5) and a SAP module (6); The EDI module (6) includes a ParseEdi module (7), an UnparseEdi module (8), a RecognizeEdi module (9), a GetControlNumber module (10), a ParseDaffodilSchema module (11) and an UnparseDaffodilSchema module (12); the ParseEdi module (7) is used to parse an X12 format file or an EDIFact format file to generate an XML format file or a Json format file; the UnparseEdi module (8) is used to parse an XML format file or a Json format file to generate an X12 format file or an EDIFact format file; The RecognizeEdi module (9) is used to recognize the format of the file of input data; the GetControlNumber module (10) is used to generate EDI ControlNumber; the ParseDaffodilSchema module (11) is used to parse the FlatFile format file to generate a Json format file or an XML format file; the UnparseDaffodilSchema module (12) is used to parse the XML format file and the Json format file to generate a FlatFil format file; the Mapping module (2) is used to generate converted data according to the input and mapping script; The AS2 module (3) is used to match the AS2 protocol; the OFTP2 module (4) is used to match; the SAP module is used to match the SAP Java Connector protocol (5); The method for using the lightweight low-code integration platform system comprises the following steps: first, setting the format file to be converted by using the template engine module (2), and then using the RecognizeEdi module (9) to identify the file format of the file to be converted. When the file format is identified, the EDI module (1) is called, and the ParseEdi module (7), UnparseEdi module (8), GetControlNumber module (10), ParseDaffodilSchema module (11) and UnparseDaffodilSchema module (12) in the EDI module (1) are called according to the set conversion format to perform format conversion.

2. A data integration platform based on low-code configuration according to claim 1, characterized in that: The EDI module (1) is also provided with a distribution module (13), and the distribution module (13) is used to set the format file for conversion according to the template engine module (2), and the distribution module (13) selects a suitable module for format conversion.

3. A data integration platform based on low-code configuration according to claim 1, characterized in that: It also comprises an input cache module (14) and an output cache module (15), wherein the input cache module (13) is used to cache the input data file format, and the output cache module (15) is used to cache the data after the format conversion.

4. A data integration platform based on low-code configuration according to claim 3, characterized in that: The system further comprises a return calling module (16), wherein the return calling module (16) calls the ParseEdi module (7), the UnparseEdi module (8), the GetControlNumber module (10), the ParseDaffodilSchema module (11) and the UnparseDaffodilSchema module (12) in the EDI module (1) according to the output format file stored in the output cache module (15), so as to perform format conversion on the output format file, so as to convert the output format into an intermediate file having a format consistent with the input format file, and then compare the content of the intermediate file with the content of the input file to determine whether the conversion is successful; when the content of the intermediate file is consistent with the content of the input file, the format conversion is successful; when the content of the intermediate file is different from the content of the input file, the format conversion fails and is re-converted until the content of the intermediate file is consistent with the content of the input file.

5. A data integration platform based on low-code configuration according to claim 4, characterized in that: The method of comparing the content of the intermediate file with the content of the input file to determine whether the conversion is successful is that when the content of the intermediate file is consistent with the content of the input file, the format conversion is successful; when the content of the intermediate file is different from the content of the input file, the format conversion fails and re-conversion is performed until the content of the intermediate file is consistent with the content of the input file.

6. A data integration platform based on low-code configuration according to claim 5, characterized in that: The method for comparing the content of the intermediate file with the content of the input file is to check whether the file attribute type and file size of the intermediate file and the input file are consistent. When the file attribute type and file size of the intermediate file and the input file are consistent, it indicates that the format conversion is successful. When the file attribute type and file size of the intermediate file and the input file are inconsistent, it indicates that the format conversion has failed and the conversion is performed again.

7. A data integration platform based on low-code configuration according to claim 5, characterized in that: The system also includes a warning module (17), which gives a warning when the intermediate files, file attribute types and file sizes are inconsistent.

8. A data integration platform based on low-code configuration according to claim 1, characterized in that: It also includes an environment management module (18), which is used to provide environment configuration and management functions, and manage and configure the integration center cluster, connected third party and other systems at the front end.

9. A data integration platform based on low-code configuration according to claim 1, characterized in that: It also includes a data anomaly warning module (19). During the data processing process, if abnormal situations such as data accumulation and a large number of processing failures occur, it is necessary to use SLS log monitoring or ARMS alarm to monitor and alarm; and different alarm frequencies for different types of abnormalities are achieved through log levels.

Images