A data lake-based data middle platform API automatic generation system

Abstract

The application provides a data lake-based data middle platform API automatic generation system, comprising a data acquisition module, a data cataloging module, an API automatic configuration module, an API automatic generation module and an API automatic deployment module; different theme data of a data resource catalog can be adaptively adjusted to theme data service API interfaces, and the service API interfaces are automatically generated; the method can realize automatic generation of the service API interfaces in a simple and efficient manner, can realize decoupling of a data layer and an application layer, can efficiently generate API interfaces and automatically deploy, and can provide effective atomic capabilities for actual project constructors.

Description

TECHNICAL FIELD

[0001] The application relates to a data lake-based data middle platform API automatic generation system. BACKGROUND

[0002] At present, a data lake mainly solves the problems of a large amount of structured data and unstructured data entering the lake and data management after entering the lake. The data middle platform is an intermediate layer between the back end and the front end, is constructed around the data service provided to the upper layer application, and forms a benign closed loop between the data platform and the business system, that is, the decoupling between the application and the data is realized, and the close interaction is realized. After the data middle platform is established, the data service API is formed, and various data services are provided for enterprises and customers. The defects of the prior art mainly include the following two aspects:

[0003] 1. The existing data lake solution has great coupling to the upper layer application, and API interfaces need to be repeatedly developed for different applications;

[0004] 2. A large number of API interfaces for the upper layer application need to consume a large amount of manpower for maintenance. SUMMARY

[0005] The technical problem to be solved by the application is to provide a data lake-based data middle platform architecture, which adopts an automatic cataloging method to catalog data resources, generates an automatic configuration file through an automatic configuration module, generates API interfaces through an automatic API service module, and finally deploys the generated API interface program through an automatic deployment module. The method can effectively improve the efficiency of data cataloging, the development efficiency of service API, the maintenance cost and the reusability.

[0006] The application specifically provides a data lake-based data middle platform API automatic generation system, which comprises a data acquisition module, a data cataloging module, an API automatic configuration module, an API automatic generation module and an API automatic deployment module.

[0007] The data acquisition module is used for acquiring structured and semi-structured data into the lake and storing the data in a data lake raw data area.

[0008] The data acquisition module stores the data collected in the raw data area in the production data area after processing such as outlier elimination and aggregation, and the data in the production data area comprises basic information, dynamic information and statistical information.

[0009] The data cataloging module is used for cataloging the data capable of providing external services in the production data area of the data lake, and the cataloging comprises a Chinese resource name, an English resource name and an information resource abstract. After the data resources are cataloged, data resource metadata is formed.

[0010] The API automatic configuration module reads the configuration module automatically generated by the data catalog module and selects possible input and output parameters, and automatically generates request parameter and response parameter configuration files;

[0011] The API automatic generation module generates API service interfaces according to the API automatic configuration module;

[0012] The API automatic deployment module adopts an automatic deployment method based on Git and Openshift, and after the request parameter and response parameter configuration files generated by the API automatic configuration module are changed, the request parameter and response parameter configuration files are automatically pushed to the GIT server. After the Git server listens to the new files, the Openshift is automatically triggered by the webhook to pull the source files on the Git and automatically compile, package and deploy to a Docker container, and complete the automatic deployment process of the API interface.

[0013] The data resource metadata format is as follows:

[0014]

[0015] The data resource metadata includes basic information and information items;

[0016] The basic information includes a series of single attributes, including resourceId, chineseName, englishName, sourceSystem, publishTime and shareType. resourceId refers to the unique code of the resource directory, chineseName refers to the Chinese name of the resource, englishName refers to the English name of the resource, sourceSystem refers to the source system of the data resource, vendor refers to the provider of the resource, publishTime refers to the resource publishing time, and shareType refers to the sharing method;

[0017] The information items are a series of items composed of items, representing the specific fields of the data resource. The items are composed of a column of items, and each item is composed of name, datatype, and comment. name, datatype, and comment represent the data item name, data type, and description, respectively;

[0018] The catalog is generated in an automatic way and saved in the data catalog resource center. resourceId is generated by a random function, items are obtained from the metadata of the data resource in the database and automatically generated, and other fields are described by obtaining and collecting information and extracted by word segmentation.

[0019] The criteria for the input parameters of the API automatic configuration module are as follows:

[0020] Items in which a data primary key exists in item;

[0021] Items in which an index exists in item;

[0022] Items in which the data type is time in item;

[0023] The output parameters are all item items in items.

[0024] The API automatic configuration module automatically generates request parameter and response parameter configuration files according to the criteria for the input parameters. The request parameter can be dynamically adjusted and is composed of a series of fields lists. The fields list includes two or more single-element fields. The field includes the parameter, type, and compulsory attributes. The parameter is the field name, the type is the field type, and the compulsory is whether it is compulsory. Whether the request parameter is dynamic is determined by the value of the compulsory attribute.

[0025] The response parameter is composed of a series of fields lists. The fields list includes two or more single-element fields. The field includes the parameter and type attributes.

[0026] The generated request parameter fields are as follows:

[0027] <request>

[0028] <field>

[0029] <parameter>param1< / parameter>

[0030] <type>Integer< / type>

[0031] <compulsory>true< / compulsory>

[0032] < / field>

[0033] <field>

[0034] <parameter>param2< / parameter>

[0035] <type>String< / type>

[0036] <compulsory>false< / compulsory>

[0037] < / field>

[0038] < / request>

[0039] The parameter is the field name, the type is the field type, and the compulsory is whether it is compulsory.

[0040] The generated response parameter field format is as follows:

[0041] <response>

[0042] <field>

[0043] <parameter>param1< / parameter>

[0044] <type>Integer< / type>

[0045] < / field>

[0046] <field>

[0047] <parameter>param2< / parameter>

[0048] <type>String< / type>

[0049] < / field>

[0050] < / response>

[0051] The parameter is the field name, the type is the field type, and the request parameter and response parameter configuration files are saved in the configuration resource center.

[0052] The API service interface includes:

[0053] The URL part of the API service;

[0054] The code automatic generation part of the API service;

[0055] The API service URL is composed of API.

[0056] The code automatic generation part of the API service includes:

[0057] Step a1, reading the request parameter and response parameter configuration file generated by the API automatic configuration module;

[0058] Step a2, dynamically generating a query SQL dynamic statement according to the request parameter and response parameter configuration file, and the query statement skeleton is select <field>from table where<filter condition>, skeleton <field>To respond to the field in the parameter configuration file, the <filter condition> is generated according to the rule for the request parameter configuration file, and the rule is as follows:

[0059] If the Compulsory attribute of an item in the items in the request parameter configuration file is false, the field does not need to be filtered, and when the Compulsory attribute is true, the field needs to be filtered, and the filtering rule is accurate filtering;

[0060] If the Compulsory attribute of an item in the items in the request parameter configuration file is true and the type of the item is time, the filtering rule is range filtering;

[0061] After the data is dynamically acquired, the request parameters and the returned data are generated in the corresponding json format.

[0062] The API automatically generates a dynamic Docker container during deployment, the deployment program acquires the container IP and port, replaces the IP and port of the URL of the API service, and automatically generates an online interface document.

[0063] Beneficial effects: the application can adaptively adjust the theme data service API interface of different theme data in the data resource directory, automatically generate the service API interface, and realize the automatic generation of the service API interface in an efficient and simple way, can realize the decoupling of the data layer and the application layer and efficiently generate the API interface, and provides effective atomic capabilities for actual project builders. BRIEF DESCRIPTION OF DRAWINGS

[0064] The above and / or other aspects of the application will become more apparent by describing in detail the application with reference to the accompanying drawings, and the specific embodiments.

[0065] Figure 1 is the architecture diagram of the application. DETAILED DESCRIPTION

[0066] As Figure 1 shown, the application provides a data lake-based data middle platform API automatic generation system, provides an automated resource interface generation method for software developers, and forms a standardized interface specification document, so that the data interface can be developed and deployed more quickly, and a large amount of interface development and deployment time is reduced.

[0067] The application describes a data lake-based data middle platform architecture, and the main module components include a data acquisition module, a data cataloging module, an API automatic configuration module, an API automatic generation module and an API service automatic deployment module. The overall process is as follows:

[0068] Firstly, the data collection module uses various collection means such as Http, JDBC and MQ to collect structured and semi-structured data into the data lake and store it in the data lake raw data area.

[0069] Then, the data collection module stores the processed data from the raw data area into the production data area, which mainly includes three types of basic information, dynamic information and statistical information.

[0070] Next, the data cataloging module catalogs the data that can provide external services in the business subject partition of the data lake. The cataloging includes Chinese resource name, English resource name, information resource abstract, etc. After cataloging, the data resource metadata is formed, and the format of the data resource metadata is as follows:

[0071]

[0072]

[0073] Among them, resourceId refers to the unique code of the resource directory, chineseName refers to the Chinese name of the resource, englishName refers to the English name of the resource, sourceSystem refers to the source system of the data resource, vendor refers to the provider of the resource, publishTime refers to the resource publishing time, shareType refers to the sharing method, items refers to a series of data items representing specific fields of the data resource, items are composed of a column of item, each item is composed of name, datatype, comment, which represent data item name, data type and description respectively.

[0074] The above cataloging method is generated automatically and saved in the data cataloging resource center. The resourceId in the above field content is generated by a random function, the items are obtained and generated by data metadata, and the other fields are obtained and generated by collecting information description and using word segmentation and power acquisition methods.

[0075] Next, the API automatic configuration module reads the above configuration file and selects possible input and output parameters. The selection criteria for input parameters are as follows:

[0076] Items with data primary keys in item;

[0077] Items with indexes in item;

[0078] Items with time type in item;

[0079] The output parameter is all item items in items.

[0080] The API automatic configuration module automatically generates the request parameter and response parameter configuration files according to the above criteria, wherein the generated request parameter field is as follows:

[0081] <request>

[0082] <field>

[0083] <parameter>param1< / parameter>

[0084] <type>Integer< / type>

[0085] <compulsory>true< / compulsory>

[0086] < / field>

[0087] <field>

[0088] <parameter>param2< / parameter>

[0089] <type>String< / type>

[0090] <compulsory>false< / compulsory>

[0091] < / field>

[0092] < / request>

[0093] Wherein parameter is the field name, type is the field type, and Compulsory is whether it is compulsory.

[0094] The generated response parameter field format is as follows:

[0095] <response>

[0096] <field>

[0097] <parameter>param1< / parameter>

[0098] <type>Integer< / type>

[0099] < / field>

[0100] <field>

[0101] <parameter>param2< / parameter>

[0102] <type>String< / type>

[0103] < / field>

[0104] < / response>

[0105] Wherein parameter is the field name, type is the field type. The above configuration files are saved in the configuration resource center.

[0106] Then, the API automatic generation module generates the API service interface according to the automatic configuration module. The API service interface is divided into the following parts:

[0107] The URL part of the API service;

[0108] The code automatic generation part of the API service;

[0109] Wherein the URL of the API service is composed of the API skeleton, the format adopts the restful style, and the skeleton is as follows:

[0110] http: / / ip:port / resourceId

[0111] ip and port are the ip and port after deployment, which are automatically generated by the automatic deployment module, and resourceId is the resource id, which is generated by the English name englishName in the data resource catalog module.

[0112] The code automatic generation part of the API service is generated as follows:

[0113] Step a1, reading the request parameter and response parameter configuration files generated by the API automatic configuration module;

[0114] Step a2, dynamically generating a query SQL dynamic statement according to the request parameter and response parameter configuration files, and the query statement skeleton is select <field>from table where<filter condition>, skeleton <field>To respond to the field in the parameter configuration file, the <filter condition> is generated according to the rule for the request parameter configuration file, and the rule is as follows:

[0115] If the Compulsory attribute of an item in the items in the request parameter configuration file is false, the field does not need to be filtered, and when the Compulsory attribute is true, the field needs to be filtered, and the filtering rule is accurate filtering;

[0116] If the Compulsory attribute of an item in the items in the request parameter configuration file is true and the type of the item is time, the filtering rule is range filtering;

[0117] After the data is dynamically acquired, the request parameters and the returned data are generated into corresponding json formats.

[0118] Finally, the API automatic deployment module adopts an automatic deployment method based on K8S and GIT, after the API automatic configuration module generates the API configuration file, the configuration file is pushed to the GIT server, the source code file of the API automatic generation module on the GIT server is automatically triggered and compiled by the Webhook, and the source code file is packaged and deployed into a K8S container together with the configuration file of the API automatic configuration module, so that the automatic deployment process of the API interface is completed.

[0119] The API automatic deployment generates a dynamic container, the deployment program acquires the container IP and port, replaces the IP and port of the URL of the API service, and automatically generates an online document of the interface.

[0120] The application provides a data lake-based data center API automatic generation system, and there are many methods and approaches to specifically implement the technical solution, and the above description is only a preferred embodiment of the application, and it should be pointed out that, for ordinary technical personnel in the technical field, some improvements and refinements can be made without departing from the principle of the application, and these improvements and refinements should be regarded as the protection range of the application. The components not explicitly described in the embodiment can be realized by using the existing technology.< / field> < / field> < / field> < / field>

Claims

1. A data lake-based data middle platform API automatic generation system, characterized in that, The data collection module, the data cataloging module, the API automatic configuration module, the API automatic generation module and the API automatic deployment module are included. The data collection module is used for collecting structured and semi-structured data into a data lake and storing the data in a data lake raw data area. The data collection module processes the data stored in the raw data area after collection and stores the processed data in a production data area. The data cataloging module is used for cataloging data in the production data area of the data lake that can provide external services, and the cataloging includes Chinese resource names, English resource names, information resource abstracts, and data resource metadata formed after cataloging. The API automatic configuration module reads the configuration module automatically generated by the data cataloging module and selects possible input and output parameters to automatically generate request parameter and response parameter configuration files. The API automatic generation module generates API service interfaces according to the API automatic configuration module. The API automatic deployment module uses a Git and Openshift-based automatic deployment method, and after the request parameter and response parameter configuration files generated by the API automatic configuration module are changed, the configuration files are automatically pushed to a GIT server. The API automatic configuration module automatically generates request parameter and response parameter configuration files according to the standards of input parameters. The request parameter can be dynamically adjusted and is composed of a series of fields lists, the fields list includes two or more single-element fields, and the field includes parameter, type and compulsory attributes, wherein parameter is a field name, type is a field type, and compulsory is whether it is compulsory. The response parameter is composed of a series of fields lists, and the fields list includes two or more single-element fields, and the field includes parameter and type attributes. The code automatic generation part of the API service includes: Step a1, reading the request parameter and response parameter configuration files generated by the API automatic configuration module; Step a2, dynamically generate query SQL dynamic statement according to request parameter and response parameter configuration file, query statement skeleton is select <field>from table where<filter condition>, skeleton <field>For the field in the response parameter configuration file, <filter condition> is generated according to the rules of the request parameter configuration file, and the rules are as follows:< / field> < / field> If the Compulsory attribute of an item in the items item in the request parameter configuration file is false, the field does not need to be filtered, and when the Compulsory attribute is true, the field needs to be filtered, and the filtering rule is exact filtering. If the Compulsory of an item in the items in the request parameter profile is true and the type of the item is time, the filtering rule is range filtering; After the data is dynamically acquired, the request parameters and the returned data are generated into corresponding json formats.

2. The system of claim 1, wherein, The data resource metadata includes basic information and information items; The basic information includes resourceId, chineseName, englishName, sourceSystem, publishTime and shareType, the resourceId is the unique code of the resource directory, the chineseName is the Chinese name of the resource, the englishName is the English name of the resource, the sourceSystem is the source system of the data resource, the vendor is the provider of the resource, the publishTime is the resource publishing time, and the shareType is the sharing mode; The information items are a series of items composed of items, representing the specific fields of the data resource, the items are composed of a column of items, each item is composed of name, datatype and comment, the name, datatype and comment represent the data item name, data type and description respectively; The cataloging is generated in an automated manner and saved in the data cataloging resource center, the resourceId is generated by a random function, the items are obtained from the metadata of the data resource in the database and automatically generated, and the other fields are described by the acquired information and obtained and generated by the word segmentation extraction method.

3. The system of claim 2, wherein, The selection criteria of the API automatic configuration module are as follows: The item in which the data primary key exists; The item in which the index exists; The item in which the data type is time; The output parameter is all item items in the items.

4. The system of claim 3, wherein, The API service interface includes: The URL part of the API service; The code automatic generation part of the API service; The API service URL is composed of API.

5. The system of claim 4, wherein, The dynamic Docker container is generated when the API is automatically deployed, the container ip and port are obtained by the deployment program, the ip and port of the API service URL are replaced, and the online document of the interface is automatically generated.

Images