Integration method for API interface data

Abstract

The invention relates to an integration method for API interface data. The method comprises the following steps: carrying out basic configuration on an API interface data acquisition task; in the task configuration stage, data levels and nodes needing to be collected are specified, and a data format conversion relation between the data nodes needing to be collected and target end table fields is set; in the task execution stage, interface return result data analysis and data level and node selection are performed, and data format conversion is completed; interface data needing to be collected and a corresponding node path are screened out, the node path is designed as a data table, the node path serves as a field English name, and the node name serves as a field remark; configuring a scheduling strategy called by the API interface; and executing an API (Application Program Interface) data acquisition task through the ApiReader read plug-in and the JdbcWriter write plug-in. According to the method, automatic collection and library falling of API interface data can be achieved, API interface requests of a large number of parameter combinations are supported, and the problems that the collection efficiency is low and the labor cost is high when an API interface is manually called to collect data can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of data synchronization technology, and in particular to an integration method and device for API interface data in the field of data synchronization. Background Art

[0002] Nowadays, data has become a new production factor and has greatly promoted social development. Many companies have obtained huge commercial value from data. Government departments, as owners of massive data, are also actively promoting the sharing and opening of government data. One form of openness is to share anonymized data in the form of API interfaces. Companies promote their business success by collecting data from these API interfaces and making reasonable use of them.

[0003] However, traditional API interface data collection generally requires customized development through coding, which has problems such as low collection efficiency and high labor costs. It is also impossible to accurately control the call frequency and call interval, and is unable to automatically perform data result analysis and data conversion. There is also a lack of mature automated API interface collection tools on the market. Summary of the invention

[0004] In order to overcome the above-mentioned defects of traditional API interface data collection methods, the present invention proposes a new integration method for API interface data in the field of data synchronization. This method can effectively solve the problems of low efficiency and high labor cost of manually calling API interface to collect data.

[0005] In order to achieve the above object, the present invention mainly makes the following designs:

[0006] (1) This method can realize the automatic collection and storage of API interface data through visual front-end configuration and back-end data collection engine. Moreover, during the automatic collection of API interface data, this method can accurately set the request frequency, request interval and request validity period of the interface; it can also automatically parse and convert the result data collected by the API interface. When the amount of collected data is large, paging requests can be made according to the configuration; this method also supports a request failure retry mechanism, which can automatically retry the request after the interface request fails.

[0007] (2) This method uses a visual interface to flatten the complex response data results, simplifying the configuration process of API interface data collection.

[0008] Specifically, the present invention provides an integration method for API interface data, such as Fig.13 As shown, the method comprises the following steps:

[0009] S1. Perform basic configuration for the API interface data collection task. Basic configuration parameters include but are not limited to: interface request method, data response format, interface address, request parameters, verification method, successful response example, and successful response parameters;

[0010] S2. Parse and convert the data format of the result (response result) returned by the API interface, including: specifying the data level and node to be collected in the task configuration stage, parsing the interface return result data and selecting the data level and node in the task execution stage; filtering out the interface data to be collected and the corresponding node path according to the selected data nodes of different levels, and designing the obtained node path into a data table, in which the node path is used as the English name of the field and the node name is used as the field remark; in the task configuration stage, setting the data format conversion relationship between the data node to be collected and the target end table field, and completing the data format conversion according to the data format conversion relationship in the task execution stage;

[0011] S3. Configure the scheduling strategy for API interface calls, including configuration: limit the number of calls, interface call interval, scheduling cycle, scheduling start time, scheduling validity period, and paging requests;

[0012] S4. Execute API interface data collection tasks through ApiReader read plug-in and JdbcWriter write plug-in.

[0013] Furthermore, in the basic configuration parameters described in step S1 of the API interface data integration method of the present invention:

[0014] The interface request method supports GET request and POST request methods;

[0015] The data response format supports interface response data in JSON and XML formats;

[0016] The interface address is a URL for an http request;

[0017] The request parameters include paging request parameters;

[0018] The successful response example provides an example of the returned result data when the requested API interface responds successfully. It supports the interface return result data in JSON and XML formats. When the type of the interface return result data is uncertain, the interface connectivity test is performed. When the interface request is successful, the interface return result is automatically filled in.

[0019] The successful response parameter is an indicator of successful interface request. Select an attribute node from the successful response example as the successful response parameter. The successful response parameter is identified by the code response code and supports setting specified values ​​and user-defined values.

[0020] Furthermore, the request parameters in the basic configuration parameters described in step S1 of the API interface data integration method of the present invention include the following types:

[0021] (1) KV (Key-Value) type request parameter: Set the parameter name, parameter type, whether it is a paging parameter, parameter description, and the parameter value corresponding to the parameter name of the KV type request parameter;

[0022] When setting parameter values ​​for KV type request parameters, two configuration methods are available: constant set and table field;

[0023] When selecting a constant set, enter the fixed request parameter value. If there are multiple values, separate them with commas.

[0024] When selecting a table field, select a table in a relational database, and then select a field in the table, which means that all column values ​​of a field in the table are used as parameter values;

[0025] (2) Header type request parameter: In this method, the header information when requesting the interface is set in the form of Key-Value. The header information includes but is not limited to: authentication information and response data format;

[0026] (3) Body type request parameters: When the interface request method is selected as POST request method, set the Body type request parameters.

[0027] Furthermore, the present invention is directed to an integration method for API interface data. In this method, for an API interface with multiple parameters, the interface parameters are combined by Cartesian product according to the number of parameters and parameter values, and finally a Cartesian product parameter group is formed;

[0028] When there are many parameter values ​​corresponding to a parameter, a Cartesian product parameter group is formed by the following parameter group caching method:

[0029] (1) After the task configuration is completed, the program calculates the parameter combinations of all parameters and caches them in ElasticSearch (distributed search and analysis engine) in advance;

[0030] (2) Before making an API call, first retrieve all parameter combinations for the task from ElasticSearch, and use one set of parameters to make sequential API calls;

[0031] (3) When the parameter values ​​in the task configuration are adjusted, the parameter combination stored in ElasticSearch is updated synchronously to ensure that the parameter combination obtained when the next API interface is called is the latest.

[0032] Furthermore, step S2 of the integration method for API interface data of the present invention also includes: in the task configuration stage, the program parses the successful response example data, displays the nodes of different levels to form row labels, and the user checks the level nodes that need to be collected through the visual operation interface; when a certain level node is checked, except for the root node, a node path composed of the various parent nodes of the node will be formed, and the node path is spliced ​​with each parent node name through the period ".". In this way, the data node that needs to be collected can be accurately located. In addition, for the node names with the same name in different levels, this method can also be used to distinguish them.

[0033] Furthermore, the method for integrating API interface data of the present invention further comprises in step S2: when the target end table does not exist, generating the field name of the corresponding target end table through field remarks (i.e. node name) mapping;

[0034] When the target end table is newly created, by establishing a mapping relationship between the node path and the target end table field, it is specified that the different node data of the interface return result will be saved in the corresponding target end table field.

[0035] Furthermore, the present invention is directed to the API interface data integration method step S2, in the task configuration phase, setting the data format conversion relationship between the data node to be collected and the target end table field, including:

[0036] (1) Based on the mapping relationship between the source node path and the target table field, determine the target time field CJSJ mapped to the source node path;

[0037] (2) Determine the data format of the time field CJSJ according to the target data source type and the type of the target time field CJSJ;

[0038] (3) After determining the data format of the target end time field CJSJ, set the corresponding data conversion format for the source end node path.

[0039] Furthermore, the scheduling strategy described in step S3 of the API interface data integration method of the present invention includes:

[0040] Limit the number of calls: Set the total number of calls to the API interface. If the API interface provider does not limit the number of calls to the API interface, set the total number of calls to unlimited.

[0041] Interface call interval: used to control the time interval between two adjacent interface calls;

[0042] Scheduling cycle: used to set the calling frequency of the API interface;

[0043] Scheduling validity period: used to set the validity period of the API interface data collection task. When the validity period is reached, the API interface collection task will automatically stop;

[0044] Paging request: configure the paging request based on the paging request parameters in the basic configuration parameters; the paging request parameters include a page number parameter (pageNo) and a number of requests per page parameter (pageSize); the page number parameter includes three types of transformation: increasing, decreasing, and unchanged:

[0045] (a) When the page number parameter is set to increment, the page number parameter passed will increase in sequence each time the API interface is requested;

[0046] (b) When the page number parameter is set to decrease, the page number parameter passed will decrease in sequence each time the API interface is requested;

[0047] (c) When the page number parameter is set to unchanged, the page number parameter passed will not change each time the API interface is requested and will remain the same each time.

[0048] Furthermore, the API interface data integration method of the present invention, step S4, performs the API interface data collection task by using the ApiReader read plug-in and the JdbcWriter write plug-in, including:

[0049] (1) The ApiReader plug-in obtains the basic configuration parameters of the task, queries all parameter groups from ElasticSearch, and starts requesting the API interface to obtain the result data;

[0050] (2) After the ApiReader plug-in obtains the result data, it parses and extracts the result data according to the configured node path that needs to be collected;

[0051] (3) After the ApiReader plug-in obtains the data value corresponding to the node path, it converts the data format according to the configured data format conversion relationship;

[0052] (4) The ApiReader plug-in writes the format-converted data into the channel pipeline;

[0053] (5) The JdbcWriter writing plug-in obtains input data from the channel pipeline, and then writes the source data into the corresponding fields of the target table based on the field mapping relationship.

[0054] On the other hand, the present invention also provides an integration device for API interface data, which implements the steps of the above-mentioned integration method for API interface data when the device is running, and the device includes:

[0055] Task basic configuration module: This module provides a visual interface for basic configuration of API interface data collection tasks. Basic configuration parameters include but are not limited to: interface request method, data response format, interface address, request parameters, verification method, successful response example, and successful response parameters.

[0056] Task running module: This module is used to configure and execute API interface data collection tasks, including: parsing of API interface return result data, data format conversion configuration, scheduling strategy configuration, and task execution;

[0057] Interface request exception handling module: The function of this module is to configure the number of failed retries. When an interface request fails during the process of calling the API interface to collect data, the program automatically retries in sequence according to the set number of retries until the request succeeds.

[0058] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the above-mentioned integration method for API interface data are implemented.

[0059] In summary, the API interface data integration method of the present invention has the following advantages:

[0060] (1) The parameter group caching scheme designed by this method can support API interface requests with a large number of parameter combinations.

[0061] (2) This method can realize the automatic collection and storage of API interface data through visual front-end configuration and back-end data collection engine.

[0062] (3) This method parses and flattens complex response data results through a visual interface, simplifying the configuration process of API interface data collection.

[0063] (4) During the automated collection of API interface data, this method can accurately set the interface request frequency, request interval, and request validity period; it can also automatically parse and convert the result data collected by the API interface, thereby ensuring the accuracy of the data stored in the database.

[0064] (5) During the automated collection of API interface data, when the amount of collected data is large, this method can perform paging requests based on the configuration, thereby ensuring the integrity of the data collected by the API interface.

[0065] (6) During the automated collection of API interface data, this method also supports a request failure retry mechanism, which can automatically retry requests after interface request failure, and has good fault tolerance. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the description of the present invention. Obviously, the following drawings are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 Schematic diagram of the basic configuration of tasks in an embodiment of the present invention.

[0068] Figure 2 Schematic diagram of the Cartesian product combination of two parameters in an embodiment of the present invention.

[0069] Figure 3 The figure is a schematic diagram of the configuration of Header type request parameters in an embodiment of the present invention.

[0070] Figure 4 This is a schematic diagram of response result analysis and node selection in an embodiment of the present invention.

[0071] Figure 5 Schematic diagram of field mapping in an embodiment of the present invention.

[0072] Figure 6 The figure is a schematic diagram of the data format conversion field mapping relationship in an embodiment of the present invention.

[0073] Figure 7 These are eight data conversion formats built into the embodiments of the present invention.

[0074] Figure 8 This is a schematic diagram of paging request parameter configuration of a task basic configuration module in an embodiment of the present invention.

[0075] Fig. 9 The diagram is a schematic diagram of paging request parameter configuration in the scheduling strategy configuration in an embodiment of the present invention.

[0076] Fig.10 Schematic diagram of reading and writing of data synchronization tool in an embodiment of the present invention.

[0077] Fig.11 This is a task execution flow chart of API interface collection in an embodiment of the present invention.

[0078] Fig.12 The figure is a schematic diagram of failure retry configuration in an embodiment of the present invention.

[0079] Fig.13 The figure is a flow chart of the overall implementation of the method of the present invention. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. The present invention can also be implemented or applied through other different specific implementation methods. The details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0081] At the same time, it should be understood that the protection scope of the present invention is not limited to the specific embodiments described below; it should also be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the protection scope of the present invention.

[0082] Embodiment: A method and device for integrating API interface data

[0083] The device for integrating API interface data of the present invention includes the following modules:

[0084] (I) Task basic configuration module

[0085] The function of the task basic configuration module is to provide a visual interface for basic configuration of API interface data collection tasks.

[0086] like Figure 1 As shown, for API interface data collection, the core basic configurations include: interface request method, data response format, interface address, request parameters, verification method, successful response example, successful response parameters, etc.

[0087] 1. Interface request method

[0088] Supports GET and POST request methods.

[0089] 2. Data response format

[0090] Supports interface response data in JSON and XML formats.

[0091] 3. Interface address

[0092] That is, the address of the API interface that needs to be requested. The interface address is generally a URL for an http request.

[0093] 4. Request parameters

[0094] (1) KV type request parameter configuration

[0095] Supports configuration of Key-Value type request parameters, where you can set the parameter name, parameter type, whether it is a paging parameter, parameter description, and the parameter value corresponding to the parameter name.

[0096] When setting parameter values, you can choose between two configuration methods: constant set and table field. When selecting constant set, you can enter a fixed request parameter value. If there are multiple values, they are separated by commas. When selecting table field, you can select a table in a relational database and then select a field in the table. This means that all column values ​​of a field in the table are used as parameter values.

[0097] Table 1 Request parameter configuration example table

[0098] Parameter name Parameter Type Whether to paginate Configuration Parameter Value param1 Character no Constant Set 1,2,3,4,5 param2 Numeric no Table fields Database, table, field name

[0099] For better explanation, here we take Table 1 as an example. Assume that two request parameters param1 and parma2 are configured. The parameter value corresponding to parameter param1 is a constant set type. The constant set entered by the user is: "1,2,3,4,5", separated by commas. The program will automatically recognize and split it into five parameter values: 1,2,3,4,5.

[0100] If the parameter value corresponding to the parameter param2 is the table field type, then the parameter value corresponding to this parameter is all column values ​​of the selected field.

[0101] In this method, for such API interfaces with multiple parameters, the interface parameters are combined by Cartesian product based on the number of parameters and parameter values. Figure 2 As shown, all parameter values ​​of parameter param1 will be paired with all parameter values ​​of parameter param2 one by one, and finally form a parameter group of Cartesian product. This embodiment takes 2 parameters as an example. If there are 3 or even more parameters, the composition principle of the parameter group is the same.

[0102] GET http: / / 127.0.0.1 / appgateway / user / getUserInfo? param1=1¶m2=11

[0103] When there are many parameter values ​​corresponding to a parameter, the number of parameter groups forming the Cartesian product is very large. It is difficult to accurately complete the assembly of such a large number of parameters for manual calling of API interface data collection. In actual application scenarios, API interface data collection often needs to obtain different data by passing many parameters. Therefore, in order to solve this problem, the present invention specially designs a parameter group caching solution.

[0104] The parameter group caching scheme in the method of the present invention is as follows:

[0105] (a) After the task configuration is completed, the program will calculate the parameter combinations of all parameters and cache them in ElasticSearch (distributed search and analysis engine) in advance. ElasticSearch can easily store and retrieve massive data.

[0106] (b) Before making an API call, all parameter combinations for the task are retrieved from ElasticSearch, and a set of parameters is used to make sequential API calls.

[0107] (c) When the parameter values ​​in the task configuration are adjusted, the parameter combination stored in ElasticSearch is updated synchronously to ensure that the parameter combination obtained during the next API call is the latest one.

[0108] (2) Header type request parameter configuration

[0109] Header is the header information carried when sending a request. Usually, you can set the authentication information of the request: authentication / cookie, the data format of the response: content-type, etc. Figure 3 As shown, in the method of the present invention, it is also supported to set the Header information when requesting the interface in the form of Key-Value.

[0110] (3)Body type request parameter configuration

[0111] When the interface request method is selected as the POST request method, the request parameter of the Body type can be set. The request parameter of the Body type is more flexible. In the present invention, the parameter type of application / json is supported, which is also the most common json format, and its form is as follows:

[0112] {"param1":"value1","param2":"value2","param3":value3}

[0113] 5. Successful response example

[0114] The successful response example is an example of the returned result data when the requested API interface successfully responds. In the present invention, JSON type and XML type interface return result data are supported.

[0115] When it is uncertain what type of data the interface returns, the present invention also provides an interface connectivity test. Under the premise of ensuring that the interface request address and request parameters are correctly configured, by clicking the connectivity test button on the visual interface, when the interface request is successful, the return result of the interface will be automatically filled in.

[0116] 6. Success response parameters

[0117] The successful response parameter is an identifier that needs to be selected for a successful interface request. In the present invention, an attribute node is selected from a successful response example as an identifier for a successful interface request, and a code response code is generally used as an identifier.

[0118] The method of the present invention supports setting a specified value, such as 2xx; it also supports users setting a custom response code value.

[0119] (II) Task running module

[0120] The task running module is mainly used for configuration and task execution related to the API interface data collection task operation, including: parsing of the result data returned by the API interface and data format conversion configuration, scheduling strategy configuration, and task execution.

[0121] 1. Data analysis and data format conversion configuration

[0122] (1) Response result data analysis and node selection

[0123] When collecting API interface data, the data format returned by the interface is generally JSON or XML. Here, the commonly used JSON type is used as an example. When the returned result data is more complex, there will generally be more levels. When collecting data, it is often necessary to obtain part of the data in the JSON data structure. If the data is collected by manually calling the interface, the required data must be screened each time the data is collected, which is not only time-consuming and labor-intensive, but also prone to errors. In order to simplify the API interface data synchronization task configuration and collection process, the present invention specifies the data level and node to be collected in the task configuration stage, realizes one-time configuration, automatic parsing, and no intervention is required for subsequent operation.

[0124] The specific implementation method is as follows: During the configuration phase, the program will parse the successful response sample data, display the nodes at different levels to form row labels, and through the visual operation interface, the user can quickly check the level nodes that need to be collected; when the corresponding level node is checked, except for the root node, a node path consisting of the various parent nodes of the node will be formed; the node path uses the dot "." to splice the names of the various parent nodes. Figure 4 As shown in the figure, when the id node is selected, the node path is data.rwpzRwpzxx.id. This way we can accurately locate the data node we want. In addition, this way can also be used to distinguish nodes with the same name in different levels.

[0125] (2) Field Mapping

[0126] like Figure 5 As shown, in the previous step, after selecting data nodes at different levels and filtering out the interface data and corresponding node paths to be collected, this solution will design these node paths into a data table, with the node path as the English name of the field and the node name as the field note.

[0127] The ultimate goal of API interface data collection is to save the collected data into the database to facilitate subsequent statistics and analysis. Therefore, in this step, when the target table does not exist, the field name of the corresponding target table will be mapped through the field notes on the left, that is, the node name.

[0128] When the target table is created, by establishing a mapping relationship between the left node path and the target table field, you can specify that the different node data of the interface return result be saved in the corresponding target table field, thereby eliminating the manual processing process.

[0129] (3) Data format conversion

[0130] When collecting data through API interfaces, the format of the returned data often does not meet the statistical and analytical requirements of the target data table. The most common format is the date format. For example, the time in the result data obtained through API interface calls is in a 10-digit or 13-digit timestamp format (timestamp refers to the total number of seconds from 00:00:00 Greenwich time on January 1, 1970 (08:00:00 Beijing time on January 1, 1970) to the present), while the time format of the target data table needs to be "yyyy-MM-ddHH:mm:ss". For manual calls to API interfaces, this data conversion work is quite cumbersome, and there are many problems in the manual processing process, such as long time consumption and easy errors.

[0131] In view of this, the present invention sets the data format conversion relationship between the data nodes to be collected and the target data table fields in advance during the task configuration stage, so that the data format conversion work can be completed automatically and efficiently during the actual execution of the API interface call.

[0132] The specific steps are:

[0133] (a) Based on the mapping relationship between the source node path and the target field, for example Figure 6 The source node path is data.rwpzRwpzxx.cjsj, which represents the creation time in the result data collected through the API interface. Confirm that it is mapped to the time field CJSJ on the target end.

[0134] (b) According to the type of the target data source and the type of the time field CJSJ, determine the data format of the field. For example, the target data source here is an Oracle data source, and the type of the CJSJ field is a DATE type. In the Oracle database, the data storage format of the DATE type is YYYY-MM-DD HH24:MI:SS, where YYYY represents the year, MM represents the month, DD represents the date, HH24 represents the hours in the 24-hour system, MI represents the minutes, and SS represents the seconds. For example, the value of the DATE type at 2:30:10 p.m. on May 1, 2019 is 2019-05-01 14:30:10. Therefore, for the data to be written to the CJSJ field on the target side, it needs to be formatted as YYYY-MM-DD HH24:MI:SS to ensure the accuracy of the data.

[0135] (c) After determining the data type of the field to be written on the target side, it is necessary to set the corresponding data conversion format for the node path on the source side. In the present invention, a total of eight built-in data formats are provided, such as Figure 7 As shown in the figure, if the source node path is data.rwpzRwpzxx.cjsj, the target field CJSJ corresponding to the path node is of DATE field type, so it needs to be set to date type (YMD H24:MI:S). After setting the data format of the source node path, the configuration will be saved and used when the API interface call is officially executed.

[0136] 2. Scheduling strategy configuration

[0137] The scheduling policy configuration mainly involves configuration related to the API interface call policy, including: limiting the number of calls, interface call interval, scheduling cycle, scheduling start time, scheduling validity period, and paging request configuration.

[0138] In order to avoid the tedious work and possible deviation caused by manually calling the API interface to collect data, the present invention realizes the automatic calling and paging request of the API interface by configuring the API interface calling strategy in advance.

[0139] The following describes the above configuration in detail:

[0140] (1) Limit the number of calls

[0141] Since some API interface providers limit the number of calls to the API interface, in the present invention, the total number of calls to the API interface can be set. When the number of calls to the API interface is unlimited, it can also be set to an unlimited number.

[0142] (2)Interval between interface calls

[0143] The interface call interval is mainly used to control the time interval between two adjacent interface calls to avoid excessive pressure on the interface caused by too many requests in a short period of time.

[0144] (3) Scheduling cycle

[0145] The scheduling cycle is mainly used to set the calling frequency of the API interface, which can be set to yearly, monthly, weekly, daily, hourly, or minutely.

[0146] (4) Scheduling validity period

[0147] The scheduling validity period is used to set the validity period of the API interface data collection task. When the validity period is reached, the API interface collection will automatically stop.

[0148] (5) Paging request configuration

[0149] When configuring paging request parameters in the task basic configuration module, such as Figure 8 As shown, in the scheduling configuration module, you can configure the paging request, such as Fig. 9 shown.

[0150] For some API interfaces with large data volumes, if a large amount of data is requested at one time, the interface request will take too long, resulting in interface request failure. Therefore, in order to prevent too much data from being requested at one time, the present invention is designed to perform interface paging requests through interface paging parameters, thereby controlling the amount of data requested at a single API interface.

[0151] Generally, two paging parameters are required (the parameter names may be different depending on the actual interface):

[0152] pageNo: page number parameter.

[0153] pageSize: parameter indicating the number of requests per page.

[0154] In the present invention, three types of transformations of page number parameters are supported: increasing, decreasing and unchanged.

[0155] When the page number parameter is set to increment, the page number parameter passed will increase in sequence each time the API interface is requested;

[0156] When the page number parameter is set to decrement, the page number parameter passed will decrement in sequence each time the API interface is requested;

[0157] When the page number parameter is set to unchanged, the page number parameter passed will not change each time the API interface is requested, and will remain the same each time;

[0158] In this way, the API interface can automatically perform paging requests even when the amount of data returned is large, so that the data can be collected completely.

[0159] 3. Task Execution

[0160] The present invention is based on an open source data synchronization tool and has developed its own API interface data acquisition plug-in. As an offline data synchronization framework, the tool adopts a Framework+plugin architecture, which abstracts data source reading and writing into Reader / Writer plug-ins and incorporates them into the entire synchronization framework.

[0161] The channel pipeline mechanism is used to realize the communication between the source and the target: the Reader plug-in writes data to the channel, and the Writer plug-in reads data from the channel.

[0162] We need to collect API interface data here. The first step is to obtain input data from the API interface and then write it into the channel pipeline.

[0163] like Fig.10 and Fig.11 As shown, the specific execution process is as follows:

[0164] (1) The ApiReader plug-in obtains the basic configuration parameters of the task, queries all parameter groups from ElasticSearch, and starts requesting the API interface to obtain the result data.

[0165] (2) After the ApiReader plug-in obtains the result data, it parses and retrieves the result data according to the configured node path that needs to be collected.

[0166] (3) After the ApiReader plug-in obtains the data value corresponding to the node path, it converts the data format according to the configured data format conversion relationship.

[0167] (4) The ApiReader plug-in writes the format-converted data into the channel pipeline.

[0168] (5) The JdbcWriter writing plug-in obtains the input data from the channel pipeline, and then writes the data from the source end (input end) to the corresponding fields of the target end data table based on the field mapping relationship.

[0169] (III) Interface request exception handling module

[0170] When requesting an API interface, there is a possibility of call failure due to network jitter and instability. When the interface request fails, there may be a risk of data loss if it is not processed. Therefore, in the present invention, a failed interface request retry mechanism is also designed to improve the fault tolerance of API interface data collection.

[0171] like Fig.12 As shown in the figure, after configuring the number of failed retries, when a request fails during the process of calling the API interface to collect data, the program will automatically retry according to the set number of retries until the request succeeds.

[0172] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession can make some changes or modifications to the technical contents disclosed above without departing from the scope of the technical solution of the present invention to obtain equivalent embodiments of equivalent changes. However, any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for integrating API interface data, characterized in that: The method comprises: S1. Perform basic configuration for the API interface data collection task. The basic configuration parameters include: interface request method, data response format, interface address, request parameters, successful response example, and successful response parameters; S2. Parse and convert the result data returned by the API interface, including: specifying the data level and node to be collected in the task configuration stage, parsing the result data returned by the interface and selecting the data level and node in the task execution stage; filtering out the interface data to be collected and the corresponding node path according to the selected data nodes of different levels, and designing the obtained node path into a data table, in which the node path is used as the English name of the field and the node name is used as the field remark; in the task configuration stage, setting the data format conversion relationship between the data node to be collected and the target end table field, and completing the data format conversion according to the data format conversion relationship in the task execution stage; S3. Configure the scheduling strategy for API interface calls, including configuration: limit the number of calls, interface call interval, scheduling cycle, scheduling start time, scheduling validity period, and paging requests; S4. Execute API interface data collection tasks through ApiReader read plug-in and JdbcWriter write plug-in.

2. The method for integrating API interface data according to claim 1, characterized in that: In the basic configuration parameters described in step S1: The interface request method supports GET request and POST request; The data response format supports interface response data in JSON and XML formats; The interface address is a URL for an http request; The request parameters include paging request parameters; The successful response example provides an example of the returned result data when the requested API interface responds successfully. It supports the interface return result data in JSON and XML formats. When the type of the interface return result data is uncertain, the interface connectivity test is performed. When the interface request is successful, the interface return result is automatically filled in. The successful response parameter is an indicator of successful interface request. Select an attribute node from the successful response example as the successful response parameter. The successful response parameter is identified by the code response code and supports setting specified values ​​and user-defined values.

3. The method for integrating API interface data according to claim 1, characterized in that: The request parameters in the basic configuration parameters described in step S1 include the following types: (1) KV type request parameter: set the parameter name, parameter type, whether it is a paging parameter, parameter description, and parameter value corresponding to the parameter name of the KV type request parameter; When setting parameter values ​​for KV type request parameters, two configuration methods are available: constant set and table field; When selecting a constant set, enter the fixed request parameter value. If there are multiple values, separate them with commas. When selecting a table field, select a table in a relational database, and then select a field in the table, which means that all column values ​​of a field in the table are used as parameter values; (2) Header type request parameter: In this method, the header information of the request interface is set in the form of Key-Value. The header information includes: authentication information and response data format; (3) Body type request parameters: When the interface request method is selected as POST request method, set the Body type request parameters.

4. The method for integrating API interface data according to claim 3, characterized in that: In this method, for an API interface with multiple parameters, the interface parameters are combined by Cartesian product according to the number of parameters and parameter values, and finally a Cartesian product parameter group is formed; When there are many parameter values ​​corresponding to a parameter, a Cartesian product parameter group is formed by the following parameter group caching method: (1) After the task configuration is completed, the program calculates the parameter combination of all parameters and caches them in ElasticSearch in advance; (2) Before making an API call, first retrieve all parameter combinations for the task from ElasticSearch, and use one set of parameters to make sequential API calls; (3) When the parameter values ​​in the task configuration are adjusted, the parameter combination stored in ElasticSearch is updated synchronously to ensure that the parameter combination obtained when the next API interface is called is the latest.

5. The method for integrating API interface data according to claim 1, characterized in that: Step S2 also includes: in the task configuration stage, the program parses the successful response example data, displays the nodes at different levels to form row labels, and the user selects the level nodes to be collected through the visual operation interface; when a level node is selected, in addition to the root node, a node path composed of the various parent nodes of the node will be formed, and the node path uses the dot ". " to splice the names of the various parent nodes.

6. The method for integrating API interface data according to claim 1, characterized in that: Step S2 also includes: when the target end table does not exist, generating the corresponding field name of the target end table through field remark mapping; When the target end table is newly created, by establishing a mapping relationship between the node path and the target end table field, it is specified that the different node data of the interface return result will be saved in the corresponding target end table field.

7. The method for integrating API interface data according to claim 1, characterized in that: In the task configuration phase described in step S2, setting the data format conversion relationship between the data nodes to be collected and the target end table fields includes: (1) Based on the mapping relationship between the source node path and the target table field, determine the target time field CJSJ mapped to the source node path; (2) Determine the data format of the time field CJSJ according to the target data source type and the type of the target time field CJSJ; (3) After determining the data format of the target end time field CJSJ, set the corresponding data conversion format for the source end node path.

8. The method for integrating API interface data according to claim 1, characterized in that: The scheduling strategy described in step S3 includes: Limit the number of calls: Set the total number of calls to the API interface. If the API interface provider does not limit the number of calls to the API interface, set the total number of calls to unlimited. Interface call interval: used to control the time interval between two adjacent interface calls; Scheduling cycle: used to set the calling frequency of the API interface; Scheduling validity period: used to set the validity period of the API interface data collection task. When the validity period is reached, the API interface collection task will automatically stop; Paging request: paging request configuration is performed based on the paging request parameters in the basic configuration parameters; the paging request parameters include page number parameters and number of requests per page parameters; the page number parameters include three types of transformation: increasing, decreasing and unchanged: (a) When the page number parameter is set to increment, the page number parameter passed will increase in sequence each time the API interface is requested; (b) When the page number parameter is set to decrease, the page number parameter passed will decrease in sequence each time the API interface is requested; (c) When the page number parameter is set to unchanged, the page number parameter passed will not change each time the API interface is requested and will remain the same each time.

9. The method for integrating API interface data according to claim 1, characterized in that: The API interface data collection task is performed by using the ApiReader read plug-in and the JdbcWriter write plug-in described in step S4, including: (1) The ApiReader plug-in obtains the basic configuration parameters of the task, queries all parameter groups from ElasticSearch, and starts requesting the API interface to obtain the result data; (2) After the ApiReader plug-in obtains the result data, it parses and extracts the result data according to the configured node path that needs to be collected; (3) After the ApiReader plug-in obtains the data value corresponding to the node path, it converts the data format according to the configured data format conversion relationship; (4) The ApiReader plug-in writes the format-converted data into the channel pipeline; (5) The JdbcWriter writing plug-in obtains input data from the channel pipeline, and then writes the source data into the corresponding fields of the target table based on the field mapping relationship.

10. An integration device for API interface data, characterized in that: When the device is running, the steps of the method for integrating API interface data according to any one of claims 1 to 9 are implemented, and the device includes: Task basic configuration module: This module provides a visual interface for basic configuration of API interface data collection tasks. The basic configuration parameters include: interface request method, data response format, interface address, request parameters, successful response example, and successful response parameters. Task running module: This module is used to configure and execute API interface data collection tasks, including: parsing of API interface return result data, data format conversion configuration, scheduling strategy configuration, and task execution; Interface request exception handling module: The function of this module is to configure the number of failed retries. When an interface request fails during the process of calling the API interface to collect data, the program automatically retries in sequence according to the set number of retries until the request succeeds.

Images