An integration method for API interface data

Abstract

The present invention relates to an integration method for API interface data. The method includes: performing basic configuration for the API interface data collection task; specifying the data level and node to be collected and setting the data format conversion relationship between the data node to be collected and the target end table field in the task configuration phase; performing interface return result data analysis and data level and node selection and completing data format conversion in the task execution phase; screening out the interface data to be collected and the corresponding node path, designing the node path as a data table, using the node path as the field English name and the node name as the field note; configuring the scheduling strategy for API interface calls; executing the API interface data collection task through the ApiReader read plug-in and the JdbcWriter write plug-in. The method can realize the automatic collection and storage of API interface data, and supports API interface requests with a large number of parameter combinations, which can effectively solve the problems of low efficiency and high labor cost of manually calling the API interface to collect data.

Description

Technical Field

[0001] The present invention relates to the field of data synchronization technology, and in particular to a method and device for integrating API interface data in the field of data synchronization. Background Art

[0002] Nowadays, data has become a new factor of production and is vigorously promoting social development. Many companies have obtained huge commercial value from data, and government departments, as the owners of massive data, are also actively promoting the sharing and opening of government data. One form of openness is to share anonymized data in the form of API interfaces. Companies promote their business success by collecting data from these API interfaces and making reasonable use of them.

[0003] However, traditional API interface data collection generally requires customized development through coding, which has problems such as low collection efficiency and high labor costs. It is also impossible to accurately control the call frequency and call interval, and cannot automatically perform data result analysis and data conversion. There is also a lack of mature automated API interface collection tools on the market. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of traditional API interface data collection methods, the present invention proposes a new integration method for API interface data in the field of data synchronization. This method can effectively solve the problems of low efficiency and high labor cost of manually calling API interface to collect data.

[0005] In order to achieve the above object, the present invention mainly makes the following designs:

[0006] (1) This method can realize the automatic collection and storage of API interface data through visual front-end configuration and back-end data collection engine. Moreover, during the automatic collection of API interface data, this method can accurately set the interface request frequency, request interval and request validity period; it can also automatically parse and convert the result data collected by the API interface into data format. When the amount of collected data is large, paging requests can be made according to the configuration; this method also supports a request failure retry mechanism, which can automatically retry the request after the interface request fails.

[0007] (2) This method uses a visual interface to flatten the complex response data results, simplifying the configuration process of API interface data collection.

[0008] Specifically, the present invention provides an integration method for API interface data, such as Figure 13 As shown, the method includes the following steps:

[0009] S1. Perform basic configuration for the API interface data collection task. Basic configuration parameters include but are not limited to: interface request method, data response format, interface address, request parameters, verification method, successful response example, and successful response parameters;

[0010] S2. Parsing and converting the data format of the API interface return result (response result) includes: specifying the data level and nodes to be collected during the task configuration phase; parsing the interface return result data and selecting the data level and nodes during the task execution phase; filtering the interface data to be collected and the corresponding node paths based on the selected data nodes at different levels; designing the obtained node paths into a data table, where the node paths are used as the English names of the fields and the node names are used as the field comments; setting the data format conversion relationship between the data nodes to be collected and the target end table fields during the task configuration phase; and completing the data format conversion according to the data format conversion relationship during the task execution phase;

[0011] S3. Configure the scheduling policy for API interface calls, including configuration: limit the number of calls, interface call interval, scheduling cycle, scheduling start time, scheduling validity period, and paging requests;

[0012] S4. Execute API interface data collection tasks through the ApiReader read plug-in and the JdbcWriter write plug-in.

[0013] Furthermore, in the basic configuration parameters described in step S1 of the API interface data integration method of the present invention:

[0014] The interface request method supports GET request and POST request;

[0015] The data response format supports interface response data in JSON and XML formats;

[0016] The interface address is a URL for an http request;

[0017] The request parameters include paging request parameters;

[0018] The successful response example provides an example of the return result data when the requested API interface successfully responds. It supports interface return result data in JSON and XML formats. If you are unsure about the type of interface return result data, perform an interface connectivity test. When the interface request is successful, the interface return result is automatically filled in.

[0019] The successful response parameter is an indicator of a successful interface request. Select an attribute node from the successful response example as the successful response parameter. The successful response parameter is identified by the code response code and supports setting specified values ​​and user-defined values.

[0020] Furthermore, the request parameters in the basic configuration parameters described in step S1 of the API interface data integration method of the present invention include the following types:

[0021] (1) KV (Key-Value) type request parameter: Set the parameter name, parameter type, whether it is a paging parameter, parameter description, and the parameter value corresponding to the parameter name of the KV type request parameter;

[0022] When setting parameter values ​​for KV type request parameters, there are two configuration methods: constant set and table field;

[0023] When selecting a constant set, enter the fixed request parameter value. If there are multiple values, separate them with commas.

[0024] When selecting a table field, select a table in a relational database and then select a field in the table. This means that all column values ​​of a field in the table will be used as parameter values.

[0025] (2) Header type request parameters: In this method, the header information of the request interface is set in the form of key-value. The header information includes but is not limited to: authentication information and response data format;

[0026] (3) Body type request parameters: When the interface request method is selected as POST request method, set the Body type request parameters.

[0027] Furthermore, the present invention is directed to an integration method for API interface data. In this method, for an API interface with multiple parameters, the interface parameters are combined by performing a Cartesian product based on the number of parameters and the parameter values, and finally forming a Cartesian product parameter group.

[0028] When there are many parameter values ​​corresponding to a parameter, the parameter group of the Cartesian product is formed by the following parameter group caching method:

[0029] (1) After the task configuration is completed, the program calculates the parameter combinations of all parameters and caches them in ElasticSearch (distributed search and analysis engine) in advance;

[0030] (2) Before making an API call, all parameter combinations for the task are retrieved from ElasticSearch, and a set of parameters is used to make sequential API calls.

[0031] (3) When the parameter values ​​in the task configuration are adjusted, the parameter combination stored in ElasticSearch is updated synchronously to ensure that the parameter combination obtained when the next API interface is called is the latest.

[0032] Furthermore, step S2 of the integration method for API interface data of the present invention also includes: in the task configuration stage, the program parses the successful response example data, displays the nodes of different levels to form row labels, and the user checks the level nodes that need to be collected through the visual operation interface; when a level node is checked, in addition to the root node, a node path composed of the various parent nodes of the node will be formed, and the node path is spliced ​​with the names of the various parent nodes through the dot ".". In this way, the data node that needs to be collected can be accurately located. In addition, for node names with the same name in different levels, this method can also be used to distinguish them.

[0033] Furthermore, the method for integrating API interface data of the present invention further comprises, in step S2: when the target end table does not exist, generating the corresponding field name of the target end table by mapping the field note (i.e., the node name);

[0034] When the target table is created, a mapping relationship between the node path and the target table field is established to specify that the different node data of the interface return result be saved in the corresponding target table field.

[0035] Furthermore, the present invention is directed to the API interface data integration method step S2, in the task configuration phase, setting the data format conversion relationship between the data nodes to be collected and the target end table fields, including:

[0036] (1) Based on the mapping relationship between the source node path and the target table field, confirm the target time field CJSJ mapped to the source node path;

[0037] (2) Determine the data format of the time field CJSJ based on the target end data source type and the type of the target end time field CJSJ;

[0038] (3) After determining the data format of the target end time field CJSJ, set the corresponding data conversion format for the source end node path.

[0039] Furthermore, the scheduling strategy described in step S3 of the API interface data integration method of the present invention includes:

[0040] Limit the number of calls: Set the total number of calls to the API interface. If the API interface provider does not limit the number of calls to the API interface, set the total number of calls to unlimited.

[0041] Interface call interval: used to control the time interval between two adjacent interface calls;

[0042] Scheduling cycle: used to set the calling frequency of the API interface;

[0043] Scheduling validity period: used to set the validity period of the API interface data collection task. When the validity period is reached, the API interface collection task will automatically stop.

[0044] Paging request: Configure paging request based on the paging request parameters in the basic configuration parameters; the paging request parameters include the page number parameter (pageNo) and the number of requests per page parameter (pageSize); the page number parameter includes three transformation types: increasing, decreasing, and unchanged.

[0045] (a) When the page number parameter is set to increment, the page number parameter passed will increase in sequence each time the API interface is requested;

[0046] (b) When the page number parameter is set to decrement, the page number parameter passed will decrease in sequence each time the API interface is requested;

[0047] (c) When the page number parameter is set to unchanged, the page number parameter passed will not change each time the API interface is requested and will remain the same each time.

[0048] Furthermore, the present invention is directed to the API interface data integration method step S4, wherein the API interface data collection task is performed by using the ApiReader reading plug-in and the JdbcWriter writing plug-in, including:

[0049] (1) The ApiReader plug-in obtains the basic configuration parameters of the task, queries all parameter groups from ElasticSearch, and starts requesting the API interface to obtain the result data;

[0050] (2) After the ApiReader plug-in obtains the result data, it parses and extracts the result data according to the configured node path that needs to be collected;

[0051] (3) After the ApiReader plug-in obtains the data value corresponding to the node path, it converts the data format according to the configured data format conversion relationship;

[0052] (4) The ApiReader plug-in writes the format-converted data into the channel pipeline;

[0053] (5) The JdbcWriter write plug-in obtains input data from the channel pipeline, and then writes the source data to the corresponding fields of the target table based on the field mapping relationship.

[0054] On the other hand, the present invention also provides an integration device for API interface data, which implements the steps of the above-mentioned integration method for API interface data when the device is running, and the device includes:

[0055] Task Basic Configuration Module: This module provides a visual interface for basic configuration of API interface data collection tasks. Basic configuration parameters include but are not limited to: interface request method, data response format, interface address, request parameters, verification method, successful response example, and successful response parameters.

[0056] Task running module: This module is used to configure and execute API interface data collection tasks, including parsing the result data returned by the API interface, configuring data format conversion, configuring scheduling strategies, and executing tasks.

[0057] Interface request exception handling module: The function of this module is to configure the number of failed retries. When an interface request fails during the process of calling the API interface to collect data, the program automatically retries in sequence according to the set number of retries until the request is successful.

[0058] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the above-mentioned integration method for API interface data are implemented.

[0059] In summary, the API interface data integration method of the present invention has the following advantages:

[0060] (1) The parameter group caching scheme designed by this method can support API interface requests with a large number of parameter combinations.

[0061] (2) This method can realize the automatic collection and storage of API interface data through visual front-end configuration and back-end data collection engine.

[0062] (3) This method parses and flattens the complex response data results through a visual interface, simplifying the configuration process of API interface data collection.

[0063] (4) During the automated data collection process of the API interface, this method can accurately set the request frequency, request interval, and request validity period of the interface; it can also automatically parse and convert the result data collected by the API interface into data format, thereby ensuring the accuracy of the stored data.

[0064] (5) During the automated data collection process of the API interface, when the amount of collected data is large, this method can perform paging requests based on the configuration, thereby ensuring the integrity of the data collected by the API interface.

[0065] (6) During the automated collection of API interface data, this method also supports a request failure retry mechanism, which can automatically retry requests after an interface request fails, and has good fault tolerance. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the description of the present invention. Obviously, the following drawings are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 Schematic diagram of the basic configuration of tasks in an embodiment of the present invention.

[0068] Figure 2 Schematic diagram of the Cartesian product combination of two parameters in an embodiment of the present invention.

[0069] Figure 3 This is a schematic diagram of the Header type request parameter configuration in an embodiment of the present invention.

[0070] Figure 4 This is a schematic diagram of response result analysis and node selection in an embodiment of the present invention.

[0071] Figure 5 Schematic diagram of field mapping in an embodiment of the present invention.

[0072] Figure 6 Schematic diagram of the data format conversion field mapping relationship in an embodiment of the present invention.

[0073] Figure 7 These are eight data conversion formats built into the embodiment of the present invention.

[0074] Figure 8 This is a schematic diagram of paging request parameter configuration of the task basic configuration module in an embodiment of the present invention.

[0075] Figure 9 This is a schematic diagram of paging request parameter configuration in the scheduling policy configuration in an embodiment of the present invention.

[0076] Figure 10 This is a schematic diagram of reading and writing of the data synchronization tool in an embodiment of the present invention.

[0077] Figure 11 This is a task execution flow chart for API interface collection in an embodiment of the present invention.

[0078] Figure 12 This is a schematic diagram of a failure retry configuration in an embodiment of the present invention.

[0079] Figure 13 The figure is a flow chart of the overall implementation of the method of the present invention. DETAILED DESCRIPTION

[0080] To make the objectives, technical solutions, and advantages of the present invention more clearly apparent, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The present invention may also be implemented or applied through different specific implementation methods, and the details in this specification may be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0081] At the same time, it should be understood that the scope of protection of the present invention is not limited to the specific embodiments described below; it should also be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the scope of protection of the present invention.

[0082] Example: A method and device for integrating API interface data

[0083] The device for integrating API interface data of the present invention includes the following modules:

[0084] (1) Task basic configuration module

[0085] The function of the task basic configuration module is to provide a visual interface for basic configuration of API interface data collection tasks.

[0086] like Figure 1 As shown in the figure, for API interface data collection, the core basic configuration includes: interface request method, data response format, interface address, request parameters, verification method, successful response example, successful response parameters, etc.

[0087] 1. Interface request method

[0088] Supports GET and POST request methods.

[0089] 2. Data response format

[0090] Supports interface response data in JSON and XML formats.

[0091] 3. Interface address

[0092] That is, the address of the API interface that needs to be requested. The interface address is generally a URL for an http request.

[0093] 4. Request parameters

[0094] (1) KV type request parameter configuration

[0095] Supports configuring Key-Value type request parameters, where you can set the parameter name, parameter type, whether it is a paging parameter, parameter description, and the parameter value corresponding to the parameter name.

[0096] When setting parameter values, you can choose between two configuration methods: constant set and table field. When selecting constant set, you can enter a fixed request parameter value. If there are multiple values, they are separated by commas. When selecting table field, you can select a table in a relational database and then select a field in the table. This means that all column values ​​of a field in the table are used as parameter values.

[0097] Table 1 Request parameter configuration example

[0098] Parameter name Parameter Type Whether to paginate Configuration method Parameter value param1 Character no Constant Set 1,2,3,4,5 param2 Numeric no Table fields Database, table, field name

[0099] For better explanation, here we take Table 1 as an example. Assume that two request parameters, param1 and parma2, are configured. The parameter value corresponding to parameter param1 is a constant set type. The user enters the constant set as "1,2,3,4,5" separated by commas. The program automatically recognizes it and splits it into five parameter values: 1,2,3,4,5.

[0100] If the parameter value corresponding to the parameter param2 is the table field type, the parameter value corresponding to this parameter is all column values ​​of the selected field.

[0101] In this method, for such API interfaces with multiple parameters, the parameters passed to the interface will be combined by Cartesian product based on the number of parameters and parameter values. Figure 2 As shown, all parameter values ​​of parameter param1 will be paired one by one with all parameter values ​​of parameter param2, and finally form a parameter group of Cartesian product. This embodiment takes 2 parameters as an example. If there are 3 or even more parameters, the composition principle of the parameter group is the same.

[0102] GET http: / / 127.0.0.1 / appgateway / user / getUserInfo? param1=1¶m2=11

[0103] When there are many parameter values ​​corresponding to the parameters, the number of parameter groups forming the Cartesian product is very large. It is difficult to accurately complete the assembly of such a large number of parameters for manual calling of API interface data collection. In actual application scenarios, API interface data collection often requires passing many parameters to obtain different data. Therefore, in order to solve this problem, the present invention specially designs a parameter group caching solution.

[0104] The parameter group caching scheme in the method of the present invention is as follows:

[0105] (a) After the task configuration is completed, the program will calculate the parameter combinations of all parameters and cache them in ElasticSearch (a distributed search and analysis engine) in advance. ElasticSearch can easily store and retrieve massive amounts of data.

[0106] (b) Before making an API call, all parameter combinations for the task are retrieved from ElasticSearch, and a set of parameters is used to make sequential API calls.

[0107] (c) When the parameter values ​​in the task configuration are adjusted, the parameter combination stored in ElasticSearch is updated synchronously to ensure that the parameter combination obtained during the next API call is the latest.

[0108] (2) Header type request parameter configuration

[0109] Header is the header information carried when sending a request, in which you can usually set the authentication information of the request: authentication / cookie, the data format of the response: content-type, etc. Figure 3 As shown, in the method of the present invention, it is also supported to set the header information when requesting the interface in the form of Key-Value.

[0110] (3)Body type request parameter configuration

[0111] When the interface request method is selected as POST request method, you can set the request parameter of Body type. The request parameter of Body type is more flexible. In this invention, the parameter type of application / json type is supported, which is also the most common json format. Its form is as follows:

[0112] {"param1":"value1","param2":"value2","param3":value3}

[0113] 5. Successful response example

[0114] The successful response example is an example of the return result data when the requested API interface successfully responds. In the present invention, the interface return result data of JSON type and XML type is supported.

[0115] When it is uncertain what type of data the interface returns, the present invention also provides an interface connectivity test. On the premise of ensuring that the interface request address and request parameters are correctly configured, by clicking the connectivity test button on the visual interface, when the interface request is successful, the interface return result will be automatically filled in.

[0116] 6. Success response parameters

[0117] The successful response parameter is an identifier that needs to be selected for a successful interface request. In the present invention, an attribute node is selected from a successful response example as an identifier for a successful interface request, and a code response code is generally used as the identifier.

[0118] The method of the present invention supports setting a specified value, such as 2xx; it also supports users setting a custom response code value.

[0119] (2) Task running module

[0120] The task running module is mainly used to perform configuration and task execution related to the API interface data collection task operation, including: parsing of the result data returned by the API interface and data format conversion configuration, scheduling strategy configuration, and task execution.

[0121] 1. Data analysis and data format conversion configuration

[0122] (1) Response result data analysis and node selection

[0123] When collecting data from an API interface, the data format returned by the interface is generally JSON or XML. Here, we take the commonly used JSON type as an example. When the returned result data is complex, there are generally many levels. When collecting data, it is often only necessary to obtain part of the data in the JSON data structure. If data is collected manually by calling the interface, the required data must be screened each time it is collected, which is not only time-consuming and labor-intensive, but also prone to errors. In order to simplify the configuration and collection process of API interface data synchronization tasks, the present invention specifies the data levels and nodes to be collected during the task configuration phase, achieving one-time configuration, automatic parsing, and no intervention required for subsequent operations.

[0124] The specific implementation method is as follows: During the configuration phase, the program will parse the successful response sample data and display the nodes at different levels to form row labels. Through the visual operation interface, users can quickly check the level nodes that need to be collected; when the corresponding level node is checked, except for the root node, a node path consisting of each parent node of the node will be formed; the node path is connected by the dot "." to connect the names of each parent node. Figure 4 As shown, when the id node is selected, the node path is data.rwpzRwpzxx.id. This way we can accurately locate the data node we want. In addition, this method can also be used to distinguish nodes with the same name in different levels.

[0125] (2) Field Mapping

[0126] like Figure 5 As shown in the figure, after selecting data nodes at different levels in the previous step and filtering out the interface data and corresponding node paths to be collected, this solution will design these node paths into a data table, with the node path as the field English name and the node name as the field comment.

[0127] The ultimate goal of collecting API data is to save the collected data in a database to facilitate subsequent statistics and analysis. Therefore, in this step, if the target table does not exist, the field notes on the left, i.e., the node name, will be mapped to the corresponding field name of the target table.

[0128] Once the target table is created, by establishing a mapping relationship between the left node path and the target table field, you can specify that the different node data returned by the interface be saved in the corresponding target table field, thus eliminating the manual processing process.

[0129] (3) Data format conversion

[0130] When collecting data through an API, the returned data format often doesn't meet the statistical and analytical requirements of the target data table. The most common issue is the date format. For example, the time in the result data obtained through an API call is in a 10- or 13-digit timestamp format (a timestamp is the total number of seconds from 00:00:00 Greenwich Mean Time (08:00:00 Beijing Time) on January 1, 1970, to the present). However, the target data table requires a time format of "yyyy-MM-ddHH:mm:ss." This data conversion is tedious for manual API calls, and manual processing is time-consuming and error-prone.

[0131] In view of this, the present invention sets the data format conversion relationship between the data nodes to be collected and the target end data table fields in advance during the task configuration stage, so that during the actual execution of the API interface call, the data format conversion work can be completed automatically and efficiently.

[0132] The specific steps are:

[0133] (a) Based on the mapping relationship between the source node path and the target field, for example Figure 6 The source node path is data.rwpzRwpzxx.cjsj, which represents the creation time in the result data collected through the API interface. Confirm that it is mapped to the time field CJSJ on the target end.

[0134] (b) Determine the data format of the time field CJSJ based on the target data source type and the type of the time field. For example, if the target data source is an Oracle data source and the CJSJ field is of type DATE, the Oracle database stores data of the DATE type in the format YYYY-MM-DD HH24:MI:SS, where YYYY represents the year, MM represents the month, DD represents the day, HH24 represents the hour in a 24-hour clock, MI represents the minutes, and SS represents the seconds. For example, the DATE value of 2:30:10 PM on May 1, 2019, is 2019-05-01 14:30:10. Therefore, data to be written to the target CJSJ field must be formatted as YYYY-MM-DD HH24:MI:SS to ensure data accuracy.

[0135] (c) After determining the data type of the field to be written on the target side, it is necessary to set the corresponding data conversion format for the node path on the source side. In the present invention, a total of eight built-in data formats are provided, such as Figure 7 As shown in the figure, if the source node path is data.rwpzRwpzxx.cjsj, the target field CJSJ corresponding to the path node is of the DATE field type, so it needs to be set to the date type (YMD H24:MI:S). After setting the data format of the source node path, the configuration will be saved and used when the API interface call is officially executed.

[0136] 2. Scheduling strategy configuration

[0137] Scheduling policy configuration mainly involves configuration related to API interface call policy, including: limiting the number of calls, interface call interval, scheduling cycle, scheduling start time, scheduling validity period, and paging request configuration.

[0138] In order to avoid the tedious work and possible deviations caused by manually calling the API interface to collect data, the present invention realizes automatic calling and paging requests of the API interface by configuring the API interface calling strategy in advance.

[0139] The following describes the above configuration in detail:

[0140] (1) Limit the number of calls

[0141] Since some API interface providers limit the number of calls to the API interface, in the present invention, the total number of calls to the API interface can be set. When the number of calls to the API interface is unlimited, it can also be set to unlimited.

[0142] (2)Interval between interface calls

[0143] The interface call interval is mainly used to control the time interval between two adjacent interface calls to avoid excessive pressure on the interface caused by too many requests in a short period of time.

[0144] (3) Scheduling cycle

[0145] The scheduling cycle mainly sets the calling frequency of the API interface, which can be set to yearly, monthly, weekly, daily, hourly, or minutely.

[0146] (4) Scheduling validity period

[0147] The scheduling validity period is used to set the validity period of the API interface data collection task. When the validity period is reached, the API interface collection will automatically stop.

[0148] (5) Paging request configuration

[0149] When configuring the paging request parameters in the task basic configuration module, such as Figure 8 As shown, in the scheduling configuration module, you can configure the paging request, such as Figure 9 shown.

[0150] For some API interfaces with large data volumes, if a large amount of data is requested at one time, the interface request may take too long, resulting in interface request failure. Therefore, in order to prevent excessive data from being requested at one time, the present invention designs an interface paging parameter to perform interface paging requests, thereby controlling the amount of data in a single API interface request.

[0151] Generally, two paging parameters are required (the parameter names may be different depending on the actual interface):

[0152] pageNo: page number parameter.

[0153] pageSize: parameter indicating the number of requests per page.

[0154] In the present invention, three types of transformations of page number parameters are supported: increasing, decreasing and unchanged.

[0155] When the page number parameter is set to increment, the page number parameter passed will increase in sequence each time the API interface is requested;

[0156] When the page number parameter is set to decrement, the page number parameter passed will decrease in sequence each time the API interface is requested;

[0157] When the page number parameter is set to unchanged, the page number parameter passed each time the API interface is requested will not change and will remain the same each time;

[0158] In this way, the API interface can automatically perform paging requests even when the amount of data returned is large, so that the data can be collected completely.

[0159] 3. Task Execution

[0160] This invention is based on an open source data synchronization tool and has developed its own API interface data acquisition plug-in. As an offline data synchronization framework, this tool adopts a Framework+plugin architecture. It abstracts the reading and writing of data sources into Reader / Writer plug-ins and incorporates them into the entire synchronization framework.

[0161] The channel pipeline mechanism is used to realize the communication between the source and the target: the Reader plug-in writes data to the channel, and the Writer plug-in reads data from the channel.

[0162] We need to collect API interface data here. The first step is to obtain input data from the API interface and then write it into the channel pipeline.

[0163] like Figure 10 and Figure 11 As shown, the specific execution process is as follows:

[0164] (1) The ApiReader plug-in obtains the basic configuration parameters of the task, queries all parameter groups from ElasticSearch, and starts requesting the API interface to obtain the result data.

[0165] (2) After the ApiReader plug-in obtains the result data, it parses and retrieves the result data according to the configured node path that needs to be collected.

[0166] (3) After the ApiReader plug-in obtains the data value corresponding to the node path, it converts the data format according to the configured data format conversion relationship.

[0167] (4) The ApiReader plug-in writes the format-converted data into the channel pipeline.

[0168] (5) The JdbcWriter writing plug-in obtains the input data from the channel pipeline, and then writes the data from the source end (input end) to the corresponding fields of the target end data table based on the field mapping relationship.

[0169] (3) Interface request exception handling module

[0170] When requesting an API interface, there is a possibility of call failure due to network jitter and instability. When the interface request fails, if it is not processed, there may be a risk of data loss. Therefore, in the present invention, an interface request failure retry mechanism is also designed to improve the fault tolerance of API interface data collection.

[0171] like Figure 12 As shown, after configuring the number of failed retries, when a request fails during the process of calling the API interface to collect data, the program will automatically retry according to the set number of retries until the request succeeds.

[0172] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession can make some changes or modifications to the technical content disclosed above without departing from the scope of the technical solution of the present invention to obtain equivalent embodiments with equivalent changes. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for integrating API interface data, characterized in that: The method comprises: S1. Perform basic configuration for the API interface data collection task. Basic configuration parameters include: interface request method, data response format, interface address, request parameters, successful response example, and successful response parameters. S2. Parsing and converting the result data returned by the API interface, including: specifying the data level and nodes to be collected during the task configuration phase; parsing the result data returned by the interface and selecting the data level and nodes during the task execution phase; filtering the interface data to be collected and the corresponding node paths based on the selected data nodes at different levels; designing the obtained node paths into a data table, with the node paths as the English names of the fields and the node names as the field comments; setting the data format conversion relationship between the data nodes to be collected and the target end table fields during the task configuration phase; and completing the data format conversion according to this data format conversion relationship during the task execution phase; S3. Configure the scheduling policy for API interface calls, including configuration: limit the number of calls, interface call interval, scheduling cycle, scheduling start time, scheduling validity period, and paging requests; S4. Execute API interface data collection tasks through the ApiReader read plug-in and the JdbcWriter write plug-in.

2. The method for integrating API interface data according to claim 1, characterized in that: Among the basic configuration parameters described in step S1: The interface request method supports GET request and POST request; The data response format supports interface response data in JSON and XML formats; The interface address is a URL for an http request; The request parameters include paging request parameters; The successful response example provides an example of the return result data when the requested API interface successfully responds. It supports interface return result data in JSON and XML formats. If you are unsure about the type of interface return result data, perform an interface connectivity test. When the interface request is successful, the interface return result is automatically filled in. The successful response parameter is an indicator of a successful interface request. Select an attribute node from the successful response example as the successful response parameter. The successful response parameter is identified by the code response code and supports setting specified values ​​and user-defined values.

3. The method for integrating API interface data according to claim 1, characterized in that: The request parameters in the basic configuration parameters described in step S1 include the following types: (1) KV type request parameters: Set the parameter name, parameter type, whether it is a paging parameter, parameter description, and the parameter value corresponding to the parameter name of the KV type request parameter; When setting parameter values ​​for KV type request parameters, there are two configuration methods: constant set and table field; When selecting a constant set, enter the fixed request parameter value. If there are multiple values, separate them with commas. When selecting a table field, select a table in a relational database and then select a field in the table. This means that all column values ​​of a field in the table will be used as parameter values. (2) Header type request parameters: In this method, the header information of the request interface is set in the form of key-value. The header information includes: authentication information and response data format; (3) Body type request parameters: When the interface request method is selected as POST request method, set the Body type request parameters.

4. The method for integrating API interface data according to claim 3, characterized in that: In this method, for API interfaces with multiple parameters, the interface parameters are combined by Cartesian product according to the number of parameters and parameter values, and finally a Cartesian product parameter group is formed; When there are many parameter values ​​corresponding to a parameter, the parameter group of the Cartesian product is formed by the following parameter group caching method: (1) After the task configuration is completed, the program calculates the parameter combinations of all parameters and caches them in ElasticSearch in advance; (2) Before making an API call, all parameter combinations for the task are retrieved from ElasticSearch, and a set of parameters is used to make sequential API calls. (3) When the parameter values ​​in the task configuration are adjusted, the parameter combination stored in ElasticSearch is updated synchronously to ensure that the parameter combination obtained when the next API interface is called is the latest.

5. The method for integrating API interface data according to claim 1, characterized in that: Step S2 also includes: in the task configuration stage, the program parses the successful response example data, displays the nodes at different levels to form row labels, and the user selects the level nodes to be collected through the visual operation interface; when a level node is checked, in addition to the root node, a node path consisting of the various parent nodes of the node will be formed, and the node path is spliced ​​with the names of the various parent nodes through the dot symbol ". ".

6. The method for integrating API interface data according to claim 1, characterized in that: Step S2 further includes: when the target end table does not exist, generating the corresponding field name of the target end table through field remark mapping; When the target table is created, a mapping relationship between the node path and the target table field is established to specify that the different node data of the interface return result be saved in the corresponding target table field.

7. The method for integrating API interface data according to claim 1, characterized in that: In the task configuration phase described in step S2, the data format conversion relationship between the data nodes to be collected and the target end table fields is set, including: (1) Based on the mapping relationship between the source node path and the target table field, confirm the target time field CJSJ mapped to the source node path; (2) Determine the data format of the time field CJSJ based on the target end data source type and the type of the target end time field CJSJ; (3) After determining the data format of the target end time field CJSJ, set the corresponding data conversion format for the source end node path.

8. The method for integrating API interface data according to claim 1, characterized in that: The scheduling strategy described in step S3 includes: Limit the number of calls: Set the total number of calls to the API interface. If the API interface provider does not limit the number of calls to the API interface, set the total number of calls to unlimited. Interface call interval: used to control the time interval between two adjacent interface calls; Scheduling cycle: used to set the calling frequency of the API interface; Scheduling validity period: used to set the validity period of the API interface data collection task. When the validity period is reached, the API interface collection task will automatically stop. Paging request: Configure paging request based on the paging request parameters in the basic configuration parameters; the paging request parameters include page number parameters and number of requests per page parameters; the page number parameters include three transformation types: increasing, decreasing, and unchanged. (a) When the page number parameter is set to increment, the page number parameter passed will increase in sequence each time the API interface is requested; (b) When the page number parameter is set to decrement, the page number parameter passed will decrease in sequence each time the API interface is requested; (c) When the page number parameter is set to unchanged, the page number parameter passed will not change each time the API interface is requested and will remain the same each time.

9. The method for integrating API interface data according to claim 1, characterized in that: The API interface data collection task described in step S4 is performed through the ApiReader reading plug-in and the JdbcWriter writing plug-in, including: (1) The ApiReader plug-in obtains the basic configuration parameters of the task, queries all parameter groups from ElasticSearch, and starts requesting the API interface to obtain the result data; (2) After the ApiReader plug-in obtains the result data, it parses and extracts the result data according to the configured node path that needs to be collected; (3) After the ApiReader plug-in obtains the data value corresponding to the node path, it converts the data format according to the configured data format conversion relationship; (4) The ApiReader plug-in writes the format-converted data into the channel pipeline; (5) The JdbcWriter write plug-in obtains input data from the channel pipeline, and then writes the source data to the corresponding fields of the target table based on the field mapping relationship.

10. An integration device for API interface data, characterized in that: When the device is running, the steps of the method for integrating API interface data according to any one of claims 1 to 9 are implemented, and the device includes: Task basic configuration module: This module provides a visual interface for basic configuration of API interface data collection tasks. Basic configuration parameters include: interface request method, data response format, interface address, request parameters, successful response example, and successful response parameters. Task running module: This module is used to configure and execute API interface data collection tasks, including parsing the result data returned by the API interface, configuring data format conversion, configuring scheduling strategies, and executing tasks. Interface request exception handling module: The function of this module is to configure the number of failed retries. When an interface request fails during the process of calling the API interface to collect data, the program automatically retries in sequence according to the set number of retries until the request is successful.

Images