A method and system for implementing API call information statistics based on InfluxDB

Abstract

The present invention discloses a method and system for implementing API call information statistics based on InfluxDB, belonging to the field of API calls; the specific steps of the method are as follows: S1 uses the gateway kong to save the published API call information to a cache database; S2 regularly saves the call information in the cache database to InfluxDB; S3 uses the interface of InfluxDB to query the call status of the API; S4 directly saves the processed standard data to InfluxDB in the plug-in; the present invention provides a method for implementing API call information statistics based on InfluxDB, wherein an API is created by an API management console, and the call information is saved to the cache database when the API is called, and then the call information is saved from the cache database to InfluxDB by a scheduled task; the preservation of API call information based on InfluxDB is simpler and more high-performance than that of msql, and InfluxDB supports sql-like syntax and index serialization, can be automatically aggregated, and is faster and more efficient in subsequent queries.

Description

Technical Field

[0001] The invention discloses a method and a system for realizing API call information statistics based on InfluxDB, and relates to the technical field of API calls. Background Art

[0002] Kong is a highly available and easily extensible APIGateway project based on the Nginx_Lua module and open sourced by Mashape. Since Kong is based on Nginx, multiple Kong servers can be horizontally expanded, and requests can be evenly distributed to each server through the front-end load balancing configuration to handle large quantities of network requests.

[0003] Kong uses a plug-in mechanism for function customization. A set of plug-ins (can be 0 or n) is executed during the life cycle of the API request response cycle. The plug-in is written in Lua and currently has several basic functions: HTTP basic authentication, key authentication, CORS (Cross-origin Resource Sharing), TCP, UDP, file logging, API request current limiting, request forwarding, and nginx monitoring.

[0004] Cache database is one of the most popular NoSQL databases. It is an open source key-value pair storage database written in ANSI C, which contains multiple data structures, supports networks, is memory-based, and has optional persistence. It has the following features:

[0005] Based on memory operation, high performance

[0006] Supports distribution and can be expanded infinitely in theory

[0007] Key-value storage system

[0008] Open source written in ANSI C, compliant with BSD protocol, supports network, can be based on memory or persistent log type, Key-Value database, and provides API in multiple languages

[0009] Compared with other database types, cache databases have the following characteristics:

[0010] C / S communication model

[0011] Single process single thread model

[0012] Rich data types

[0013] The operation is atomic

[0014] persistence

[0015] High concurrent read and write

[0016] After adding Lua script support to Kong, each client request to the API will first reach Kong and then be proxied to the final API. Between the request and the response, Kong will execute any installed plug-ins to expand the API feature set. Kong effectively becomes the entry point for each API.

[0017] InfluxDB is an open source time series database developed by InfluxData. A time series database is a database that stores data based on time. Each piece of data has a timestamp, so this database is particularly suitable for storing data that changes over time. After being processed by some tools, the trend of data changes over time can be analyzed. InfluxDB is written in Go and focuses on high-performance query and storage of time series data. InfluxDB is widely used in scenarios such as monitoring data of storage systems and real-time data in the IoT industry. Summary of the invention

[0018] In view of the problems of the prior art, the present invention provides a method and system for implementing API call information statistics based on InfluxDB. The technical solution adopted is: a method for implementing API call information statistics based on InfluxDB, wherein the specific steps of the method are as follows:

[0019] S1 uses the gateway kong to save the published API call information to the cache database;

[0020] S2 periodically saves the call information in the cache database to InfluxDB;

[0021] S3 uses the InfluxDB interface to query API calls;

[0022] S4 saves the data processed in the plugin directly to InfluxDB.

[0023] The specific steps of S2 periodically saving the call information in the cache database to InfluxDB are as follows:

[0024] S201 is a scheduled task that periodically obtains data to be saved from the cache database;

[0025] S202 scheduled task saves the acquired data to InfluxDB;

[0026] S203 The scheduled task determines whether the data is saved successfully and deletes the saved data successfully.

[0027] The specific steps of using the InfluxDB interface to query the API call status of S3 are as follows:

[0028] S301 uses a scheduled task to batch obtain API call information from the cache database;

[0029] S302 uses the InfluxDB SDK to save data in batches to InfluxDB. After the data is saved successfully, the call information is deleted from the cache database.

[0030] The specific steps of using a scheduled task to batch obtain API call information from the cache database in S301 are as follows:

[0031] S3011API calls save information in the plugin using APIg:monitor: as the beginning of the key;

[0032] S3012 uses the scan command of the cache database to obtain a certain number of key sets to obtain the cursor of the cache database scan data;

[0033] S3013 repeats the above operation until the cursor value reaches zero, then ends the current task.

[0034] The S302 uses the InfluxDB SDK to save the data in batches to InfluxDB. After the data is saved successfully, the specific steps of deleting the call information from the cache database are as follows:

[0035] S3021 determines whether there are other operation processes according to the value in the cache database;

[0036] S3022 writes identification data into the cache database;

[0037] S3023 identifies the data value using a timestamp;

[0038] S3024 The current thread deletes the identification data in the cache database after completing the synchronization of data.

[0039] A system for implementing API call information statistics based on InfluxDB, the system specifically includes a data cache module, an information storage module, a scheduled task module and a data processing module:

[0040] Data cache module: Use the gateway kong to save the published API call information to the cache database;

[0041] Information saving module: regularly saves the call information in the cache database to InfluxDB;

[0042] Scheduled task module: Use the InfluxDB interface to query API call status;

[0043] Data processing module: In the plug-in, the processed data is saved directly into InfluxDB.

[0044] The information storage module specifically includes a timing acquisition module, a timing storage module and a timing processing module:

[0045] Scheduled acquisition module: Scheduled tasks periodically retrieve the data to be saved in the cache database;

[0046] Scheduled saving module: The scheduled task saves the acquired data into InfluxDB;

[0047] Timing processing module: The timing task determines whether the data is saved successfully and deletes the saved data successfully.

[0048] The scheduled task module specifically includes an information acquisition module and an information processing module:

[0049] Information acquisition module: Use scheduled tasks to batch acquire API call information from the cache database;

[0050] Information processing module: Use the InfluxDB SDK to save data in batches to InfluxDB, and delete the call information from the cache database after successful saving.

[0051] The information acquisition module specifically includes a plug-in configuration module, a cursor processing module and a task processing module:

[0052] Plugin configuration module: API call save information in the plugin using APIg:monitor: as the beginning of the key;

[0053] Cursor processing module: uses the scan command of the cache database to obtain a certain number of key sets to obtain the cursor of the cache database scan data;

[0054] Task processing module: Repeat the above operations until the cursor value reaches zero, then end the current task.

[0055] The information processing module specifically includes a process determination module, an identification writing module, an identification processing module and an identification deletion module:

[0056] Process judgment module: judges whether there are other operation processes according to the value in the cache database;

[0057] Identification writing module: writes identification data into the cache database;

[0058] Identification processing module: The value of identification data uses a timestamp;

[0059] Identification deletion module: The current thread deletes the identification data in the cache database after the synchronization data is completed.

[0060] The beneficial effects of the present invention are as follows: the present invention provides a method for implementing API call information statistics based on InfluxDB, wherein an API is created by an API management console, and the call information is saved in a cache database when the API is called, and then a scheduled task saves the call information from the cache database to InfluxDB; the preservation of API call information based on InfluxDB is simpler and more performant than that of msql, and InfluxDB supports sql-like syntax and index serialization, can automatically aggregate, and is faster and more efficient in subsequent queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0062] Figure 1 is a flow chart of an embodiment of the method of the present invention; Figure 2 It is a flow chart of the scheduled task execution capability of an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0064] Embodiment 1:

[0065] A method for implementing API call information statistics based on InfluxDB, wherein the specific steps of the method are as follows:

[0066] S1 uses the gateway kong to save the published API call information to the cache database;

[0067] S2 periodically saves the call information in the cache database to InfluxDB;

[0068] S3 uses the InfluxDB interface to query API calls;

[0069] S4 saves the processed specification data directly to InfluxDB in the plugin;

[0070] The Lua code is used to implement the function of saving Kong's API call status to the cache. Redis is used as the cache database, and InfluxDB is used as the database for saving API call status. When the user's API is called, the call status is saved in the cache, and then a scheduled task is used to save the data in the cache to InfluxDB.

[0071] Users do not need to bind plug-ins when creating APIs on the console page. Kong will execute the plug-in that saves the call status when the API is called and save the data into the cache database;

[0072] Cache Format

[0073] Use hash format to store application and API call information

[0074]

[0075]

[0076] Furthermore, the specific steps of S2 periodically saving the call information in the cache database to InfluxDB are as follows:

[0077] S201 is a scheduled task that periodically obtains data to be saved from the cache database;

[0078] S202 scheduled task saves the acquired data to InfluxDB;

[0079] S203 The scheduled task determines whether the data is saved successfully and deletes the saved data successfully;

[0080] Use a scheduled task to batch obtain API call information from the redis cache, use the InfluxDB SDK to batch save the data to InfluxDB, and delete the call information from the redis cache after successful saving; the data has been processed into standard data saved in InfluxDB in the plug-in, so there is no need for a scheduled task to process the data separately, and the data can be saved directly;

[0081] Furthermore, the specific steps of using the InfluxDB interface to query the API call status of S3 are as follows:

[0082] S301 uses a scheduled task to batch obtain API call information from the cache database;

[0083] S302 uses the InfluxDB SDK to save the data in batches to InfluxDB. After the data is saved successfully, the call information is deleted from the cache database.

[0084] Use the @Scheduled annotation in SpringBoot to create a scheduled task, and the frequency of scheduled task execution is 1 minute

[0085] @Scheduled(cron="0 0 / 1***?")

[0086] Furthermore, the specific steps of using a scheduled task in S301 to batch obtain API call information from the cache database are as follows:

[0087] S3011API calls save information in the plugin using APIg:monitor: as the beginning of the key;

[0088] S3012 uses the scan command of the cache database to obtain a certain number of key sets to obtain the cursor of the cache database scan data;

[0089] S3013 repeats the above operation until the cursor value reaches zero, then the current task is terminated;

[0090] The API call saves information in the plug-in with APIg:monitor: as the beginning of the key, and uses the redis scan command to take a certain number of key sets each time, and at the same time get the redis scan data cursor. The cursor is the key value of the scan data to ensure that no duplicate data is obtained.

[0091] Using the pipeline feature of redis, you only need one request to batch retrieve API call save information based on the key set obtained from redis;

[0092] Repeat the above steps until the cursor value is 0, indicating that there is no unsynchronized API call information in redis, and the current task can be terminated.

[0093] Furthermore, the S302 uses the InfluxDB SDK to save the data in batches to InfluxDB. After the data is saved successfully, the specific steps of deleting the call information from the cache database are as follows:

[0094] S3021 determines whether there are other operation processes according to the value in the cache database;

[0095] S3022 writes identification data into the cache database;

[0096] S3023 identifies the data value using a timestamp;

[0097] S3024 The current thread deletes the identification data in the cache database after completing the synchronization of data;

[0098] At the beginning of the task, determine whether there are other processes operating based on the value in redis. If there are other processes synchronizing data, immediately end the current task and wait for the next task cycle to execute again. If there are no other processes synchronizing data, proceed to the following process;

[0099] When the task starts, a set of data is written to redis, indicating that there is currently a thread synchronizing data and other threads do not need to operate;

[0100] The identification data is as follows

[0101] Key APIg:kong:monitor

[0102] Value Timestamp

[0103] Using timestamps to identify data values ​​can prevent exceptions during the execution of the current thread, which can cause long-term data synchronization issues.

[0104] The current thread deletes the identification data in redis after the synchronization data is completed

[0105] Use process locks to ensure that only one process operates the data, ensuring that there will be no confusion in data processing and ensuring the validity and security of the data;

[0106] Each piece of API call information data retrieved from redis has been processed into a standard InfluxDB insert statement in the plug-in and does not need to be processed again;

[0107] Create an InfluxDB connection and specify the database name and storage strategy. The storage strategy is created when the database is created and must be specified when creating a connection. Use the batch data saving method in the SDK to save the API call information data obtained from redis.

[0108] After saving successfully, close the InfluxDB connection just created to avoid wasting InfluxDB resources and affecting InfluxDB queries;

[0109] After synchronizing the API call information data obtained from Reis to InfluxDB, you need to delete this data from Redis to ensure that this data is synchronized only once

[0110] Use redis's transactional call method to delete multiple keys at one time. If the deletion fails, roll back the transaction to ensure data integrity.

[0111] Embodiment 2:

[0112] A system for implementing API call information statistics based on InfluxDB, the system specifically includes a data cache module, an information storage module, a scheduled task module and a data processing module:

[0113] Data cache module: Use the gateway kong to save the published API call information to the cache database;

[0114] Information saving module: regularly saves the call information in the cache database to InfluxDB;

[0115] Scheduled task module: Use the InfluxDB interface to query API call status;

[0116] Data processing module: saves the processed data directly into InfluxDB in the plug-in;

[0117] Furthermore, the information storage module specifically includes a timing acquisition module, a timing storage module and a timing processing module:

[0118] Scheduled acquisition module: Scheduled tasks periodically retrieve the data to be saved in the cache database;

[0119] Scheduled saving module: The scheduled task saves the acquired data into InfluxDB;

[0120] Timing processing module: The timing task determines whether the data is saved successfully and deletes the saved data successfully;

[0121] Furthermore, the scheduled task module specifically includes an information acquisition module and an information processing module:

[0122] Information acquisition module: Use scheduled tasks to batch acquire API call information from the cache database;

[0123] Information processing module: Use the InfluxDB SDK to save data in batches to InfluxDB. After successful saving, delete the call information from the cache database;

[0124] Furthermore, the information acquisition module specifically includes a plug-in configuration module, a cursor processing module and a task processing module:

[0125] Plugin configuration module: API call save information in the plugin using APIg:monitor: as the beginning of the key;

[0126] Cursor processing module: uses the scan command of the cache database to obtain a certain number of key sets to obtain the cursor of the cache database scan data;

[0127] Task processing module: Repeat the above operations until the cursor value reaches zero, then the current task ends;

[0128] Furthermore, the information processing module specifically includes a process determination module, an identification writing module, an identification processing module and an identification deletion module:

[0129] Process judgment module: judges whether there are other operation processes according to the value in the cache database;

[0130] Identification writing module: writes identification data into the cache database;

[0131] Identification processing module: The value of identification data uses a timestamp;

[0132] Identification deletion module: The current thread deletes the identification data in the cache database after the synchronization data is completed.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for implementing API call information statistics based on InfluxDB, characterized by: The specific steps of the method are as follows: S1 uses the gateway kong to save the published API call information to the cache database; S2 periodically saves the call information in the cache database to InfluxDB; Specifically: The specific steps for S2 to periodically save the call information in the cache database to InfluxDB are as follows: S201 scheduled task periodically obtains data from the cache database; S202 scheduled task saves the acquired data to InfluxDB; S203 The scheduled task determines whether the data is saved successfully. If the data is saved successfully, the data saved in the cache database is deleted; S3 uses the InfluxDB interface to query API calls; Specifically, the specific steps of using the InfluxDB interface to query the API call status of S3 are as follows: S301 uses a scheduled task to batch obtain API call information from the cache database; Specifically, the specific steps of using a scheduled task to batch obtain API call information from a cache database in S301 are as follows: S3011API calls save information in the plugin using APIg:monitor: as the beginning of the key; S3012 uses the scan command of the cache database to obtain a certain number of key sets to obtain the cursor of the cache database scan data; S3013 repeats the above operation until the cursor value reaches zero, then the current task is terminated; S302 uses the InfluxDB SDK to save the data in batches to InfluxDB. After the data is saved successfully, the call information is deleted from the cache database. S4 saves the data processed in the plugin directly to InfluxDB.

2. The method according to claim 1, characterized in that The S302 uses the InfluxDB SDK to save the data in batches to InfluxDB. After the data is saved successfully, the specific steps of deleting the call information from the cache database are as follows: S3021 determines whether there are other operation processes according to the value in the cache database; S3022 writes identification data into the cache database; S3023 identifies the data value using a timestamp; S3024 The current thread deletes the identification data in the cache database after completing the synchronization of data.

3. A system for implementing API call information statistics based on InfluxDB, characterized by: The system specifically includes a data cache module, an information storage module, a timed task module and a data processing module: Data cache module: Use the gateway kong to save the published API call information to the cache database; Information saving module: regularly saves the call information in the cache database to InfluxDB; Scheduled task module: Use the InfluxDB interface to query API call status; Data processing module: saves the processed data directly into InfluxDB in the plug-in; The information storage module specifically includes a timing acquisition module, a timing storage module and a timing processing module: Scheduled acquisition module: Scheduled tasks are used to obtain data from the cache database at regular intervals; Scheduled saving module: The scheduled task saves the acquired data into InfluxDB; Timing processing module: The timing task determines whether the data is saved successfully. If the data is saved successfully, the data saved in the cache database is deleted; The scheduled task module specifically includes an information acquisition module and an information processing module: Information acquisition module: Use scheduled tasks to batch obtain API call information from the cache database; Information processing module: Use the InfluxDB SDK to save data in batches to InfluxDB, and delete the call information from the cache database after successful saving; The information acquisition module specifically includes a plug-in configuration module, a cursor processing module and a task processing module: Plugin configuration module: API call save information in the plugin using APIg:monitor: as the beginning of the key; Cursor processing module: uses the scan command of the cache database to obtain a certain number of key sets to obtain the cursor of the cache database scan data; Task processing module: Repeat the above operations until the cursor value reaches zero, then end the current task.

4. The system according to claim 3, characterized in that The information processing module specifically includes a process determination module, an identification writing module, an identification processing module and an identification deletion module: Process judgment module: judges whether there are other operation processes according to the value in the cache database; Identification writing module: writes identification data into the cache database; Identification processing module: The value of identification data uses a timestamp; Identification deletion module: The current thread deletes the identification data in the cache database after the synchronization data is completed.

Images