API (Application Program Interface) service integrated management method and device based on network data warehouse, and medium

Abstract

The invention provides an API service integrated management method and device based on a network data warehouse and a medium, and the method comprises the steps: collecting data in a data source through a corresponding ETL tool according to at least one data source selected and configured by a user based on a first visual interface, carrying out the preprocessing of the data, and storing the data in a preset database; correspondingly updating the API service directory according to the management operation of the user on the API service directory in the second visual interface; receiving an API service custom instruction triggered by the user based on the API service directory, and popping up a third visual interface according to the API service custom instruction; and according to the service configuration information and the execution logic configuration information of the target API service set by the user through a guide type configuration mode in the third visual interface. According to the method, the device and the readable storage medium, the problems that an existing API service management method based on the network data warehouse is poor in API interface configuration flexibility and excessively depends on a third-party platform can be solved.

Description

Technical Field

[0001] The present invention relates to the field of network technology, and in particular to a method, device, and medium for integrated management of API services based on a network data warehouse. Background Art

[0002] With the rapid development of the big data era, data has become the cornerstone of enterprise growth. To meet the challenges of growing structured and unstructured data, network data warehouses (NDWs) play a key role in data processing and storage, becoming a vital engine driving enterprise digital transformation and upgrades.

[0003] However, existing API (Application Programming Interface) service management methods based on network data warehouses mostly use the method of automatically generating API service interfaces. This method is difficult to flexibly configure according to actual needs, resulting in poor flexibility in API interface configuration. In addition, the service is also highly dependent on third-party platforms such as Git (an open source distributed version control system) and Openshift (Red Hat's cloud development platform). Once these tools or technologies fail or have problems, the normal operation of the entire data center architecture may be affected. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the existing technology and provide an integrated management method, device and medium for API services based on a network data warehouse, so as to solve the problems of poor flexibility in API interface configuration and excessive dependence on third-party platforms in the existing API service management method based on a network data warehouse.

[0005] In a first aspect, the present invention provides an integrated management method for API services based on a network data warehouse, the method comprising:

[0006] According to at least one data source selected and configured by the user based on the first visual interface, data in the data source is collected through corresponding extraction, transformation and loading (ETL) tools, pre-processed, and then saved in a preset database;

[0007] Update the API service catalog accordingly based on management operations performed by the user on the API service catalog in the second visual interface, wherein the management operations include at least one of the following: adding, deleting, and modifying operations;

[0008] receiving an API service custom instruction triggered by a user based on the API service catalog, and popping up a third visual interface according to the API service custom instruction;

[0009] service configuration information and execution logic configuration information of the target API service set by the user through the wizard-style configuration in the third visual interface, wherein the execution logic configuration information includes pre-processed data specified by the user as a data source of the target API service;

[0010] Based on the service configuration information and execution logic configuration information of the target API service, the target API service is published and launched using a combination of lightweight Jetty service and Apache Camel rule engine.

[0011] Receive a user's subscription request for the target API service, perform a security audit on the subscription request, and authorize the user to call the target API service after the security audit passes.

[0012] Furthermore, according to at least one data source selected and configured by the user based on the first visual interface, the data in the data source is collected by using a corresponding extraction, transformation, and loading (ETL) tool, and is pre-processed and then saved in a preset database, specifically including:

[0013] receiving at least one data source and a corresponding data source type selected and configured by a user through a first visual interface;

[0014] receiving an ETL tool selected by a user and corresponding to the data source type;

[0015] For each of the data sources, obtain the rule information of the ETL tool configured by the user, as well as the set collection operation process and operation scheduling strategy;

[0016] Collecting data from corresponding data sources based on the rule information of the ETL tool, the collection operation process and the operation scheduling strategy; and

[0017] The collected data is preprocessed according to the business preprocessing rules configured by the user, and the preprocessed data is saved in the preset database.

[0018] Furthermore, the method further comprises:

[0019] Receive a detailed query instruction triggered by a user for an API service in the API service catalog;

[0020] The detailed information of the API service is displayed according to the detailed query instruction, and the detailed information includes the service routing address, input parameter information, version number, request parameter information, response parameter information, response parameter example information, and return status code information.

[0021] Furthermore, the service configuration information includes basic service configuration information and service parameter configuration information. The basic service configuration information includes the service name, service type, whether authentication is performed, service routing address, version number, and service description. The service parameter configuration information includes Headers information, Params information, Body information, request parameter information, response parameter information, response parameter example information, and return status code information.

[0022] The execution logic configuration information also includes structured query language SQL logic information and parameter condition configuration information.

[0023] Furthermore, after publishing and launching the target API service based on the service configuration information and execution logic configuration information of the target API service using a combination of a lightweight Jetty service and an Apache Camel rule engine, the method further includes:

[0024] Use the lightweight Jetty service combined with the Apache Camel rule engine to take the target API service offline.

[0025] Add the target API service to the API service catalog.

[0026] Furthermore, the authorized user calls the target API service, specifically including:

[0027] Determine the validity period of the authorization for the user to call the target API service and the calling system;

[0028] The authorized user calls the target API service through the calling system within the authorization validity period.

[0029] Furthermore, the method further comprises:

[0030] Comprehensively monitor and analyze the calls to all API services subscribed by users in the API service catalog, where the analysis includes at least one of the following:

[0031] Analyze the number of calls to each API service within a preset time period;

[0032] Analyze the top N API services in terms of the number of calls among all API services;

[0033] Analyze the top N calling systems in terms of the number of calls to each API service, where N ≥ 1.

[0034] In a second aspect, the present invention provides an integrated management device for API services based on a network data warehouse, the device comprising:

[0035] A data collection and preprocessing module is used to collect data from at least one data source selected and configured by the user based on the first visual interface through corresponding extraction, transformation and loading (ETL) tools, perform preprocessing on the data source, and save the data into a preset database;

[0036] An API service catalog management module, connected to the data acquisition and preprocessing module, is used to update the API service catalog according to the management operations performed by the user on the API service catalog in the second visual interface, wherein the management operations include at least one of the following: adding, deleting, and modifying operations;

[0037] a custom instruction receiving module, connected to the API service catalog management module, configured to receive an API service custom instruction triggered by a user based on the API service catalog, and pop up a third visual interface according to the API service custom instruction;

[0038] a custom configuration acquisition module, connected to the custom instruction receiving module, for obtaining service configuration information and execution logic configuration information of the target API service set by the user through the wizard-style configuration in the third visual interface, wherein the execution logic configuration information includes pre-processed data specified by the user as a data source of the target API service;

[0039] A service publishing and launch module, connected to the custom configuration acquisition module, is used to publish and launch the target API service based on the service configuration information and execution logic configuration information of the target API service by combining the lightweight Jetty service with the Apache Camel rule engine;

[0040] The subscription and audit module is connected to the service publishing and online module, and is used to receive the user's subscription request for the target API service, perform a security audit on the subscription request, and authorize the user to call the target API service after the security audit passes.

[0041] In the third aspect, the present invention provides an integrated management device for API services based on a network data warehouse, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the integrated management method for API services based on a network data warehouse as described in the first aspect above.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for integrated management of API services based on a network data warehouse as described in the first aspect above is implemented.

[0043] The present invention provides an integrated management method, device, and medium for API services based on a network data warehouse. First, based on at least one data source selected and configured by a user based on a first visual interface, data in the data source is collected through corresponding extraction, transformation, and loading (ETL) tools, pre-processed, and saved in a preset database. Then, based on the management operations performed by the user on the API service catalog in a second visual interface, the API service catalog is updated accordingly, wherein the management operations include at least one of the following: adding, deleting, and modifying operations. Simultaneously, an API service custom instruction triggered by the user based on the API service catalog is received, and a third visual interface is popped up according to the API service custom instruction. Then, based on the service configuration information and execution logic configuration information of the target API service set by the user through the wizard-style configuration in the third visual interface, the execution logic configuration information includes pre-processed data specified by the user as the target API service data source. Based on the service configuration information and execution logic configuration information of the target API service, the target API service is published and put online using a lightweight Jetty service combined with an Apache Camel rule engine. Finally, a subscription request for the target API service is received from the user, a security audit is performed on the subscription request, and after the security audit passes, the user is authorized to call the target API service. The present invention provides users with a wizard-style visual interface, allowing users to easily customize API services and service catalogs. At the same time, by integrating the lightweight Jetty service with the Apache Camel rule engine, dynamic publishing and online release of API services are achieved, significantly improving the flexibility of API interface definition, while reducing dependence on third-party platforms. In addition, the present invention also introduces a security audit mechanism and subscription authorization function to ensure that only audited users can call the target API service, thereby standardizing the use and management of API services while improving security. These improvements not only improve the configuration flexibility of API services, but also accelerate the development, release, deployment and sharing efficiency of API services through simplified operating procedures and automated processing, while ensuring the security and standardization of services, and solving the problems of poor API interface configuration flexibility and over-reliance on third-party platforms in existing network data warehouse-based API service management methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flowchart of an integrated management method for API services based on a network data warehouse according to Example 1 of the present invention;

[0045] Figure 2 This is an architecture diagram of an integrated API service management platform based on a network data warehouse according to an embodiment of the present invention;

[0046] Figure 3Flowchart of data collection and preprocessing according to an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of basic directory operations for API service directory management according to an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of basic operations customized by the API according to an embodiment of the present invention;

[0049] Figure 6 A schematic diagram of the basic operations of API service auditing according to an embodiment of the present invention;

[0050] Figure 7 This is a structural diagram of an API service integrated management device based on a network data warehouse according to Example 2 of the present invention;

[0051] Figure 8 This is a structural diagram of an API service integrated management device based on a network data warehouse according to Example 3 of the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0053] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.

[0054] It is understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments may be combined with each other.

[0055] It can be understood that, for the convenience of description, the drawings of the present invention only show parts related to the present invention, while parts unrelated to the present invention are not shown in the drawings.

[0056] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0057] It can be understood that the terms "first", "second", etc. in the embodiments of the present invention are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0058] It will be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.

[0059] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to various embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented using a hardware-based system that implements the specified functions, or may be implemented using a combination of hardware and computer instructions.

[0060] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.

[0061] Example 1:

[0062] This embodiment provides an integrated management method for API services based on network data warehouse, such as Figure 1 As shown, the method includes:

[0063] Step S101: According to at least one data source selected and configured by the user based on the first visual interface, data in the data source is collected through a corresponding ETL (Extract-Transform-Load) tool, pre-processed, and then saved in a preset database.

[0064] In this embodiment, to help users easily manage, integrate, and optimize various data sources, a first visual interface is provided for users to select and configure data sources, as well as to collect and preprocess data. Through this visual interface, users can intuitively select the desired data source and complete data source connection, data collection, and preprocessing tasks through simple configuration operations.

[0065] Optionally, according to at least one data source selected and configured by the user based on the first visual interface, collecting data from the data source by using a corresponding extraction, transformation, and loading (ETL) tool, pre-processing the data, and then saving the data into a preset database may specifically include:

[0066] receiving at least one data source and a corresponding data source type selected and configured by a user through a first visual interface;

[0067] receiving an ETL tool selected by a user and corresponding to the data source type;

[0068] For each of the data sources, obtain the rule information of the ETL tool configured by the user, as well as the set collection operation process and operation scheduling strategy;

[0069] Collecting data from corresponding data sources based on the rule information of the ETL tool, the collection operation process and the operation scheduling strategy; and

[0070] The collected data is preprocessed according to the business preprocessing rules configured by the user, and the preprocessed data is saved in the preset database.

[0071] In this embodiment, the data sources selected and configured cover a variety of data source types such as SFTP (SSH File Transfer Protocol, secure file transfer protocol) files, databases (such as Mysql), Kafka, API interfaces, etc. After selecting the data source type, the user needs to select the corresponding ETL tool, such as Sqoop, Kettle, Kafka, etc., and configure the rule information of the ETL tool according to the specific data source. Then, the user needs to set up the collection job process and job scheduling strategy to ensure that the data can be collected according to the predetermined rules and schedule. After completing the above steps, the user needs to further configure the business preprocessing rules, which will guide the subsequent data preprocessing operations. Finally, the preprocessed data will be stored in the specified database for subsequent analysis and application.

[0072] The service pre-processing rules may include any of the following:

[0073] 1. Missing value processing, such as filling null values ​​with default values;

[0074] 2. Filter outliers, such as filtering out values ​​that do not conform to business logic;

[0075] 3. Format unification, such as date format conversion (such as MM / DD / YYYY → YYYY-MM-DD)

[0076] 4. Unit conversion, such as weight unit conversion to kilograms (500g→0.5kg);

[0077] 5. Masking of sensitive information, such as partially masking the email username (user@example.com → u***@example.com);

[0078] 6. Business rule verification, such as the order amount must be greater than the shipping fee (amount>=shipping_fee);

[0079] 7. Pre-aggregation calculations, such as counting the total number of daily orders by region (written to an aggregate table for reporting).

[0080] Step S102: updating the API service catalog accordingly according to the management operation performed by the user on the API service catalog in the second visual interface, where the management operation includes at least one of the following: adding, deleting, and modifying operations.

[0081] In this embodiment, the second visual interface provides users with flexible service catalog management capabilities. This interface allows users to customize the service catalog and easily add, delete, and modify it. Newly added services are displayed in a tree-like structure, which not only clearly reflects the hierarchical relationships of the service catalog but also facilitates intuitive management and browsing, thereby improving operational convenience and efficiency.

[0082] Optionally, the method further includes:

[0083] Receive a detailed query instruction triggered by a user for an API service in the API service catalog;

[0084] The detailed information of the API service is displayed according to the detailed query instruction, and the detailed information includes the service routing address, input parameter information, version number, request parameter information, response parameter information, response parameter example information, and return status code information.

[0085] In this embodiment, when a user clicks on an API service in the API service directory, the interface will display detailed information about the API service, including the service routing address, input parameter information, version number, request parameter information, response parameter information, response parameter example information, return status code information, etc., so that the user can fully understand the specific details of the API service. At the same time, a search box function is also provided in the second visual interface, allowing users to quickly search for API service information bound to the corresponding API service directory by keyword.

[0086] Among them, the service routing address is the entry point for calling the API service; the input parameter information indicates which input parameters the API service requires, as well as the type, format, and whether these parameters are required; the version number is used to identify different versions of the API service; the request parameter information includes the request method and request header information; the response parameter information clarifies the data structure and fields returned by the API service; the response parameter example information is used to provide specific response examples to help users better understand the API's return data, and supports the standard JSON (JavaScript Object Notation) format; the return status code information is important feedback on the API service call result. The status code can be used to determine whether the request is successful and the reason for failure.

[0087] Step S103: receiving an API service custom instruction triggered by a user based on the API service catalog, and popping up a third visual interface according to the API service custom instruction.

[0088] In this embodiment, users can trigger custom instructions through operations such as "Create Service" and "Edit Service" in the API service catalog. The third visual interface uses a wizard-style configuration process to guide users step by step to complete the customization process of the API service.

[0089] Step S104: according to the service configuration information and execution logic configuration information of the target API service set by the user through the wizard-style configuration in the third visual interface, the execution logic configuration information includes the pre-processed data specified by the user as the target API service data source.

[0090] In this embodiment, the third visual interface uses a wizard-style configuration method to guide users to flexibly select data sources according to actual needs and easily set REST (Representational State Transfer) style API service information. Specifically, users can complete the following configurations through the wizard-style configuration interface:

[0091] (1) Service configuration information: including basic service configuration information and service parameter configuration information. The basic service configuration information includes the service name, service type, authentication, service routing address, version number, and service description; the service parameter configuration information includes header information, params information, body information, request parameter information, response parameter information, response parameter example information, and return status code information;

[0092] (2) Execution logic configuration information: In addition to the pre-processed data specified by the user as the target API service data source, it also includes SQL (Structured Query Language) logic information and parameter condition configuration information.

[0093] Through this wizard-style configuration method, users can intuitively complete the detailed configuration of API services to ensure that API services can meet specific business needs.

[0094] Step S105: Based on the service configuration information and execution logic configuration information of the target API service, the target API service is published and put online using a combination of lightweight Jetty service and Apache Camel rule engine.

[0095] In this embodiment, in order to realize the automation and efficient API interface service management process, a lightweight Jetty service is combined with the Apache Camel rule engine to publish and launch the target API service. This combination not only improves the flexibility and scalability of the platform, but also ensures the efficient and stable operation of the API service. Specifically, Jetty is a lightweight Web container that acts as an embedded server and provides HTTP (HyperText Transfer Protocol) endpoints for APIs, supporting high concurrency and fast response. Apache Camel is a lightweight rule engine that can be used to define routing rules. Through XML (eXtensible Markup Language) configuration, Apache Camel can flexibly define the processing flow of API services, including data conversion, routing selection, and exception handling.

[0096] Optionally, after publishing and launching the target API service based on the service configuration information and execution logic configuration information of the target API service by combining a lightweight Jetty service with an Apache Camel rule engine, the method further includes:

[0097] Use the lightweight Jetty service combined with the Apache Camel rule engine to take the target API service offline.

[0098] Add the target API service to the API service catalog.

[0099] In this example, Jetty, a lightweight HTTP server, receives external HTTP requests and dispatches them to Camel routes. Apache Camel, an integration framework, implements message routing, transformation, and processing using the Enterprise Integration Pattern (EIP). Spring XML configuration can be used to define Camel contexts, routes, and beans, achieving component decoupling. Dynamic management of service publishing, rollout, and decommissioning can be achieved by sending specific POST requests (e.g., with operation=publish / serviceId=xxx) to the management interface.

[0100] (1) Publishing services

[0101] Send a POST request to the management interface:

[0102] POST / api / management

[0103] Headers:

[0104] -operation:publish

[0105] -serviceId:getDataService

[0106] (2) Online service

[0107] Send a POST request to the management interface:

[0108] POST / api / management

[0109] Headers:

[0110] -operation:online

[0111] -serviceId:getDataService

[0112] (3) Offline service

[0113] Send a POST request to the management interface:

[0114] POST / api / management

[0115] Headers:

[0116] -operation:offline

[0117] -serviceId:getDataService

[0118] In this embodiment, after the target API service goes online, the target API service is added to the API service directory.

[0119] Step S106: receiving a user's subscription request for the target API service, performing a security audit on the subscription request, and authorizing the user to call the target API service after the security audit passes.

[0120] In this embodiment, users can subscribe to the target API service using the Subscribe button in the second visual interface. Clicking this button triggers the subscription request, and the subscribed target API service enters a pending review state. This process design is designed to ensure the security and compliance of API services while providing a convenient user experience.

[0121] In this embodiment, by performing security review and authorization on the user's subscription request, it can be ensured that only reviewed users can call the target API service.

[0122] Optionally, the authorized user calling the target API service specifically includes:

[0123] Determine the validity period of the authorization for the user to call the target API service and the calling system;

[0124] The authorized user calls the target API service through the calling system within the authorization validity period.

[0125] In this embodiment, when subscribing, users are required to fill in the applicant, contact information, reason for application, and the system they wish to call. During the review process, an authorization validity period is set to ensure that the authorization is valid for a reasonable period of time. Once the API service has passed the review, the user can initiate calls to the subscribed target API service by calling the system within the authorization validity period. The API service call process triggers the service call security management mechanism (including authentication, flow control, and concurrency control) and call logging.

[0126] Optionally, the method further includes:

[0127] Comprehensively monitor and analyze the calls to all API services subscribed by users in the API service catalog, where the analysis includes at least one of the following:

[0128] Analyze the number of calls to each API service within a preset time period;

[0129] Analyze the top N API services in terms of the number of calls among all API services;

[0130] Analyze the top N calling systems in terms of the number of calls to each API service, where N ≥ 1.

[0131] In this example, by comprehensively monitoring and analyzing the calls to all subscribed API services in the API service catalog, operators are provided with a powerful tool to quickly identify potential issues, thereby significantly improving the maintainability of the platform. This feature not only improves the stability of the platform's operations but also optimizes the user experience, laying a solid foundation for the continuous optimization of API services.

[0132] Specifically, by deeply analyzing the number of calls to each API service within a preset time period (such as one day), analyzing the top N API services with the most calls among all API services, and analyzing the top N calling systems with the most calls corresponding to each API service, etc., it can help operations personnel quickly locate hidden dangers or abnormal trends that may affect service stability, performance or business continuity. For example, ① performance bottlenecks, such as abnormal traffic fluctuations: a sudden increase in the number of daily calls; ② business anomalies, such as unauthorized calls to the system calling the API, which may mean security vulnerabilities or internal abuse.

[0133] It should be noted that the network data warehouse-based API service integrated management method provided by the present invention realizes the dynamic design, release, deployment and offline functions of REST-style API services by providing on-demand capability encapsulation, visual interface, dynamic release, dynamic deployment service, dynamic offline service, self-service subscription service and instant call capability. It greatly reduces the dependence on third-party platforms and improves the flexibility and scalability of the system. Through the rapid response of Jetty services and the flexible configuration of the Apache Camel rule engine, users can more efficiently manage the life cycle of API services, reduce the complexity of system maintenance, and provide users with more efficient, secure and standardized API services. It provides a new solution for the optimization and upgrade of API services.

[0134] In a specific embodiment, in order to further improve data value operation, data sharing efficiency and production operation efficiency, an API service integrated management platform based on a network data warehouse is provided to implement an API service integrated management method based on a network data warehouse. The platform is committed to solving problems such as insufficient accuracy of API data cataloging, poor flexibility in API interface configuration, and over-reliance on third-party platforms. The platform adopts open source technology and standardized interfaces, reduces dependence on specific technologies, reduces maintenance costs, and improves the scalability and maintainability of the system. At the same time, the platform also designs a wizard-style visual configuration interface to achieve high accuracy of data cataloging and flexibility of API interface configuration. Through this platform, users can easily share and apply data capabilities, and provide data resources to developers, callers and managers in a standard, secure and reusable manner. This not only optimizes the configuration of data resources and improves utilization efficiency, but also brings users a smoother and more convenient development experience. At the same time, the platform also provides a wealth of data analysis and visualization tools to help users better understand and utilize data, and achieve data-driven decision-making and optimization. The architecture diagram of the API service integrated management platform based on the network data warehouse is as follows: Figure 2 As shown, it includes the following modules:

[0135] (1) Data acquisition and preprocessing module: responsible for data collection and cleaning, as well as storing processed data.

[0136] (2) API service catalog module: allows users to customize the service catalog, centrally categorize and present customized REST-style API service information, and facilitates rapid retrieval and self-service subscription by upper-level applications.

[0137] (3) API customization module: customize the data API model according to business needs (users can customize the REST-style data API model), output the processed data through the API interface, and publish it to the API service directory.

[0138] (4) API service audit module: performs security audits and authorizations on users’ self-service subscription capabilities to ensure the security and reliability of the service.

[0139] (5) API service call module: According to business needs, after the service review is passed, the user side can initiate a call to the subscribed service. The service call process will trigger the call security control mechanism and call log record.

[0140] (6) API service monitoring module: records service calls, analyzes and monitors service status, helps operators quickly locate problems, and thus improves the maintainability of the platform.

[0141] Through the collaborative work of these modules, the platform significantly improves the accuracy of data cataloging, flexibly configures API interfaces, and reduces reliance on third-party platforms. This not only significantly increases the efficiency of API development, deployment, and sharing, shortens delivery cycles, but also effectively reduces production and operating costs.

[0142] Based on the API service integrated management platform, the API service integrated management method based on the network data warehouse can include the following steps:

[0143] Step 1: Data collection and preprocessing

[0144] The flow chart of data collection and preprocessing is as follows Figure 3As shown, first, through the visual interface of the data collection and preprocessing module (i.e., the first visual interface), users can select and configure the required data source. These data sources include SFTP files, databases (such as MySQL), Kafka, and API interfaces (data collected and stored through API interfaces can then serve as data sources for custom API services). After selecting the data source type, users need to select the corresponding ETL (Extract-Transform-Load) tool (such as Sqoop, Kettle, Kafka, etc.) and configure the ETL tool's rules based on the specific data source. Next, users need to set up the collection workflow and job scheduling strategy to ensure that data is collected according to the predefined rules and schedule. After completing these steps, users need to further configure business preprocessing rules, which will guide subsequent data processing operations. Finally, the data collection and preprocessing module will perform data cleaning, transformation, and other processing according to the configured business rules to ensure data quality and consistency. The processed data will be stored in the designated database for subsequent analysis and application use. The entire process is designed to provide a flexible and efficient data collection and pre-processing solution to help users easily manage, integrate and optimize various data sources, providing strong support for data-driven business decisions.

[0145] Example: Collect CSV (Comma-Separated Values) files from SFTP to MySQL and preprocess them

[0146] 1) Select the data source type

[0147] Operation: Select the "SFTP file" type in the visual interface.

[0148] Input parameters:

[0149] SFTP server address: sftp.example.com:22

[0150] Username / Password: sftp_user / Password123

[0151] File path: / home / sftp_user / data / user_*.csv

[0152] (File content example: id, name, email, phone, register_time)

[0153] 2) Configure ETL script configuration

[0154] Action: Configure rules:

[0155] File format: CSV (delimiter is comma, skip the first row header).

[0156] Target database: MySQL table target_db.user_clean.

[0157] Field mapping: SFTP field → database field:

[0158] id→user_id,

[0159] register_time→registration_date.

[0160] 3) Set up the collection process

[0161] Action: Define the job name as sftp_to_mysql_user_sync.

[0162] Scheduling strategy:

[0163] Trigger method: File monitoring (detecting new or updated files in the SFTP path).

[0164] Frequency: Scan every 5 minutes, support breakpoint resuming (record processed file names).

[0165] 4) Configure business pre-processing rules

[0166] Action: Set rules in the preprocessing module:

[0167] Data cleaning:

[0168] Empty value processing: filter records with empty email addresses.

[0169] Format standardization:

[0170] The phone field is standardized to the format of +86-138****1234.

[0171] Convert register_time to MySQL DATETIME type.

[0172] Data Validation:

[0173] Verify that the email matches the regular expression ^[a-z0-9._%+-]+@[a-z0-9.-]+\.[az]{2,}$.

[0174] Desensitization treatment:

[0175] Replace the middle 4 digits of the phone field with ****.

[0176] Step 2: API service catalog management

[0177] The basic directory operation diagram of API service directory management is as follows Figure 4 As shown, through the visual interface of the API service catalog management module (i.e., the second visual interface), the service catalog can be easily added, deleted, and modified. The newly added service catalog will be displayed in a tree structure, which is convenient for users to manage and browse intuitively. In addition, the interface also provides a search box function, and users can quickly retrieve the API service information bound to the corresponding service catalog through keywords. When a user clicks on an API service, the interface will display detailed information about the API service, including the service routing address, input parameter information, version number, request parameter information, response parameter information, response parameter example information, return status code information, etc., so that users can fully understand the specific details of the service. In the upper right corner of the visual interface, users can find the subscription operation button. After clicking the button, users can subscribe to the service, and the subscribed service will enter the pending review status. This process design is designed to ensure the security and compliance of API services, while providing users with a convenient operation experience.

[0178] Among them, the service catalog can be classified by business, such as broadband quality, business delivery, transmission quality, etc.

[0179] Step 3: API Customization

[0180] The basic operation diagram of API customization is as follows Figure 5 As shown, through the API customization module's visual interface (i.e., the third visual interface), service designers can flexibly select data sources based on actual needs (using data preprocessed by the data collection and preprocessing module as the data source for the API interface). Through a wizard-based configuration, service designers can easily set up REST-style API service information, including basic information such as the service name, service type, authentication, service routing address, version number, and service description. Parameter information is also a key configuration focus, covering different types of parameters such as headers, params, and body information. Parameter names, types, and descriptions can be specified, and request and response examples are provided (all in standard JSON format). Status codes and response parameters are also important parts of configuration. Regarding execution logic, users can specify data sources and execute SQL to meet specific data processing requirements. The module also provides a rich list of operations, including service launch, launch, delist, delete, modify, and query. The service launch, launch, and delisting processes utilize a lightweight Jetty service combined with the Apache Camel rules engine. The detailed process is as follows:

[0181] 1) Environment Preparation: Use Maven to introduce necessary dependencies, including the Camel core library, Camel-Jetty components (for HTTP services), Spring context support, and XML configuration support packages.

[0182] 2) Service publishing configuration: Create the camel-context.xml main configuration file: define the Camel context container, configure Jetty's HTTP listening port (8080), declare two core beans: ServiceRouter: handles business routing, and ServiceManager: manages service status; set the service management interface route: listen to the / api / management endpoint and receive management instructions through the HTTP POST method.

[0183] 3) Route definition and implementation: Create a ServiceRouter routing class. GET data service routing: listen for GET requests to / api / data / entities; check service status (via a memory map) and throw an exception if the service is unavailable; POST data service routing: listen for POST requests to the same path, implement an independent status check mechanism, and use a thread-safe ConcurrentHashMap to record the status of each service.

[0184] 4) Service Manager Implementation: The ServiceManager class provides the manageService method to handle management requests and supports three types of operations: publish to register a new service, online to bring a specified service online, and offline to take a specified service offline. The operation results will update the service status mapping table in real time.

[0185] 5) Application startup: The main startup class DataServiceApplication executes the following steps: Loading the Spring context (reading XML configuration), initializing the Jetty server and starting it, and keeping the main thread running

[0186] 6) Service management operations: Manage service status and publish services through HTTP requests: method: POST / api / management; Header parameters: operation:publish, serviceId:[service ID]; online service: operation:online; offline service: operation:offline.

[0187] Step 4: API Service Audit

[0188] The basic operation diagram of API service audit is as follows Figure 6As shown, the API Service Audit Module performs security audits and authorizations on users' self-service subscriptions in the API Service Catalog module. This includes details such as pending review information, a list of reviewed information, and authorization validity periods. This module conducts a detailed security audit of user-submitted subscription requests and authorizes them accordingly based on the audit results. During the audit process, the module records pending review information for subsequent tracking and querying. A detailed list of reviewed information is also recorded for users and administrators to review and manage. Furthermore, the module sets an authorization validity period to ensure that authorizations remain valid for a reasonable period of time, thus ensuring platform security and stability.

[0189] Among them, when users subscribe, they need to fill in the applicant, contact information, reason for application, and need to choose to call the system (for subsequent authentication and statistical analysis). After the user subscribes and is reviewed, they will get the following Figure 6 The displayed information includes the order number, service name, application time, applicant, contact information, calling system, return status code, authorization validity period, application reason, and approval information. The return status code indicates the review status (approved / failed) and is used to mark the availability of the service. Approval information refers to the review decision and related instructions for the user's application to subscribe to the API service, and is used to record key aspects of the review process, such as the approver and approval basis.

[0190] Step 5: API service call

[0191] After the API service is reviewed and approved, the user can initiate calls to the subscribed services. During the API service call process, the service call security management mechanism (including authentication, flow control, concurrency control) and call log records will be triggered.

[0192] Step 6: API Service Monitoring

[0193] The API Service Monitoring module comprehensively monitors and analyzes the call status of approved API services. By deeply analyzing data such as the daily number of service calls, the top N service calls, and the top N application systems that call the service, this module provides operators with a powerful tool to quickly identify potential issues, greatly improving the platform's maintainability. This feature not only improves platform stability but also optimizes the user experience, laying a solid foundation for the continuous optimization of API services.

[0194] Specifically, analysis and statistics are conducted on all approved API services (similar to a service operation monitoring view) to help operations personnel quickly locate potential problems. Potential problems refer to hidden dangers or abnormal trends that may affect service stability, performance or business continuity. For example, ① performance bottlenecks, such as abnormal traffic fluctuations: a sudden increase in the number of daily calls; ② business anomalies, such as unauthorized calls to the system to call APIs, which may indicate security vulnerabilities or internal abuse.

[0195] It's important to note that the integrated API service management platform, based on a network data warehouse, provides a wizard-style visual configuration interface, allowing users to easily define API interface services and customize the data service catalog. Furthermore, the platform deeply integrates the lightweight Apache Camel rule engine with the Jetty web container to enable dynamic publishing, deployment, and full lifecycle management of data API services, significantly improving the flexibility of interface definitions and the accuracy of data cataloging. Furthermore, the platform's built-in core functional modules reduce reliance on third-party platforms (such as Git and OpenShift), significantly accelerating the development, publishing, deployment, and sharing of API interfaces.

[0196] An embodiment of the present invention provides an integrated management method for API services based on a network data warehouse. First, based on at least one data source selected and configured by a user based on a first visual interface, data in the data source is collected through corresponding extraction, transformation, and loading (ETL) tools, pre-processed, and saved to a preset database. Then, based on management operations performed by the user on the API service catalog in a second visual interface, the API service catalog is updated accordingly. The management operations include at least one of the following: adding, deleting, and modifying operations. At the same time, an API service custom instruction triggered by the user based on the API service catalog is received, and a third visual interface is popped up according to the API service custom instruction. Then, based on the service configuration information and execution logic configuration information of the target API service set by the user through a wizard-style configuration in the third visual interface, the execution logic configuration information includes pre-processed data specified by the user as the target API service data source. Based on the service configuration information and execution logic configuration information of the target API service, a lightweight Jetty service and an Apache Camel rule engine are combined to publish and launch the target API service. Finally, a subscription request for the target API service is received from the user, a security audit is performed on the subscription request, and after the security audit passes, the user is authorized to call the target API service. The present invention provides users with a wizard-style visual interface, allowing users to easily customize API services and service catalogs. At the same time, by integrating the lightweight Jetty service with the Apache Camel rule engine, dynamic publishing and online release of API services are achieved, significantly improving the flexibility of API interface definition, while reducing dependence on third-party platforms. In addition, the present invention also introduces a security audit mechanism and subscription authorization function to ensure that only audited users can call the target API service, thereby standardizing the use and management of API services while improving security. These improvements not only improve the configuration flexibility of API services, but also accelerate the development, release, deployment and sharing efficiency of API services through simplified operating procedures and automated processing, while ensuring the security and standardization of services, and solving the problems of poor API interface configuration flexibility and over-reliance on third-party platforms in existing network data warehouse-based API service management methods.

[0197] Example 2:

[0198] like Figure 7 As shown, this embodiment provides an API service integrated management device based on a network data warehouse, which is used to execute the above-mentioned API service integrated management method based on a network data warehouse. The device includes:

[0199] The data collection and preprocessing module 11 is used to collect data from at least one data source selected and configured by the user based on the first visual interface through corresponding extraction, transformation and loading (ETL) tools, preprocess the data, and save it into a preset database;

[0200] An API service catalog management module 12 is connected to the data acquisition and preprocessing module 11 and is used to update the API service catalog according to the management operations performed by the user on the API service catalog in the second visual interface, wherein the management operations include at least one of the following: adding, deleting, and modifying operations;

[0201] A custom instruction receiving module 13 is connected to the API service catalog management module 12 and is used to receive an API service custom instruction triggered by a user based on the API service catalog, and pop up a third visual interface according to the API service custom instruction;

[0202] a custom configuration acquisition module 14, connected to the custom instruction receiving module 13, for receiving service configuration information and execution logic configuration information of the target API service set by the user through the wizard-style configuration in the third visual interface, wherein the execution logic configuration information includes pre-processed data specified by the user as a data source of the target API service;

[0203] A service publishing and launch module 15 is connected to the custom configuration acquisition module 14 and is used to publish and launch the target API service based on the service configuration information and execution logic configuration information of the target API service by using a lightweight Jetty service combined with the Apache Camel rule engine;

[0204] The subscription and review module 16 is connected to the service publishing and online module 15, and is used to receive the user's subscription request for the target API service, perform a security review on the subscription request, and authorize the user to call the target API service after the security review is passed.

[0205] Optionally, the data acquisition and preprocessing module 11 includes:

[0206] A first receiving unit, configured to receive at least one data source and a corresponding data source type selected and configured by a user through a first visual interface;

[0207] A second receiving unit is configured to receive an ETL tool selected by a user and corresponding to the data source type;

[0208] A first acquisition unit is configured to acquire, for each of the data sources, rule information of the ETL tool configured by the user, as well as the set collection operation process and operation scheduling strategy;

[0209] A data collection unit, configured to collect data from a corresponding data source based on the rule information of the ETL tool, the collection operation process, and the operation scheduling strategy;

[0210] The data preprocessing unit is used to preprocess the collected data according to the business preprocessing rules configured by the user, and save the preprocessed data into a preset database.

[0211] Optionally, the device further comprises:

[0212] A details query instruction receiving module is used to receive a details query instruction triggered by a user for an API service in the API service catalog;

[0213] The detailed information display module is used to display the detailed information of the API service according to the detailed query instruction. The detailed information includes the service routing address, input parameter information, version number, request parameter information, response parameter information, response parameter example information, and return status code information.

[0214] Optionally, the service configuration information includes basic service configuration information and service parameter configuration information, wherein the basic service configuration information includes the service name, service type, whether authentication is performed, service routing address, version number, and service description; the service parameter configuration information includes Headers information, Params information, Body information, request parameter information, response parameter information, response parameter example information, and return status code information;

[0215] The execution logic configuration information also includes structured query language SQL logic information and parameter condition configuration information.

[0216] Optionally, the device further comprises:

[0217] A service offline module, which is used to offline the target API service by combining the lightweight Jetty service with the Apache Camel rule engine;

[0218] An API service adding module is used to add the target API service to the API service directory.

[0219] Optionally, the subscription and review module 16 includes:

[0220] A first determining unit is used to determine the authorization validity period and calling system of the user to call the target API service;

[0221] The service authorization unit is used to authorize the user to call the target API service through the calling system within the authorization validity period.

[0222] Optionally, the device further comprises:

[0223] A monitoring and analysis module is used to comprehensively monitor and analyze the call status of all API services subscribed by users in the API service catalog, and the analysis includes at least one of the following:

[0224] Analyze the number of calls to each API service within a preset time period;

[0225] Analyze the top N API services in terms of the number of calls among all API services;

[0226] Analyze the top N calling systems in terms of the number of calls to each API service, where N ≥ 1.

[0227] Example 3:

[0228] refer to Figure 8 This embodiment provides an API service integrated management device based on a network data warehouse, including a memory 21 and a processor 22. The memory 21 stores a computer program, and the processor 22 is configured to run the computer program to execute the API service integrated management method based on a network data warehouse in Example 1.

[0229] The memory 21 is connected to the processor 22 . The memory 21 may be a flash memory, a read-only memory, or other memory. The processor 22 may be a central processing unit or a single-chip microcomputer.

[0230] Example 4:

[0231] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the integrated management method of API services based on a network data warehouse in the above-mentioned embodiment 1 is implemented.

[0232] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0233] In summary, the embodiment of the present invention provides an integrated management method, device and medium for API services based on a network data warehouse. First, according to at least one data source selected and configured by the user based on a first visual interface, the data in the data source is collected through corresponding extraction, conversion and loading ETL tools, and the data is pre-processed and saved in a preset database; then, according to the management operations performed by the user on the API service directory in a second visual interface, the API service directory is updated accordingly, and the management operations include at least one of the following: add, delete and modify operations; at the same time, the API service custom instructions triggered by the user based on the API service directory are received, and a third visual interface is popped up according to the API service custom instructions; then, according to the service configuration information and execution logic configuration information of the target API service set by the user through the wizard-style configuration method in the third visual interface, the execution logic configuration information includes the pre-processed data specified by the user as the target API service data source; and based on the service configuration information and execution logic configuration information of the target API service, a lightweight Jetty service is used to connect to Apache The target API service is published and put online by combining the Camel rule engine; finally, a user's subscription request for the target API service is received, the subscription request is security audited, and after the security audit is passed, the user is authorized to call the target API service. The present invention provides users with a wizard-style visual interface, allowing users to easily customize API services and service catalogs. At the same time, by integrating the lightweight Jetty service with the Apache Camel rule engine, the dynamic publication and online release of API services are realized, which significantly improves the flexibility of API interface definition and reduces dependence on third-party platforms. In addition, the present invention also introduces a security audit mechanism and subscription authorization function to ensure that only audited users can call the target API service, thereby standardizing the use and management of API services while improving security. These improvements not only improve the configuration flexibility of API services, but also accelerate the development, release, deployment and sharing efficiency of API services through simplified operating procedures and automated processing, while ensuring the security and standardization of services, and solving the problems of poor API interface configuration flexibility and excessive dependence on third-party platforms in existing network data warehouse-based API service management methods.

[0234] It will be understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present invention, and the present invention is not limited thereto. Those skilled in the art will appreciate that various modifications and improvements can be made without departing from the spirit and substance of the present invention, and such modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for integrated management of application programming interface (API) services based on a network data warehouse, characterized in that: The method comprises: According to at least one data source selected and configured by the user based on the first visual interface, data in the data source is collected through corresponding extraction, transformation and loading (ETL) tools, pre-processed, and then saved in a preset database; Update the API service catalog accordingly based on management operations performed by the user on the API service catalog in the second visual interface, wherein the management operations include at least one of the following: adding, deleting, and modifying operations; receiving an API service custom instruction triggered by a user based on the API service catalog, and popping up a third visual interface according to the API service custom instruction; service configuration information and execution logic configuration information of the target API service set by the user through the wizard-style configuration in the third visual interface, wherein the execution logic configuration information includes pre-processed data specified by the user as a data source of the target API service; Based on the service configuration information and execution logic configuration information of the target API service, the target API service is published and launched using a combination of lightweight Jetty service and Apache Camel rule engine. Receive a user's subscription request for the target API service, perform a security audit on the subscription request, and authorize the user to call the target API service after the security audit passes.

2. The method according to claim 1, characterized in that The method of collecting data from the at least one data source selected and configured by the user based on the first visual interface through a corresponding extraction, transformation, and loading (ETL) tool, pre-processing the data, and then saving the data into a preset database specifically includes: receiving at least one data source and a corresponding data source type selected and configured by a user through a first visual interface; receiving an ETL tool selected by a user and corresponding to the data source type; For each of the data sources, obtain the rule information of the ETL tool configured by the user, as well as the set collection operation process and operation scheduling strategy; Collecting data from corresponding data sources based on the rule information of the ETL tool, the collection operation process and the operation scheduling strategy; and The collected data is preprocessed according to the business preprocessing rules configured by the user, and the preprocessed data is saved in the preset database.

3. The method according to claim 1, characterized in that The method further comprises: Receive a detailed query instruction triggered by a user for an API service in the API service catalog; The detailed information of the API service is displayed according to the detailed query instruction, and the detailed information includes the service routing address, input parameter information, version number, request parameter information, response parameter information, response parameter example information, and return status code information.

4. The method according to claim 1, wherein The service configuration information includes basic service configuration information and service parameter configuration information, wherein the basic service configuration information includes the service name, service type, whether authentication is performed, service routing address, version number, and service description; the service parameter configuration information includes Headers information, Params information, Body information, request parameter information, response parameter information, response parameter example information, and return status code information; The execution logic configuration information also includes structured query language SQL logic information and parameter condition configuration information.

5. The method according to claim 1, wherein After publishing and launching the target API service based on the service configuration information and execution logic configuration information of the target API service by combining the lightweight Jetty service with the Apache Camel rule engine, the method further includes: Use the lightweight Jetty service combined with the Apache Camel rule engine to take the target API service offline. Add the target API service to the API service catalog.

6. The method according to claim 1, wherein The authorized user calls the target API service, specifically including: Determine the validity period of the authorization for the user to call the target API service and the calling system; The authorized user calls the target API service through the calling system within the authorization validity period.

7. The method according to claim 6, characterized in that The method further comprises: Comprehensively monitor and analyze the calls to all API services subscribed by users in the API service catalog, where the analysis includes at least one of the following: Analyze the number of calls to each API service within a preset time period; Analyze the top N API services in terms of the number of calls among all API services; Analyze the top N calling systems in terms of the number of calls to each API service, where N ≥ 1.

8. An application programming interface API service integrated management device based on a network data warehouse, characterized in that: The device comprises: A data collection and preprocessing module is used to collect data from at least one data source selected and configured by the user based on the first visual interface through corresponding extraction, transformation and loading (ETL) tools, perform preprocessing on the data source, and save the data into a preset database; An API service catalog management module, connected to the data acquisition and preprocessing module, is used to update the API service catalog according to the management operations performed by the user on the API service catalog in the second visual interface, wherein the management operations include at least one of the following: adding, deleting, and modifying operations; a custom instruction receiving module, connected to the API service catalog management module, configured to receive an API service custom instruction triggered by a user based on the API service catalog, and pop up a third visual interface according to the API service custom instruction; a custom configuration acquisition module, connected to the custom instruction receiving module, for obtaining service configuration information and execution logic configuration information of the target API service set by the user through the wizard-style configuration in the third visual interface, wherein the execution logic configuration information includes pre-processed data specified by the user as a data source of the target API service; A service publishing and launch module, connected to the custom configuration acquisition module, is used to publish and launch the target API service based on the service configuration information and execution logic configuration information of the target API service by combining the lightweight Jetty service with the Apache Camel rule engine; The subscription and audit module is connected to the service publishing and online module, and is used to receive the user's subscription request for the target API service, perform a security audit on the subscription request, and authorize the user to call the target API service after the security audit passes.

9. An application programming interface API service integrated management device based on a network data warehouse, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the integrated management method of application programming interface API services based on a network data warehouse as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the integrated management method for application programming interface (API) services based on a network data warehouse according to any one of claims 1 to 7.

Images