Data interface processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the invention provide a data interface processing method and apparatus, an electronic device and a storage medium. The method comprises the steps of receiving data source connection information for a data source; configuring interface input and output parameters according to the data source connection information; generating analog data according to the interface input and output parameters; performing an interface test based on the simulation data; and in response to the fact that the interface test is passed, issuing the interface input and output parameters. Through the embodiment of the invention, the processing efficiency of the data service can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a data interface processing method, a data interface processing device, an electronic device and a storage medium. Background Art

[0002] As a "bridge" between databases and data applications, data services can provide data warehouse data to data users in a service-oriented and interface-oriented manner, shielding many details of underlying data storage and calculation, simplifying and strengthening data usage, and helping users to uniformly manage internal and external API (application programming interface) services. At the same time, data services can also be used to quickly generate APIs from data tables, or to register existing APIs to the data service platform for unified release and management. However, the current data services have the problem that traditional interfaces are mainly developed by server-side personnel, and data research and development does not have the ability to develop interfaces, resulting in long development cycles and low development efficiency. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a data interface processing method, a data interface processing device, an electronic device and a storage medium that overcome the above problems or at least partially solve the above problems.

[0004] In order to solve the above problem, an embodiment of the present invention discloses a data interface processing method, which is applied to an execution engine, wherein the execution engine is connected to at least one data source, and the method comprises:

[0005] receiving data source connection information for the data source;

[0006] Configure interface input and output parameters according to the data source connection information;

[0007] generating simulation data according to the interface input and output parameters;

[0008] Performing interface testing based on the simulation data;

[0009] In response to the interface test passing, the interface input and output parameters are released.

[0010] Optionally, the method further comprises:

[0011] Receive interface call request;

[0012] Verifying the interface call request;

[0013] In response to the verification being passed, querying the data source based on the interface call request to determine the real-time data;

[0014] The real-time data is fed back.

[0015] Optionally, the step of verifying the interface call request includes:

[0016] Performing a preliminary check based on the token and timestamp of the interface call request;

[0017] In response to the preliminary verification being passed, determining whether the current is currently in the upper current limit state;

[0018] In response to not being in the current limiting upper limit state, the interface call request verification passes.

[0019] Optionally, the execution engine is provided with a client cache, a server cache, a public cache and a cold data cache, and the method further comprises:

[0020] Determining a target cache from the client cache, the server cache, the public cache, and the cold data cache according to the priority of the interface call request;

[0021] The interface call request is cached in the target cache.

[0022] Optionally, the method further comprises:

[0023] Data desensitization processing is performed on the real-time data.

[0024] Optionally, the method further comprises:

[0025] The data layer of the data source is encapsulated.

[0026] Optionally, the method further comprises:

[0027] Performing query profile analysis on the data in the data source to determine query profile data;

[0028] Performing cost analysis on the query profile data to determine query cost data;

[0029] The data in the data source is cached and preheated based on a preset preheating rule and the query cost data.

[0030] In a second aspect of the present invention, an embodiment of the present invention discloses a data interface processing device, which is applied to an execution engine, wherein the execution engine is connected to at least one data source, and the device comprises:

[0031] A first receiving module, configured to receive data source connection information for the data source;

[0032] A configuration module, used to configure interface input and output parameters according to the data source connection information;

[0033] A simulation module, used for generating simulation data according to the interface input and output parameters;

[0034] A testing module, used for performing interface testing based on the simulation data;

[0035] A publishing module is used to publish the interface input and output parameters in response to the interface test passing.

[0036] According to a third aspect of the present invention, an embodiment of the present invention discloses an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the steps of the data interface processing method as described above when executed by the processor.

[0037] In a fourth aspect of the present invention, an embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data interface processing method described above are implemented.

[0038] The embodiments of the present invention include the following advantages:

[0039] The embodiment of the present invention receives data source connection information for the data source; configures interface input and output parameters according to the data source connection information; generates simulation data according to the interface input and output parameters; performs interface testing based on the simulation data; and publishes the interface input and output parameters in response to the interface test passing. The execution engine provides unified data services, and the execution engine not only supports traditional databases, but also supports semi-structured and unstructured data, solving the problem that the direct connection to the data source in the traditional mode cannot be federated query, the complex configuration of big data components and the slow query speed of large volumes, and better supports data query applications in heterogeneous multi-source scenarios with many types of databases and widely distributed data in cloud computing scenarios; thereby, R&D personnel can develop interfaces by interacting with the execution engine, shortening the development cycle and improving development efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of steps of an embodiment of a data interface processing method of the present invention;

[0041] Figure 2 is a flowchart of another data interface processing method embodiment of the present invention;

[0042] Figure 3 It is a structural schematic diagram of an application architecture of the present invention;

[0043] Figure 4 It is a hierarchical schematic diagram of an application architecture of the present invention;

[0044] Figure 5 It is a schematic diagram of a multi-level cache of the present invention;

[0045] Figure 6 It is a schematic diagram of service isolation and current limiting and degradation integration of the present invention;

[0046] Figure 7 It is a current limiting degradation schematic diagram of the present invention;

[0047] Figure 8 It is a data desensitization schematic diagram of the present invention;

[0048] Fig. 9 It is a schematic diagram of a unified data query service of the present invention;

[0049] Fig.10 is a schematic diagram of Example 1 of the data interface processing method of the present invention;

[0050] Fig.11 is a schematic diagram of Example 2 of the data interface processing method of the present invention;

[0051] Fig.12 is a schematic diagram of Example 3 of the data interface processing method of the present invention;

[0052] Fig.13 is a schematic diagram of Example 4 of the data interface processing method of the present invention;

[0053] Fig.14 is a structural block diagram of an embodiment of a data interface processing device of the present invention;

[0054] Fig.15 is a structural block diagram of an electronic device provided by an embodiment of the present invention;

[0055] Fig.16 It is a structural block diagram of a storage medium provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Reference Figure 1, showing a step flow chart of an embodiment of a data interface processing method of the present invention, the data interface processing method is applied to an execution engine, and the execution engine is connected to at least one data source. Among them, the execution engine in the embodiment of the present invention can be a Drill execution engine. The Drill execution engine is an open source distributed SQL (Structured Query Language) query engine that can query and analyze various data sources with low latency. It not only supports common relational databases, NoSQL databases, file systems, and cloud storage, but also because Drill uses a Schema-free data model, it can directly query semi-structured and unstructured data, so it can easily process complex data structures and nested data. It also provides powerful query optimization and execution functions, supports multiple data formats, and has a flexible architecture that can easily perform elastic expansion and contraction.

[0058] The data interface processing method may specifically include the following steps:

[0059] Step 101, receiving data source connection information for the data source;

[0060] In an embodiment of the present invention, data source connection information for connecting to a data source may be received.

[0061] Step 102, configuring interface input and output parameters according to the data source connection information;

[0062] Configure the input and output content of the interface according to the various configuration data in the data source connection information configuration, and generate interface input and output parameters.

[0063] Step 103, generating simulation data according to the interface input and output parameters;

[0064] Then, based on the interface input and output parameters, the data required for testing is simulated accordingly, i.e., simulation data.

[0065] Step 104, performing interface testing based on the simulation data;

[0066] Then use simulated data to test the interface to determine whether the current configuration is feasible.

[0067] Step 105: In response to the interface test passing, the interface input and output parameters are released.

[0068] After the interface test passes, the interface input and output parameters can be released in response to the interface test passing for external calls.

[0069] The embodiment of the present invention receives data source connection information for the data source; configures interface input and output parameters according to the data source connection information; generates simulation data according to the interface input and output parameters; performs interface testing based on the simulation data; and publishes the interface input and output parameters in response to the interface test passing. The execution engine provides unified data services, and the execution engine not only supports traditional databases, but also supports semi-structured and unstructured data, solving the problem that the direct connection to the data source in the traditional mode cannot be federated query, the complex configuration of big data components and the slow query speed of large volumes, and better supports data query applications in heterogeneous multi-source scenarios with many types of databases and widely distributed data in cloud computing scenarios; thereby, R&D personnel can develop interfaces by interacting with the execution engine, shortening the development cycle and improving development efficiency.

[0070] Reference Figure 2 , shows a flow chart of steps of another data interface processing method embodiment of the present invention, the data interface processing method is applied to an execution engine, the execution engine is connected to at least one data; the execution engine is provided with a client cache, a server cache, a public cache and a cold data cache. The architecture in the embodiment of the present invention can refer to Figure 3 , which may include the following parts:

[0071] 1.Api-server: API server system, mainly provides data source configuration, API interface development, application permission authorization management and other functions

[0072] 2.Api-gateway: API query gateway, which is mainly responsible for responding to API query requests that have been developed and released, and also provides authentication, time consumption analysis, service degradation, and current limiting functions;

[0073] 3.Api-alert: API alert, mainly responsible for alert notification of abnormal API requests, supports multiple methods such as email and enterprise WeChat;

[0074] 4.Api-web: API front-end system, mainly responsible for providing visual configuration and visual monitoring API chart capabilities

[0075] 5. Middleware: Drill execution engine, database, Redis / MemCache data cache, dataX data synchronization tool, K8S container deployment environment; DataX is an open source data synchronization tool that supports multiple data sources and targets, including relational databases, NoSQL databases, big data platforms, etc., with efficient and reliable data synchronization capabilities. DataX adopts a distributed architecture, which can realize high-concurrency data synchronization tasks and provides rich data cleaning, conversion and filtering functions. It supports flexible configuration and expansion to meet various complex data synchronization requirements. K8S container is a container orchestration and management platform that provides a reliable and highly scalable container cluster management solution for automated application deployment, expansion and management, and provides powerful scheduling, service discovery, load balancing, storage management and other functions.

[0076] Specifically, the architecture layering can refer to Figure 4 , which can include a 4-layer architecture: S101 data source layer, S102 API development layer, S103 API gateway layer and S104 API application layer.

[0077] The S101 data source layer is mainly responsible for the data source configuration related to the Drill execution engine, and uses Prometheus and grafana tools to monitor the status of the engine CPU, memory, threads, query queues, etc. in real time; it is also responsible for managing the Drill udf function and query data cache;

[0078] The S102 API development layer is the core layer of the system. Based on the data source, it uses a variety of implementation methods to dynamically configure the logical data table, as well as the upper-layer API request and return parameters, filter conditions and other information. Finally, after API testing, it is released, and the current limiting rules and cache rules are configured to provide services externally.

[0079] The S103 API gateway layer is mainly responsible for ensuring the security and high availability of services, authenticating the API based on the request information, and filtering illegal requests according to the configured API current limiting rules;

[0080] The S104 API application layer is mainly responsible for binding APIs to applications, realizing many-to-many mapping relationships, facilitating interface reuse, and flexibly controlling permissions. It is also responsible for monitoring the operation of published APIs and providing alarm capabilities.

[0081] The data interface processing method may specifically include the following steps:

[0082] Step 201, receiving data source connection information for the data source;

[0083] When it is necessary to start the interface, matching with the data source can be achieved. Data source connection information can be sent, and the data source connection information includes configuration data of the data source. These data sources can be unstructured data sources and semi-structured data sources, which are not specifically limited in the embodiment of the present invention.

[0084] Step 202, configuring interface input and output parameters according to the data source connection information;

[0085] You can configure the input parameters of the interface, such as data types and operators, according to the configuration contents in the data source connection information. You can also configure the output parameters of the interface, such as desensitization rules, whether to group, and aggregation function type. The configured input parameters and output parameters are the interface input and output parameters.

[0086] Step 203, generating simulation data according to the interface input and output parameters;

[0087] Based on the substantial content of the interface input and output parameters, the corresponding simulation data is generated. The simulation data is used to test the interface.

[0088] Step 204, performing interface testing based on the simulation data;

[0089] Use simulated data to perform various tests on the interface. Determine whether the interface passes the test.

[0090] Step 205: In response to the interface test passing, the interface input and output parameters are released.

[0091] When the interface passes the test, the interface can be used. In response to the interface test passing, the interface input and output parameters can be released. When the interface test fails, the configuration data can be modified and tested again until the modified configuration parameters can make the interface pass the test.

[0092] For example, based on the Drill execution engine, business developers configure data source connection information and register it with the engine to ensure consistency on both sides;

[0093] Based on a single or multiple data sources, configure the logical mapping table of data, such as data sets or data units, through a wizard or by writing SQL scripts. At the same time, based on this view, configure the input parameters (data types, operators) of the interface, and configure the output parameters (massaging rules, whether to group, and aggregation function types) of the interface; the service unit is a unified metadata configuration for the data tables in the underlying data source, and supports the abstraction of associated queries of heterogeneous data sources as a unified view. Based on the service unit, the data set adds field semantic meanings and is defined as a dimension or measurement type, which is mainly used in multidimensional analysis query scenarios.

[0094] Based on the configured parameter information, simulate the corresponding simulation data, simulate the API call process, and test whether it meets the needs of the business scenario;

[0095] After the interface test is passed, the API can be published to the production environment and bound to the corresponding application, and the current limiting strategy, whether to enable caching and other information can be configured for external calls.

[0096] Step 206, receiving an interface call request;

[0097] In the embodiment of the present invention, an interface call request may also be received, and data in a data source may be queried or acquired through the interface call request.

[0098] Step 207, verifying the interface call request;

[0099] After receiving the interface call request, the interface call request can be verified to determine whether the interface call request is a malicious request to prevent the malicious request from affecting the use of the interface.

[0100] Specifically, the step of verifying the interface call request includes: performing a preliminary verification based on the token and timestamp of the interface call request; in response to passing the preliminary verification, determining whether the current state is in the upper limit of the current limit; in response to not being in the upper limit of the current limit state, the interface call request verification passes.

[0101] For the verification process, first, the token and timestamp of the interface call request can be verified. When the token and timestamp of the interface call request meet the requirements, it is determined whether the current interface is in the upper limit state. The upper limit state can be represented by the number of requests currently processed. When the number of processed requests is greater than the target threshold, it means that it is in the upper limit state. The specific size of the target threshold is not limited. When it is in the upper limit state, it can no longer respond. When it is not in the upper limit state, it can still process requests. It can respond to the fact that it is not in the upper limit state. The interface call request passes the verification and can be processed. Interface call request.

[0102] Step 208, determining a target cache from the client cache, the server cache, the public cache, and the cold data cache according to the priority of the interface call request;

[0103] You can refer to Figure 5 , you can build a multi-level cache including client cache, server cache, public cache and cold data cache in the execution engine. You can balance the query performance of data cache for different scenarios. The cache query performance closer to the client level is higher, and the cache query reuse of the lower level is higher.

[0104] The target cache can be determined from the client cache, server cache, public cache and cold data cache according to the priority of the interface call request.

[0105] Step 209, caching the interface call request to the target cache;

[0106] The interface call request is then cached in the target cache, thereby reducing the access pressure on the engine.

[0107] In addition, in order to ensure the stability of the system, multi-level data caching and service isolation are also performed.

[0108] That is, the stability design in the data API, including service isolation and current limiting and degradation. Figure 6 Among them, the goal of service isolation is to hope that different interface services can provide services independently. If a service has availability problems, it will not affect other services. Because the query gateway service itself is built based on microservices, it can support two types of service isolation: horizontal and vertical. Horizontally, it can isolate different businesses. Through intelligent routing, requests for different businesses can be diverted in advance to achieve horizontal isolation; vertically, it is based on the priorities of different services of a single business. Vertical priority isolation. Since the provided interface is stateless, each group of isolation units is completely horizontal and isolated, and will not affect each other. At the same time, the service isolation module can be expanded and reduced independently.

[0109] In order to avoid excessive requests to a single interface, a unified flow limiting and degradation capability is provided, including both client and server. Figure 7 The client can optionally set the total number of requests for a single link and configure whether to enable the bottom-line call strategy. The request passes authentication, and reaches the specific business interface after routing and diversion by the gateway service. The business interface is configured according to the cache policy. If the cache fails, it is downgraded to the underlying metadata for query by default and the cache is updated. At the same time, if the traffic of a single interface exceeds the configured upper limit within a statistical time period, the service is automatically disconnected, waiting for the next time period to be restored and re-counted to avoid service unavailability due to excessive peak traffic and ensure high service availability.

[0110] Step 210, in response to the verification being passed, querying the data source based on the interface call request to determine the real-time data;

[0111] When the interface call request passes the verification, the interface call request can be processed and real-time data matching the interface call request can be queried from the data source.

[0112] Step 211, performing data desensitization processing on the real-time data;

[0113] In order to ensure data security, real-time data can be desensitized to ensure data security. Desensitization can include hash desensitization and shielding desensitization.

[0114] For example, see Figure 8 First, you need to configure the matching regular expression. Currently, the platform has provided common regular expressions (name, email address, mobile phone number, ID card, etc.). If there are special application scenarios, users can flexibly configure the matching regular expression. Then specify the policy type. If you choose the occlusion policy, you need to include the following configuration parameters: front and back / middle occlusion, unknown occlusion start / end, check the occlusion symbol (commonly * or #), and customize the number of occlusion symbols. If you choose the hash policy, you need to select the corresponding hash function (such as SHA256) and the number of hash value bits to be retained. Through the configuration of the above two desensitization strategies, common data desensitization scenarios can basically be covered.

[0115] Step 212, feeding back the real-time data;

[0116] The real-time data can be fed back to the interface caller so that the interface caller can obtain the corresponding real-time data and perform corresponding processing.

[0117] The process of interface calling can be explained with an example:

[0118] The interface caller configures the input parameters, output parameters, and filtering conditions required by the interface, and uses the corresponding token bound to the application to encrypt the request body content;

[0119] After receiving the call request, the API gateway will first use the token and timestamp to verify whether it is a normal request, and then check whether the interface current limit is reached to filter out malicious requests;

[0120] If query cache is enabled on the interface, it will first determine whether there is a cache based on the request information. If so, the data in the cache will be directly returned to reduce the pressure on the engine side.

[0121] If the cache is expired or not enabled, Drill will be requested to execute the query and return the real-time data;

[0122] During the interface call process, the query time and query return results of the interface will be recorded. If there is any abnormal access, an alarm will be issued via SMS, email, etc.

[0123] Step 213, performing query portrait analysis on the data in the data source to determine query portrait data;

[0124] In order to cache data in advance according to the request more proactively and timely, cache preheating can also be performed.

[0125] First, query portrait analysis can be performed on the data in the data source to determine query portrait data. The query portrait data represents the portrait data of the query analysis of the data.

[0126] Step 214, performing cost analysis on the query profile data to determine query cost data;

[0127] Then, a cost analysis can be performed based on the query profile data to determine the resources consumed by the corresponding system when querying the data, which is the query cost data.

[0128] Step 215: cache preheating is performed on the data in the data source based on a preset preheating rule and the query cost data.

[0129] In the embodiment of the present invention, different cache preheating rules can be set according to different usage requirements, and these preheating rules are preset preheating rules. The data that needs cache preheating can be screened and determined based on these preset preheating rules and query cost data, and cache preheating can be performed on the data in the data source.

[0130] It can be seen that at the query analysis level, two types of query analysis are provided according to different abstract levels of data, including raw SQL for different data sources, or query analysis close to the data source layer. At the semantic level, a query portrait analysis is performed on the model indicator dimension; at the cost evaluation level, the query analysis portrait is evaluated based on information such as query time, data volume, and query popularity; at the warm-up task level, different warm-up rules are configured to analyze the warm-up methods for different scenarios based on the evaluated costs. In addition, the cache itself also has cache elimination or data expiration. During the warm-up task stage, outdated data will also be actively eliminated.

[0131] In an optional embodiment of the present invention, the method further includes: encapsulating the data layer of the data source.

[0132] In order to avoid repeated encoding and decoding of the transport layer and conversion of data protocols between different data layers of the data request flow, which brings additional overhead, the data layer of the data source can be encapsulated. Specifically, Apache Arrow (technology) can be used to encapsulate the data layer, hoping to abstract the data format uniformly at different data layers and reduce encoding and decoding operations. At the same time, Apache Arrow's vectorization technology can also be used to improve query computing performance and transmission efficiency.

[0133] In order to make the query service unified, you can refer to Fig. 9, which can unify query language, federated query and offline and online integration. The unified query language uses standard ANSI SQL syntax and Calcite's powerful compatibility to unify query logic and shield the underlying physical differences. Federated query can support federated computing between different engines, reduce data synchronization costs, and improve data query efficiency. Offline and online integration: Offline and online execution modes are adapted to different query scenarios.

[0134] At the data request level, the query expressions of different engines and different layers are not unified, and they need to be repeatedly converted and developed for different layers and scenarios. Therefore, we want to provide a unified data abstraction that can express query requests of different layers and data sources, as well as the corresponding query plans. It shields query details at each level, unifies the expression of query requests, and supports the issuance of queries by physical execution plans, reducing the cost of query plan development.

[0135] In addition, since the indicator data set needs to better support multi-dimensional analysis scenarios, the indicator standardization process is divided into three steps:

[0136] 1. Define the business scope of the indicator: including name, description information and other business attribute data. This part of the responsibility is usually completed by the data product manager

[0137] 2. Define the calculation method of the indicator: that is, the calculation method of the specific data table field. This part of the responsibility is usually completed by the data developer.

[0138] 3. Define the data set: Plan the data set service scenario and add the indicator dimensions needed for the scenario to the data set. This part is usually completed by data product managers or business personnel.

[0139] At the same time, data modeling includes visual modeling and automated modeling. The data model can be abstracted into three levels, including data source, data table, and semantic data table. Since we do not know the actual business meaning of the data table at the data table level, we need to manually build the relationship between data tables during modeling. At the semantic data table level, we can automatically infer the relationship between data tables based on the dimensions to build the data model.

[0140] Visual modeling technology is mainly aimed at the first-level data table level. The data connection relationship between data tables is manually calibrated through low-code drag-and-drop. At the semantic data level, since we have solidified the dimension relationship of the data at the semantic layer, we can then use some automated means to automatically discover the relationship between dimensions and automatically calibrate the relationship between data tables.

[0141] The embodiment of the present invention introduces Drill as an execution engine, which not only supports traditional databases, but also supports semi-structured and unstructured data, and solves the problem that the direct connection to the data source in the traditional mode cannot be federated query, the complex configuration of big data components and the slow query speed of large volumes, and better supports the data query application in heterogeneous multi-source scenarios with many types of databases and wide data distribution in cloud computing scenarios; when developing interfaces, it provides indicator management and data modeling capabilities, and more conveniently sorts out the business semantic mapping relationship of data, which not only solves the pain point that traditional data services cannot support multi-dimensional analysis scenarios, but also provides BI with more powerful data processing capabilities. At the same time, it also sorts out and precipitates data assets, which can effectively strengthen the reuse of later interfaces, make it more closely integrated with the business, and easier for personnel to understand and apply to the upper layer; add multi-level cache and cache preheating data acceleration query design, and reduce the network and query overhead brought by traditional single-level cache in a flexible and configurable way. At the same time, scheduled data synchronization and data caching and cache preheating for calls can effectively avoid the problem of first cache breakdown and greatly speed up interface access. The stability design of adding intelligent routing, service isolation and current limiting and degradation, after intelligent routing diversion, can better cope with the high availability of services in high-concurrency scenarios of API services through horizontal business isolation and vertical priority isolation, and avoid overall performance problems caused by malicious requests in the traditional mode. Current limiting and degradation can set request thresholds on both the client and the server, which is more conducive to resource management and permission control. It supports the desensitization function of data fields, hash desensitization and obstruction desensitization, and solves the security problems of traditional plaintext output methods through a minimalist and reusable configuration method, better protects data privacy, and covers a wider range of data sharing scenarios.

[0142] In order to make the embodiments of the present invention clear to those skilled in the art, several examples are given below for illustration:

[0143] The first one can refer to Fig.10 ,The implementation process of data source module creation is described as follows:

[0144] 1. Register data source

[0145] 2. Check if there is a data source with the same name

[0146] - If a data source with the same name exists, modify the data name

[0147] - If there is no data source with the same name, go to step 33. Select the data source type

[0148] -Select data source types such as MySQL, ClickHouse, Hive, etc.

[0149] 4. Configure connection information

[0150] -Enter connection information such as host name (host), port number (port), database name (db), user name (user), password (password)

[0151] 5. Test the connection

[0152] - Click the Test Connection button to check if the connection is successful

[0153] - If the connection is successful, go to step 6

[0154] -If the connection fails, return to step 4 and modify the configuration information

[0155] 6. Synchronize data source information to the Drill execution engine

[0156] - Click the Synchronize Data Source button to synchronize the data source information to the Drill execution engine

[0157] The second one can refer to Fig.11 ,The implementation process of the API development module is described as follows:

[0158] 1. API development method

[0159] -Choose the development mode according to your needs: wizard mode, SQL mode, indicator mode.

[0160] 2. Wizard Mode

[0161] - Check if there is a service unit;

[0162] - If there is no service unit, go to step 3;

[0163] - If there is a service unit, go to step 8.

[0164] 3. Create a new service unit

[0165] -Select the associated table: Select the table to be associated from the available table list;

[0166] -Configure association relationships (optional): Set the association relationships between tables as needed;

[0167] -Configure service unit fields: define the fields of the service unit, including name, type, etc.;

[0168] -Preview Data: View the data preview of the service unit.

[0169] 4.API basic information configuration

[0170] -Select service unit: Select the service unit to be associated from the list of created service units;

[0171] -Configure request parameters and return parameters: define the request parameters and return parameters of the API, including name, type, validation rules, etc.

[0172] 5. SQL Mode

[0173] -API basic information configuration: Synchronize step 4.

[0174] -Write SQL according to grammatical rules: Use SQL grammar to write query statements;

[0175] -Parse SQL: parse and verify the correctness of SQL statements;

[0176] -Configure request parameters and return parameters: Synchronize step 4;

[0177] -Verify SQL: Verify SQL statements to ensure syntax correctness and security;

[0178] -Preview data: View the data preview of SQL query results.

[0179] 6. Indicator Mode

[0180] - Check if there is a dataset;

[0181] - If there is no dataset, go to step 7;

[0182] - If you have a dataset, go to step 8.

[0183] 7. Create a new dataset

[0184] -Select data source: Select the data source you want to use from the available data source list;

[0185] - Confirm dimensions and measures: Define the dimensions and measures of the data set, including selecting fields, setting aggregation methods, etc.;

[0186] - Create composite and derived indicators based on business scenarios: Create composite and derived indicators as needed to meet specific business needs;

[0187] -Preview Data: View the data preview of the dataset.

[0188] 8.API basic information configuration

[0189] -Select dataset: Select the dataset you want to associate from the list of created datasets;

[0190] -Select dimension column: Select the column in the dataset to be used as the dimension;

[0191] -Configure aggregation functions: configure functions for performing aggregation calculations on data sets, such as sum, average, etc.;

[0192] -Configure request parameters and return parameters: Same as step 4.

[0193] 9. Save configuration: Save the configuration information of the API.

[0194] 10. Perform API testing: Use the saved configuration information to test the API and verify the functionality and correctness of the API.

[0195] The third one can refer to Fig.12 ,The implementation process of the API market module is described as follows:

[0196] 1. Internal API testing

[0197] - Use internal tools or test frameworks to test APIs and verify their functionality and correctness.

[0198] 2. External API Registration

[0199] - Check if the external API has been registered;

[0200] - If no external API is registered, go to step 3;

[0201] - If you have already registered the external API, go to step 6.

[0202] 3. Create an application

[0203] -Create a new application to register the external API.

[0204] 4. Generate APP key pair

[0205] -Generate a unique APP key pair for the application for authentication and authorization.

[0206] 5. Binding API

[0207] -Bind external APIs to applications to ensure that only authorized applications can access the APIs.

[0208] 6. Export API

[0209] -Export relevant information of the API, including the API name, URL, request parameters, return parameters, etc.

[0210] 7. Authorize business application personnel

[0211] -Authorize API access rights to specific business application personnel or teams.

[0212] 8. Calling API

[0213] -Use the authorized application key pair, call the API through the API URL and request parameters to obtain the return result.

[0214] 9. API release completed.

[0215] The fourth one can refer to Fig.13 , the implementation process of the API gateway module is described as follows:

[0216] 1. External API call

[0217] -Call external APIs for data interaction or function calls as needed.

[0218] 2. Encapsulate the request body content such as request parameters and return fields required to call the API

[0219] -Encapsulate request body content such as request parameters and return fields according to API requirements.

[0220] 3. MD5 encryption of the request body content

[0221] - Perform MD5 encryption on the request body to ensure data integrity and security.

[0222] 4. Use the APP key pair for signature authentication

[0223] - Use the APP key pair to sign and authenticate the request to ensure the legitimacy and identity of the request.

[0224] 5. Perform request body time verification

[0225] -Verify the time in the request body to ensure the timeliness of the request and prevent replay attacks.

[0226] 6. Limit the flow of APP

[0227] -Limit API calls according to the APP's current limiting strategy to prevent excessive requests from putting pressure on the system.

[0228] 7. Perform API current limiting

[0229] -Limit API calls according to the API's current limiting strategy to prevent excessive requests from putting pressure on the system.

[0230] 8. Send Request

[0231] -Send the encapsulated request body to the API server.

[0232] 9. Get the returned results

[0233] -Receive the results returned by the API server, including data, status code and other information.

[0234] 10. Record API call behavior

[0235] -Record API call behavior, including request time, request parameters, return results and other information for subsequent analysis and monitoring.

[0236] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0237] Reference Fig.14 , shows a structural block diagram of an embodiment of a data interface processing device of the present invention, the data interface processing device is applied to an execution engine, the execution engine is connected to at least one data source, and the data interface processing device may specifically include the following modules:

[0238] The first receiving module 1401 is used to receive data source connection information for the data source;

[0239] Configuration module 1402, configured to configure interface input and output parameters according to the data source connection information;

[0240] A simulation module 1403, used to generate simulation data according to the interface input and output parameters;

[0241] A testing module 1404, configured to perform an interface test based on the simulation data;

[0242] The publishing module 1405 is used to publish the interface input and output parameters in response to the interface test passing.

[0243] In an optional embodiment of the present invention, the device further comprises:

[0244] A second receiving module, used for receiving an interface call request;

[0245] A verification module, used for verifying the interface call request;

[0246] A query module, configured to query the data source based on the interface call request in response to the verification being passed, and determine the real-time data;

[0247] The feedback module is used to feed back the real-time data.

[0248] In an optional embodiment of the present invention, the verification module includes:

[0249] A verification submodule, used for performing preliminary verification based on the token and timestamp of the interface call request;

[0250] A first response submodule, for determining whether the current is in the upper limit state in response to the preliminary verification being passed;

[0251] The second response submodule is used to respond to the interface call request verification passing in response to not being in the current limiting upper limit state.

[0252] In an optional embodiment of the present invention, the execution engine is provided with a client cache, a server cache, a public cache and a cold data cache, and the device further comprises:

[0253] A cache determination module, configured to determine a target cache from the client cache, the server cache, the public cache, and the cold data cache according to the priority of the interface call request;

[0254] A cache module is used to cache the interface call request to the target cache.

[0255] In an optional embodiment of the present invention, the device further comprises:

[0256] The desensitizing module is used to perform data desensitization processing on the real-time data.

[0257] In an optional embodiment of the present invention, the device further comprises:

[0258] The encapsulation module is used to encapsulate the data layer of the data source.

[0259] In an optional embodiment of the present invention, the device further comprises:

[0260] A portrait analysis module, used to perform query portrait analysis on the data in the data source to determine query portrait data;

[0261] A cost analysis module, used to perform cost analysis on the query profile data to determine query cost data;

[0262] The cache preheating module is used to perform cache preheating on the data in the data source based on a preset preheating rule and the query cost data.

[0263] The embodiment of the present invention introduces Drill as an execution engine, which not only supports traditional databases, but also supports semi-structured and unstructured data, and solves the problem that the direct connection to the data source in the traditional mode cannot be federated query, the complex configuration of big data components and the slow query speed of large volumes, and better supports the data query application in heterogeneous multi-source scenarios with many types of databases and wide data distribution in cloud computing scenarios; when developing interfaces, it provides indicator management and data modeling capabilities, and more conveniently sorts out the business semantic mapping relationship of data, which not only solves the pain point that traditional data services cannot support multi-dimensional analysis scenarios, but also provides BI with more powerful data processing capabilities. At the same time, it also sorts out and precipitates data assets, which can effectively strengthen the reuse of later interfaces, make it more closely integrated with the business, and easier for personnel to understand and apply to the upper layer; add multi-level cache and cache preheating data acceleration query design, and reduce the network and query overhead brought by traditional single-level cache in a flexible and configurable way. At the same time, scheduled data synchronization and data caching and cache preheating for calls can effectively avoid the problem of first cache breakdown and greatly speed up interface access. The stability design of adding intelligent routing, service isolation and current limiting and degradation, after intelligent routing diversion, can better cope with the high availability of services in high-concurrency scenarios of API services through horizontal business isolation and vertical priority isolation, and avoid overall performance problems caused by malicious requests in the traditional mode. Current limiting and degradation can set request thresholds on both the client and the server, which is more conducive to resource management and permission control. It supports the desensitization function of data fields, hash desensitization and obstruction desensitization, and solves the security problems of traditional plaintext output methods through a minimalist and reusable configuration method, better protects data privacy, and covers a wider range of data sharing scenarios.

[0264] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0265] Reference Fig.15 , an embodiment of the present invention further provides an electronic device, including:

[0266] A processor 1501 and a storage medium 1502, wherein the storage medium 1502 stores a computer program executable by the processor 1501, and when the electronic device is running, the processor 1501 executes the computer program to perform the data interface processing method as described in any one of the embodiments of the present invention.

[0267] The data interface processing method is applied to an execution engine, the execution engine is connected to at least one data source, and the data interface processing method includes:

[0268] receiving data source connection information for the data source;

[0269] Configure interface input and output parameters according to the data source connection information;

[0270] generating simulation data according to the interface input and output parameters;

[0271] Performing interface testing based on the simulation data;

[0272] In response to the interface test passing, the interface input and output parameters are released.

[0273] Optionally, the method further comprises:

[0274] Receive interface call request;

[0275] Verifying the interface call request;

[0276] In response to the verification being passed, querying the data source based on the interface call request to determine the real-time data;

[0277] The real-time data is fed back.

[0278] Optionally, the step of verifying the interface call request includes:

[0279] Performing a preliminary check based on the token and timestamp of the interface call request;

[0280] In response to the preliminary verification being passed, determining whether the current is currently in the upper current limit state;

[0281] In response to not being in the current limiting upper limit state, the interface call request verification passes.

[0282] Optionally, the execution engine is provided with a client cache, a server cache, a public cache and a cold data cache, and the method further comprises:

[0283] Determining a target cache from the client cache, the server cache, the public cache, and the cold data cache according to the priority of the interface call request;

[0284] The interface call request is cached in the target cache.

[0285] Optionally, the method further comprises:

[0286] Data desensitization processing is performed on the real-time data.

[0287] Optionally, the method further comprises:

[0288] The data layer of the data source is encapsulated.

[0289] Optionally, the method further comprises:

[0290] Performing query profile analysis on the data in the data source to determine query profile data;

[0291] Performing cost analysis on the query profile data to determine query cost data;

[0292] The data in the data source is cached and preheated based on a preset preheating rule and the query cost data.

[0293] The embodiment of the present invention introduces Drill as an execution engine, which not only supports traditional databases, but also supports semi-structured and unstructured data, and solves the problem that the direct connection to the data source in the traditional mode cannot be federated query, the complex configuration of big data components and the slow query speed of large volumes, and better supports the data query application in heterogeneous multi-source scenarios with many types of databases and wide data distribution in cloud computing scenarios; when developing interfaces, it provides indicator management and data modeling capabilities, and more conveniently sorts out the business semantic mapping relationship of data, which not only solves the pain point that traditional data services cannot support multi-dimensional analysis scenarios, but also provides BI with more powerful data processing capabilities. At the same time, it also sorts out and precipitates data assets, which can effectively strengthen the reuse of later interfaces, make it more closely integrated with the business, and easier for personnel to understand and apply to the upper layer; add multi-level cache and cache preheating data acceleration query design, and reduce the network and query overhead brought by traditional single-level cache in a flexible and configurable way. At the same time, scheduled data synchronization and data caching and cache preheating for calls can effectively avoid the problem of first cache breakdown and greatly speed up interface access. The stability design of adding intelligent routing, service isolation and current limiting and degradation, after intelligent routing diversion, can better cope with the high availability of services in high-concurrency scenarios of API services through horizontal business isolation and vertical priority isolation, and avoid overall performance problems caused by malicious requests in the traditional mode. Current limiting and degradation can set request thresholds on both the client and the server, which is more conducive to resource management and permission control. It supports the desensitization function of data fields, hash desensitization and obstruction desensitization, and solves the security problems of traditional plaintext output methods through a minimalist and reusable configuration method, better protects data privacy, and covers a wider range of data sharing scenarios.

[0294] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the processor.

[0295] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0296] Reference Fig.16 The embodiment of the present invention further provides a computer-readable storage medium 1601, on which a computer program is stored. When the computer program is executed by a processor, the data interface processing method as described in any one of the embodiments of the present invention is executed.

[0297] The data interface processing method is applied to an execution engine, the execution engine is connected to at least one data source, and the data interface processing method includes:

[0298] receiving data source connection information for the data source;

[0299] Configure interface input and output parameters according to the data source connection information;

[0300] generating simulation data according to the interface input and output parameters;

[0301] Performing interface testing based on the simulation data;

[0302] In response to the interface test passing, the interface input and output parameters are released.

[0303] Optionally, the method further comprises:

[0304] Receive interface call request;

[0305] Verifying the interface call request;

[0306] In response to the verification being passed, querying the data source based on the interface call request to determine the real-time data;

[0307] The real-time data is fed back.

[0308] Optionally, the step of verifying the interface call request includes:

[0309] Performing a preliminary check based on the token and timestamp of the interface call request;

[0310] In response to the preliminary verification being passed, determining whether the current is currently in the upper current limit state;

[0311] In response to not being in the current limiting upper limit state, the interface call request verification passes.

[0312] Optionally, the execution engine is provided with a client cache, a server cache, a public cache and a cold data cache, and the method further comprises:

[0313] Determining a target cache from the client cache, the server cache, the public cache, and the cold data cache according to the priority of the interface call request;

[0314] The interface call request is cached in the target cache.

[0315] Optionally, the method further comprises:

[0316] Data desensitization processing is performed on the real-time data.

[0317] Optionally, the method further comprises:

[0318] The data layer of the data source is encapsulated.

[0319] Optionally, the method further comprises:

[0320] Performing query profile analysis on the data in the data source to determine query profile data;

[0321] Performing cost analysis on the query profile data to determine query cost data;

[0322] The data in the data source is cached and preheated based on a preset preheating rule and the query cost data.

[0323] The embodiment of the present invention introduces Drill as an execution engine, which not only supports traditional databases, but also supports semi-structured and unstructured data, and solves the problem that the direct connection to the data source in the traditional mode cannot be federated query, the complex configuration of big data components and the slow query speed of large volumes, and better supports the data query application in heterogeneous multi-source scenarios with many types of databases and wide data distribution in cloud computing scenarios; when developing interfaces, it provides indicator management and data modeling capabilities, and more conveniently sorts out the business semantic mapping relationship of data, which not only solves the pain point that traditional data services cannot support multi-dimensional analysis scenarios, but also provides BI with more powerful data processing capabilities. At the same time, it also sorts out and precipitates data assets, which can effectively strengthen the reuse of later interfaces, make it more closely integrated with the business, and easier for personnel to understand and apply to the upper layer; add multi-level cache and cache preheating data acceleration query design, and reduce the network and query overhead brought by traditional single-level cache in a flexible and configurable way. At the same time, scheduled data synchronization and data caching and cache preheating for calls can effectively avoid the problem of first cache breakdown and greatly speed up interface access. The stability design of adding intelligent routing, service isolation and current limiting and degradation, after intelligent routing diversion, can better cope with the high availability of services in high-concurrency scenarios of API services through horizontal business isolation and vertical priority isolation, and avoid overall performance problems caused by malicious requests in the traditional mode. Current limiting and degradation can set request thresholds on both the client and the server, which is more conducive to resource management and permission control. It supports the desensitization function of data fields, hash desensitization and obstruction desensitization, and solves the security problems of traditional plaintext output methods through a minimalist and reusable configuration method, better protects data privacy, and covers a wider range of data sharing scenarios.

[0324] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0325] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0326] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0327] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0328] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0329] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0330] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0331] The data interface processing method, the data interface processing device, the electronic device and the storage medium provided by the present invention are introduced in detail above. The principle and implementation mode of the present invention are explained by using specific examples in this article. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for the general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and the scope of application. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data interface processing method, characterized in that: Applied to an execution engine, the execution engine is connected to at least one data source, the method comprising: receiving data source connection information for the data source; Configure interface input and output parameters according to the data source connection information; generating simulation data according to the interface input and output parameters; Performing interface testing based on the simulation data; In response to the interface test passing, the interface input and output parameters are released.

2. The method according to claim 1, characterized in that The method further comprises: Receive interface call request; Verifying the interface call request; In response to the verification being passed, querying the data source based on the interface call request to determine the real-time data; The real-time data is fed back.

3. The method according to claim 2, characterized in that The step of verifying the interface call request includes: Performing a preliminary check based on the token and timestamp of the interface call request; In response to the preliminary verification being passed, determining whether the current is currently in the upper current limit state; In response to not being in the current limiting upper limit state, the interface call request verification passes.

4. The method according to claim 2, characterized in that: The execution engine is provided with a client cache, a server cache, a public cache and a cold data cache, and the method further comprises: Determining a target cache from the client cache, the server cache, the public cache, and the cold data cache according to the priority of the interface call request; The interface call request is cached in the target cache.

5. The method according to claim 2, characterized in that: The method further comprises: Data desensitization processing is performed on the real-time data.

6. The method according to claim 1, characterized in that The method further comprises: The data layer of the data source is encapsulated.

7. The method according to claim 1, characterized in that The method further comprises: Performing query profile analysis on the data in the data source to determine query profile data; Performing cost analysis on the query profile data to determine query cost data; The data in the data source is cached and preheated based on a preset preheating rule and the query cost data.

8. A data interface processing device, characterized in that: Applied to an execution engine, the execution engine is connected to at least one data source, and the device comprises: A first receiving module, configured to receive data source connection information for the data source; A configuration module, used to configure interface input and output parameters according to the data source connection information; A simulation module, used for generating simulation data according to the interface input and output parameters; A testing module, used for performing interface testing based on the simulation data; A publishing module is used to publish the interface input and output parameters in response to the interface test passing.

9. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the steps of the data interface processing method as claimed in any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data interface processing method according to any one of claims 1 to 7 are implemented.

Images