Development method, device and equipment for data warehouse data acquisition interface and medium

Abstract

Embodiments of the present application specifically provide a development method, device, equipment and medium for a data warehouse data retrieval interface. The development method for the data warehouse data retrieval interface comprises: in response to a configuration operation of a developer on each parameter configuration item in a data retrieval interface configuration interface, obtaining parameter values of each parameter; generating a data warehouse data retrieval interface according to a correspondence between a data retrieval module and the parameters and the parameter values of each parameter; the correspondence is generated by disassembling a data warehouse data retrieval interface code according to an SQL syntax rule; and the data warehouse data retrieval interface is used for calling by a calling party after being published. The method of the present application generates a data warehouse data retrieval interface through a visual configuration operation on the data warehouse data retrieval interface, simplifies cumbersome operations of data warehouse data retrieval interface development, and improves the efficiency of data warehouse data retrieval interface development.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a development method, apparatus, device, and medium for a data warehouse data retrieval interface. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this application set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] In data query scenarios targeting data warehouses, in order to facilitate the caller to retrieve data from the data warehouse, it is usually necessary to develop a data warehouse retrieval interface for retrieving data from the data warehouse.

[0004] Under current technology, data warehouse data retrieval interfaces are typically developed by manually writing code.

[0005] However, because developing data warehouse data retrieval interfaces in this way is quite cumbersome, the efficiency of developing data warehouse data retrieval interfaces is low. Summary of the Invention

[0006] Because the development efficiency of data warehouse data retrieval interfaces is low under current technology, there is a great need for an improved data warehouse data retrieval interface development method to improve the efficiency of data warehouse data retrieval interface development.

[0007] In this context, embodiments of this application aim to provide a method, apparatus, device, and medium for developing data retrieval interfaces for data warehouses.

[0008] On one hand, this application provides a development method for a data warehouse data retrieval interface, applied to an interface development device, including:

[0009] In response to the developer's configuration operations for each parameter configuration item in the data retrieval interface configuration interface, obtain the parameter values ​​of each parameter;

[0010] Based on the correspondence between the data retrieval module and the parameters, as well as the parameter values, a data warehouse data retrieval interface is generated. The correspondence is generated by disassembling the data warehouse data retrieval interface code according to SQL syntax rules. The data warehouse data retrieval interface is used for callers after it is published.

[0011] In one implementation, a data warehouse data retrieval interface is generated based on the correspondence between the data retrieval module and the parameters, as well as the parameter values ​​of each parameter, including:

[0012] Based on the parameter values ​​of the corresponding parameters of the data retrieval module, generate the structured object corresponding to the data retrieval module;

[0013] Generate a data warehouse data retrieval interface based on structured objects.

[0014] In one implementation, the structured object includes at least one of the following objects:

[0015] Result set objects, table objects, conditional filter objects, aggregation objects, pagination objects, and sorting objects;

[0016] The parameters corresponding to the result set object include a list of returned result parameters; the parameters corresponding to the database table object include at least one of the following: database, data table, and database schema; the parameters corresponding to the condition filter object include at least one of the following: query condition input parameters, class attribute parameters, and calculation parameters; the parameters corresponding to the aggregation object include aggregation parameters; the parameters corresponding to the pagination object include at least one of the following: limit query parameters and offset query parameters; the parameters corresponding to the sorting object include at least one of the following: sorting type parameters and aggregate sorting parameters.

[0017] In one embodiment, the method further includes:

[0018] In the API runtime environment, publish the data warehouse data retrieval API.

[0019] In one implementation, the interface operating environment includes a sandbox environment and an online environment;

[0020] In the API runtime environment, publish the data warehouse data retrieval API, including:

[0021] In a sandbox environment, publish the data warehouse data retrieval interface;

[0022] If the data warehouse data retrieval interface test is successful, then publish the data warehouse data retrieval interface in the online environment.

[0023] In one embodiment, the interface development device includes a data gateway;

[0024] The data gateway is used to receive data retrieval requests sent by other devices, which include the domain name of the data retrieval interface. Based on the environment parameters in the data retrieval request, it selects between a sandbox environment and a production environment to call the data warehouse data retrieval interface in the selected interface operating environment.

[0025] The domain name for the data retrieval interface is obtained after the data gateway registers an instance.

[0026] In one embodiment, the interface development device also includes an SQL adapter;

[0027] The SQL adapter is used to convert structured objects in the data warehouse data retrieval interface into SQL statements based on the input parameters in the data retrieval requests sent by other devices.

[0028] In one embodiment, the interface development device also includes a driver routing table;

[0029] The driver routing table contains the drivers corresponding to each database.

[0030] The driver is used to establish a data source connection between the interface development device and the database corresponding to the data retrieval request;

[0031] The data source connection is used to enable the data warehouse data retrieval interface to retrieve data from the corresponding database.

[0032] In one embodiment, the interface development device further includes a storage management plugin library; the storage management plugin library stores at least one plugin containing a driver.

[0033] The plugin's storage path is generated based on the corresponding database type name and data source version number.

[0034] In one implementation, the driver corresponding to each database is loaded by a class loader; there is a one-to-one correspondence between each driver and each type of loader; the class loader is generated by inheriting the class loader base class and overriding the loadClass method.

[0035] In one embodiment, the interface development device further includes HikariCP;

[0036] HikariCP is used to manage data source connections established for each database.

[0037] In one embodiment, the interface development equipment further includes a database management center;

[0038] The database management center is used for registering and managing various databases.

[0039] In one embodiment, the method further includes:

[0040] Listen for configuration update information of the data warehouse data retrieval interface; the configuration update information refers to the parameter updates in the data warehouse data retrieval interface.

[0041] In response to the existence of configuration update information, the data warehouse data retrieval interface is updated based on the configuration update information.

[0042] In one implementation, updating the data warehouse data retrieval interface based on configuration update information includes:

[0043] If the storage root path for configuration update information is determined to represent the sandbox environment, then the data warehouse data retrieval interface in the sandbox environment is updated.

[0044] If the storage root path for configuration update information is determined to represent the online environment, then the data warehouse data retrieval interface in the online environment will be updated.

[0045] On one hand, this application provides a development apparatus for a data warehouse data retrieval interface, applied to interface development equipment, including:

[0046] The acquisition unit is used to respond to the developer's configuration operations on each parameter configuration item in the data retrieval interface configuration interface and to obtain the parameter value of each parameter.

[0047] The generation unit is used to generate a data warehouse data retrieval interface based on the correspondence between the data retrieval module and the parameters, as well as the parameter values ​​of each parameter. The correspondence is generated after the data warehouse data retrieval interface code is decomposed according to SQL syntax rules. The data warehouse data retrieval interface is used for callers to call after it is published.

[0048] In one implementation, the generation unit is used for:

[0049] Based on the parameter values ​​of the corresponding parameters of the data retrieval module, generate the structured object corresponding to the data retrieval module;

[0050] Generate a data warehouse data retrieval interface based on structured objects.

[0051] In one implementation, the structured object includes at least one of the following objects:

[0052] Result set objects, table objects, conditional filter objects, aggregation objects, pagination objects, and sorting objects;

[0053] The parameters corresponding to the result set object include a list of returned result parameters; the parameters corresponding to the database table object include at least one of the following: database, data table, and database schema; the parameters corresponding to the condition filter object include at least one of the following: query condition input parameters, class attribute parameters, and calculation parameters; the parameters corresponding to the aggregation object include aggregation parameters; the parameters corresponding to the pagination object include at least one of the following: limit query parameters and offset query parameters; the parameters corresponding to the sorting object include at least one of the following: sorting type parameters and aggregate sorting parameters.

[0054] In one embodiment, the generating unit is further configured to:

[0055] In the API runtime environment, publish the data warehouse data retrieval API.

[0056] In one implementation, the interface operating environment includes a sandbox environment and an online environment;

[0057] The generation unit is also used for:

[0058] In a sandbox environment, publish the data warehouse data retrieval interface;

[0059] If the data warehouse data retrieval interface test is successful, then publish the data warehouse data retrieval interface in the online environment.

[0060] In one embodiment, the interface development device includes a data gateway;

[0061] The data gateway is used to receive data retrieval requests sent by other devices, which include the domain name of the data retrieval interface. Based on the environment parameters in the data retrieval request, it selects between a sandbox environment and a production environment to call the data warehouse data retrieval interface in the selected interface operating environment.

[0062] The domain name for the data retrieval interface is obtained after the data gateway registers an instance.

[0063] In one embodiment, the interface development device also includes an SQL adapter;

[0064] The SQL adapter is used to convert structured objects in the data warehouse data retrieval interface into SQL statements based on the input parameters in the data retrieval requests sent by other devices.

[0065] In one embodiment, the interface development device also includes a driver routing table;

[0066] The driver routing table contains the drivers corresponding to each database.

[0067] The driver is used to establish a data source connection between the interface development device and the database corresponding to the data retrieval request;

[0068] The data source connection is used to enable the data warehouse data retrieval interface to retrieve data from the corresponding database.

[0069] In one embodiment, the interface development device further includes a storage management plugin library; the storage management plugin library stores at least one plugin containing a driver.

[0070] The plugin's storage path is generated based on the corresponding database type name and data source version number.

[0071] In one implementation, the driver corresponding to each database is loaded by a class loader; there is a one-to-one correspondence between each driver and each type of loader; the class loader is generated by inheriting the class loader base class and overriding the loadClass method.

[0072] In one embodiment, the interface development device further includes HikariCP;

[0073] HikariCP is used to manage data source connections established for each database.

[0074] In one embodiment, the interface development equipment further includes a database management center;

[0075] The database management center is used for registering and managing various databases.

[0076] In one embodiment, the generating unit is further configured to:

[0077] Listen for configuration update information of the data warehouse data retrieval interface; the configuration update information refers to the parameter updates in the data warehouse data retrieval interface.

[0078] In response to the existence of configuration update information, the data warehouse data retrieval interface is updated based on the configuration update information.

[0079] In one embodiment, the generating unit is further configured to:

[0080] If the storage root path for configuration update information is determined to represent the sandbox environment, then the data warehouse data retrieval interface in the sandbox environment is updated.

[0081] If the storage root path for configuration update information is determined to represent the online environment, then the data warehouse data retrieval interface in the online environment will be updated.

[0082] On one hand, this application provides an electronic device, including:

[0083] Processor; and

[0084] The memory stores computer instructions that cause the processor to execute steps of the methods provided in the various optional implementations of any of the above-described methods for developing a data warehouse data retrieval interface.

[0085] On the one hand, this application provides a storage medium storing computer instructions for causing a computer to execute the steps of the methods provided in the various optional implementations of any of the above-described methods for developing a data warehouse data retrieval interface.

[0086] The development method for a data warehouse data retrieval interface in this application includes: responding to the developer's configuration operations for each parameter configuration item in the data retrieval interface configuration interface, obtaining the parameter values ​​of each parameter; generating the data warehouse data retrieval interface based on the correspondence between the data retrieval module and the parameters, and the parameter values ​​of each parameter; the correspondence is generated after decomposing the data warehouse data retrieval interface code according to SQL syntax rules; the data warehouse data retrieval interface is used for invocation by callers after deployment. This method, through visual configuration operations for the data warehouse data retrieval interface, generates the data warehouse data retrieval interface, simplifying the cumbersome operations of data warehouse data retrieval interface development and improving the efficiency of data warehouse data retrieval interface development. Attached Figure Description

[0087] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which:

[0088] Figure 1 The diagram illustrates an example of a data warehouse data retrieval interface development and deployment process according to an embodiment of this application.

[0089] Figure 2 A flowchart illustrating a development method for a data warehouse data retrieval interface according to an embodiment of this application is shown schematically.

[0090] Figure 3 An example diagram of a data retrieval interface configuration interface in an embodiment of this application is shown schematically;

[0091] Figure 4 An example diagram illustrating a data warehouse data retrieval interface code from an embodiment of this application is shown schematically.

[0092] Figure 5 An example diagram of a structured object in an embodiment of this application is shown schematically;

[0093] Figure 6 The diagram schematically illustrates an example of a method for constructing a library table object according to an embodiment of this application.

[0094] Figure 7 An example diagram of a JSON string from an embodiment of this application is shown schematically;

[0095] Figure 8 The diagram illustrates an example of data warehouse data retrieval interface development in an embodiment of this application.

[0096] Figure 9 A flowchart illustrating a method for publishing a data warehouse data retrieval interface is shown in an embodiment of this application.

[0097] Figure 10 The diagram illustrates an example architecture of a data warehouse data retrieval interface publishing system according to an embodiment of this application.

[0098] Figure 11 A flowchart illustrating a method for updating a data warehouse data retrieval interface is shown in an embodiment of this application.

[0099] Figure 12 The diagram illustrates an example of a database or data warehouse query process according to an embodiment of this application.

[0100] Figure 13The diagram illustrates an example architecture of a data warehouse data retrieval interface development system according to an embodiment of this application.

[0101] Figure 14 The diagram schematically illustrates an example structure of a storage medium according to an embodiment of this application.

[0102] Figure 15 The diagram schematically illustrates a structural block diagram of a development apparatus for a data warehouse data retrieval interface according to an embodiment of this application.

[0103] Figure 16 The diagram schematically illustrates an example structure of an electronic device according to an embodiment of this application.

[0104] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0105] The principles and spirit of this application will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement this application, and are not intended to limit the scope of this application in any way. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0106] Those skilled in the art will understand that the embodiments of this application can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this application can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0107] According to the embodiments of this application, a development method, apparatus, device and medium for data warehouse data retrieval interfaces are proposed.

[0108] In this article, it is important to understand that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0109] First, some of the terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0110] Terminal devices can be mobile terminals, fixed terminals, or portable terminals, such as mobile phones, sites, units, devices, multimedia computers, multimedia tablets, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system devices, personal navigation devices, personal digital assistants, audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is also foreseeable that terminal devices can support any type of user-facing interface (e.g., wearable devices).

[0111] Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0112] Application Programming Interface (API): A set of definitions and protocols used to build and integrate application software.

[0113] Data warehouse: A data warehouse is a collection of data that supports management decisions. In this embodiment, for ease of explanation, the data warehouse may also be referred to as a data warehouse. The data is subject-oriented, integrated, not easily lost, and time-varying. A data warehouse is a collection of snapshots of all operating environments and external data sources. It does not need to be highly accurate, as it must be extracted from the operating environment on a specific time basis.

[0114] Application Data Store (ADS): This layer stores personalized statistical indicators and report data for data products. It also provides data for data products and data analysis to facilitate subsequent business queries, analysis, and data distribution.

[0115] Ad Hoc queries allow users to flexibly select query criteria based on their needs, and the system can then generate corresponding statistical reports based on the user's selections.

[0116] Siloed development: Each business line is built independently by different development teams. The technology stacks of different business lines are different, and the different development teams do not communicate with each other, which can easily lead to duplicate development.

[0117] Low code is a visual application development approach that delivers applications faster with less code.

[0118] Structured Query Language (SQL) is a programming language used to store and process information in relational databases.

[0119] Continuous Integration (CI) / Continuous Deployment (CD): This refers to continuous integration, continuous delivery, and continuous deployment.

[0120] Maven is a software (especially Java software) project management and automated build tool provided by the Apache Software Foundation (a web server).

[0121] Data source: The database or database server used by the database application.

[0122] Parent delegation model: This is the operating rule for multiple class loaders in Java. If a class loader needs to load a class, it will first delegate the class loading request to its parent class loader. This process is repeated at each level until it is recursively pushed to the top level.

[0123] Data warehouse data retrieval interface: This is an API used to retrieve data from a data warehouse.

[0124] Hardcoding is a software development practice that embeds data directly into the source code of a program or other executable object, unlike data obtained from external sources or generated at runtime. Hardcoded data can typically only be modified by editing the source code and recompiling the executable.

[0125] The principles and spirit of this application will be explained in detail below with reference to several representative embodiments. Invention Overview

[0127] With the development of internet technology, massive data analysis has gradually become an essential part of refined operations and product decision-making. In some data analysis scenarios, users typically need to retrieve data from a data warehouse through a data warehouse retrieval interface. This data warehouse retrieval interface is an API used to retrieve data from a data warehouse.

[0128] However, due to the diversity of data analysis scenarios, different callers typically have different data retrieval needs. To meet these needs, developers usually need to translate these different data retrieval requirements into different data warehouse data retrieval interfaces, and then develop APIs to provide callers with data retrieval services that meet their specific requirements.

[0129] In the traditional approach, R&D personnel typically first analyze the data retrieval requirements, and then, based on the analysis results, manually write database connection code and data retrieval logic code. They then deploy and test the generated data warehouse data retrieval interface in the production environment to complete the delivery.

[0130] However, this approach often results in cumbersome operational steps, low development efficiency, and difficulty in reusing data warehouse retrieval interfaces. This is because data warehouse ADS tables typically use different storage types, and the SQL syntax used during development often differs. Therefore, to ensure the logical correctness and optimal performance of the data warehouse retrieval interface, developers usually need to repeatedly adjust the code and deploy and test the server. The data processing code required in many different scenarios is usually consistent, such as post-processing and calculations on query result sets. However, since data developers in different business lines are often different, data processing-related code is difficult to reuse, leading to redundant development.

[0131] Furthermore, this approach typically suffers from lengthy and costly delivery processes for data warehouse data retrieval interfaces. This is because the standard data warehouse data retrieval interface delivery process requires server-side code development, offline integration testing, and online deployment for each iteration's data retrieval requirements. The development and delivery time for data warehouse data retrieval interfaces often takes several days or even weeks to schedule, which can severely impact business implementation. In addition, the development of data warehouse data retrieval interfaces for different business lines often results in siloed implementation. If the data source a data warehouse data retrieval interface depends on changes, leading to alterations in the data retrieval logic, the server-side data warehouse data retrieval interface code must be modified, and offline integration testing and online deployment must be redone. This data source change leads to a long time window and significantly impacts usage for business users.

[0132] Furthermore, this approach often presents challenges in adapting to multiple data source types and versions. This is because most tables in the ADS layer of the data warehouse are used for ad-hoc queries in business systems. Therefore, different data source types are typically chosen based on different scenarios. For example, operations personnel frequently use large, wide tables (i.e., database tables with many fields) in report analysis scenarios. In this scenario, the query volume is large but the query frequency is low, making a columnar database management system (Clickhouse) data source suitable. On the other hand, some C-end applications require real-time recommendations based on user profile data. In this scenario, data is typically queried by identification (ID) and stored in key-value format, requiring high concurrency and low latency performance. Data sources such as HBase and Redis are commonly chosen. However, this necessitates different drivers to implement connectors for different data source types. Moreover, it is often difficult to be compatible with different versions of drivers for a unified data source. Siloed development leads to inconsistent management of data source connections, resulting in connection abuse.

[0133] Considering that the code for data warehouse data retrieval interfaces can be broken down into multiple data retrieval modules, and a data retrieval interface configuration interface can be generated based on each module, developers can then visually configure the data warehouse data retrieval interfaces through this interface. This addresses the problems of cumbersome operation steps, low development efficiency, and difficulty in reusability in traditional data warehouse data retrieval interface development. Furthermore, a multi-cluster deployment architecture for sandbox and online environments can be built, centrally managing and centralizing the data warehouse data retrieval interfaces. A data gateway can then be used to uniformly manage data retrieval requests for these interfaces, resolving the cross-functionality issues still present in traditional data warehouse data retrieval interface development. This application addresses the issues of lengthy and costly payment processes. Furthermore, it proposes a solution to the problem of adapting to multiple data sources of various types and versions, which is a challenge in traditional data warehouse data retrieval interface development. This solution allows API developers to develop API data retrieval logic through a configuration service in a "configuration-as-development" model. Since the data gateway can process data API requests, developers can use API CI / CD to publish their developed data APIs to the API marketplace for business system calls. The data gateway also provides centralized management of all data APIs. As the unified data retrieval traffic entry point for the data warehouse, the data gateway processes call requests and provides API call services to upstream services. It also features basic functions such as rate limiting, circuit breaking, authentication, and auditing. API request parameters are parsed and transformed by the business logic layer into executable SQL. The appropriate data source connection driver is then obtained through the plugin routing table in the plugin layer, and the SQL query is executed.

[0134] After introducing the basic principles of this application, the various non-limiting embodiments of this application will be described in detail below.

[0135] Application Scenarios Overview

[0136] This application provides a method for developing a data warehouse data retrieval interface, applied to an interface development device. The interface development device can be any suitable type of electronic device, such as a terminal device or a server, which will not be elaborated further in this application.

[0137] See Figure 1 The diagram shown is an example of a data warehouse data retrieval interface development and deployment process provided in an embodiment of this application. The following is a combination of... Figure 1The development process of the data warehouse data retrieval interface is explained.

[0138] Specifically, developers first configure the API to generate a data warehouse data retrieval interface, obtaining a data warehouse data retrieval interface in a pre-release state. Then, they perform internal unit tests on the data warehouse data retrieval interface using the unit test node. After confirming the unit tests pass, they deploy the data warehouse data retrieval interface in the sandbox environment using the sandbox environment node, and test the deployed data warehouse data retrieval interface in the sandbox environment using the test node. After confirming the tests pass, they officially deploy the data warehouse data retrieval interface to the online environment using the online deployment node.

[0139] Furthermore, update information can be configured and the data warehouse data retrieval interface can be updated to obtain a new version of the data warehouse data retrieval interface.

[0140] Furthermore, after the data warehouse data retrieval interface is released, API decommissioning information can be configured to control the decommissioning of the data warehouse data retrieval interface.

[0141] Exemplary methods

[0142] The following is combined with Figure 1 ,refer to Figure 2 This application describes a method for developing a data warehouse data retrieval interface according to exemplary embodiments thereof. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way. Rather, the embodiments of this application can be applied to any applicable scenario.

[0143] See Figure 2 The diagram shows a flowchart of a method for developing a data warehouse data retrieval interface. The implementation method for developing a data warehouse data retrieval interface provided in this embodiment includes the following steps:

[0144] Step 101: In response to the developer's configuration operations for each parameter configuration item in the data retrieval interface configuration interface, obtain the parameter values ​​of each parameter.

[0145] Optionally, configuration operations may include parameter value input operations or parameter value selection operations.

[0146] In one implementation, developers can visually configure the data warehouse data retrieval interface through a data retrieval interface configuration interface to generate the data warehouse data retrieval interface. The data retrieval interface configuration interface displays multiple parameter configuration items. Developers can perform parameter value input operations or parameter value selection operations for different parameter configuration items, thereby inputting the parameter values ​​for each parameter. See also... Figure 3 The image shown is an example of a data retrieval interface configuration interface. The following will demonstrate... Figure 3This document provides an example of visual configuration based on the data retrieval interface. Figure 3 The data retrieval interface configuration interface includes multiple parameters. Users can input the corresponding parameter values ​​to generate the data warehouse data retrieval interface. Specifically, in the data retrieval interface configuration interface, users can enter the database name and table name to configure the `from` module, and multiple parameter names, their corresponding bound fields, and parameter types to configure the `select` module. After clicking the submit button, the data warehouse data retrieval interface will be generated.

[0147] In this way, developers can configure the parameter values ​​of various parameters of the data warehouse data retrieval interface to be developed through visual configuration, without having to develop the data warehouse data retrieval interface by writing code.

[0148] Step 102: Generate the data warehouse data retrieval interface based on the correspondence between the data retrieval module and the parameters, as well as the parameter values ​​of each parameter.

[0149] This mapping relationship was generated by disassembling the data warehouse data retrieval interface code according to the syntax rules of Structured Query Language (SQL). The data warehouse data retrieval interface is used by callers after it is published.

[0150] In one embodiment, the implementation process of step 102 may include the following steps:

[0151] S1021: Generate a structured object corresponding to the data retrieval module based on the parameter values ​​of the corresponding parameters of the data retrieval module.

[0152] In one implementation, after developers perform visual configuration on the data warehouse data retrieval interface configuration frontend, the frontend generates lightweight data exchange format (JavaScript Object Notation, JSON) data based on the parameter values ​​of the data retrieval module and its corresponding parameters, and sends this JSON data to the backend (e.g., a server). The data warehouse data retrieval interface configuration frontend can be a World Wide Web (Web) frontend. The backend then converts the JSON data to obtain the structured object corresponding to the data retrieval module.

[0153] The data retrieval module may include at least one of the following modules: a query (Select) module, a source (From) module, a filter (Where) module, an aggregation (Group by) module, a pagination module (Limit offset) and a sorting (Orderby) module.

[0154] See Figure 4The image shown is an example diagram of a data warehouse data retrieval interface code. Figure 4 The example demonstrates a code snippet for a data warehouse data retrieval interface. Figure 4 By disassembling the data warehouse data retrieval interface code, multiple data retrieval modules can be obtained. Specifically, Figure 4 The SELECT select_expr[……] in the code can be broken down into the Select module. Figure 4 The `FROMtable_references[...]` in the `<Form>` module can be broken down into the `From` module. Figure 4 The WHERE clause with `where_condition` can be broken down into the `Where` module. Figure 4 In this context, GROUP BY{……} can be broken down into the Group by module, ORDER BY{……} into the Order by module, and LIMIT{[offset,]……} into the Limit offset module.

[0155] The structured object may include at least one of the following objects: result set object, database table object, conditional filter object, aggregation object, pagination object, and sorting object.

[0156] The query module, source module, filter module, aggregation module, pagination module, and sorting module have the following structured objects, which are respectively: result set object, database table object, condition filter object, aggregation object, pagination object, and sorting object.

[0157] The system comprises the following modules: Query module for retrieving result sets; Source module for configuring database information; Filter module for configuring query filtering conditions; Aggregation module for aggregation calculations; Pagination module for paginated queries; and Sorting module for sorting data based on a sorting field.

[0158] In this way, the data warehouse data retrieval interface code can be broken down into multiple data retrieval modules, allowing developers to configure them visually and generate multiple structured objects based on the parameters configured by the developers for each data retrieval module.

[0159] See Figure 5 The image shown is an example diagram of a structured object. The following section will combine... Figure 5 This describes the structured objects and their associated parameters.

[0160] Figure 5In this context, the parameters corresponding to the result set object include a list of returned result parameters; the parameters corresponding to the database / table object include at least one of the following: database (db), table, and schema; the parameters corresponding to the condition filter object include at least one of the following: query condition input, class attribute parameter (field), and calculation parameter (caculate); the parameters corresponding to the aggregation object include the aggregation parameter (group key); the parameters corresponding to the pagination object include at least one of the following: limit query parameter (limit) and offset query parameter (offset); and the parameters corresponding to the sorting object include at least one of the following: sorting type parameter (order type) and aggregate sorting parameter (order By key).

[0161] In one implementation, the parameters can be categorized according to their application method. Specifically, the types of parameters can include request input parameters, output result parameters, and built-in parameters. Request input parameters are used for data input, output result parameters are used for result output, and built-in parameters are used for system configuration.

[0162] The returned result parameter list is the output result parameter; the parameters of the database table object, the condition filter object, and the aggregation object are built-in parameters; and the parameters of the pagination object and the sorting object are pagination input parameters. The query condition input parameter and the pagination input parameter are both types of request input parameters.

[0163] S1022: Generate a data warehouse data retrieval interface based on structured objects.

[0164] The following is a breakdown of the data warehouse data retrieval interface code.

[0165] Since SQL is the core logic of the data warehouse data retrieval interface, the execution process of the data warehouse data retrieval interface can be summarized as request parsing, SQL assembly, querying, and returning results. The parameters of the data warehouse data retrieval interface can be divided into request input parameters, output result parameters, and built-in parameters. Therefore, the implementation process of step 102 can also include:

[0166] Following SQL syntax rules, the data warehouse data retrieval interface code is decomposed to obtain multiple data retrieval modules, and a correspondence between the data retrieval modules and their parameters is generated. Based on this correspondence, a structured object corresponding to each data retrieval module is defined.

[0167] The result set object corresponds to the SELECT clause in SQL and contains the fields returned by the query. API developers only need to configure the naming and type of the parameters returned by the data warehouse data retrieval interface to generate the result set object.

[0168] In this context, the database table objects correspond to the FROM clause in SQL. A metadata center can be used to uniformly register and manage the various database tables used to determine the data source. API developers can then configure and generate database table objects by selecting database tables from the metadata center.

[0169] See Figure 6 The image shown is an example diagram of a method for constructing a library table object. Figure 6 In this process, developers first register the data source in the metadata center. Then, they configure the front-end through the data warehouse data retrieval interface, opening the configuration interface and selecting candidate database tables from the options provided. Based on the selected database tables, the front-end generates corresponding JSON data and sends it to the server. The server then constructs the database table object based on this JSON data.

[0170] In this way, registering and managing each database table through the metadata center makes database table management more convenient. When configuring the data warehouse data retrieval interface, selecting database tables from the metadata center allows developers to intuitively query and trace the reference relationship between the data warehouse data retrieval interface and the data source, avoiding the problems of poor flexibility and difficulty in management that exist in the traditional method.

[0171] The conditional filtering object includes a calculation type, a target field, and a union operator. Developers configure the target field, calculation type, and request parameters for each conditional expression object, as well as the union operators between conditional expression objects, forming a parallel or nested structure. Then, the data warehouse data retrieval interface configuration frontend converts this structure into a JSON string and passes it to the server. See also... Figure 7 The image shown is an example diagram of a JSON string. Figure 7 The example shows a JSON string of a conditional filter object. This JSON is a recursively nested structure. Figure 7 The `subObjList` in the code represents a list of sub-condition JSON objects. This list can contain either conditional expression objects or sub-condition structures formed by multiple sets of conditional expressions. Through this recursive nested structure, the server can convert the JSON string into a conditional expression: `WHERE("A"=${param1}AND"B"LIKE"%${param2}%")OR("C"IN%{param3}AND"D"<%{param4})`. Here, `%{}` is an SQL placeholder, and the content within `{}` represents the request parameters. Replacing the placeholders with the request parameters yields the complete conditional expression.

[0172] The aggregation object forms an aggregation query clause, GROUP BY key1, key2, by configuring a list of grouped keys.

[0173] The pagination object needs to define API pagination input parameters and set the pagination page number, page size, and sorting field. The pagination input parameters and sorting field are then bound to construct the LIMIT...OFFSET...ORDER BY pagination clause.

[0174] See Figure 8 The image shown is an example diagram of a data warehouse data retrieval interface development. Figure 8 In this process, the system configures return parameters for the output parameters of the query module and constructs the result set object for the query module using these parameters. It also configures the data source for the built-in parameters of the source module and constructs the database table object for the source module. Furthermore, it configures filtering conditions for the built-in parameters of the filtering module and constructs a conditional filtering object. Similarly, it configures aggregation parameters for the built-in parameters of the aggregation module and constructs an aggregation object. Finally, it configures pagination parameters for the pagination module and constructs a pagination object. Finally, it configures sorting parameters for the sorting module and constructs a sorting object. Finally, it generates a data warehouse data retrieval interface based on each structured object. When using the data warehouse data retrieval interface, the code can be converted based on the data retrieval parameters in the user's data warehouse data retrieval interface request to obtain the corresponding SQL statement.

[0175] In this embodiment of the application, only the above-mentioned Figure 8 The configuration order of each data retrieval module is used as an example for explanation. In actual applications, the configuration order of each data retrieval module can be arbitrary and is not restricted here.

[0176] In this embodiment, a configurable data warehouse data retrieval interface development process is implemented based on structured objects, realizing the transformation from a hard-coded development mode to a low-code configurable mode, thereby improving the development efficiency of the data team.

[0177] Furthermore, after generating the data warehouse data retrieval interface, the data warehouse data retrieval interface is published.

[0178] Traditional data warehouse data retrieval interface delivery processes involve multiple stages, including code writing, building and deploying server-side packages, and integration and release to the specific runtime environment. This entire process is a server-side code engineering CI / CD process, which is cumbersome, time-consuming, and has low fault tolerance. This application proposes a lifecycle management model for APIs based on metadata distribution to implement a CI / CD process for the data warehouse data retrieval interface itself, addressing the problems of lengthy and inflexible traditional data warehouse data retrieval interface delivery processes.

[0179] The method for publishing the data warehouse data retrieval interface in this application embodiment is described below.

[0180] In one implementation, a data warehouse data retrieval interface is published in the interface runtime environment. Optionally, the interface runtime environment may include a sandbox environment and a production environment.

[0181] See Figure 9 The diagram shown is a flowchart of a method for publishing a data warehouse data retrieval interface. The following section will combine... Figure 9 This document describes the method for publishing the data warehouse data retrieval interface. The specific implementation process of this method includes:

[0182] Step 801: In the sandbox environment, publish the data warehouse data retrieval interface.

[0183] In one implementation, after generating the data warehouse data retrieval interface through the developer's above-mentioned data warehouse data retrieval interface configuration, the data warehouse data retrieval interface in the pending release state is obtained, and the data warehouse data retrieval interface is released to the sandbox environment, and the data warehouse data retrieval interface in the sandbox environment is tested.

[0184] Furthermore, before testing the data warehouse data retrieval interface, unit testing can be performed on the data warehouse data retrieval interface first, and the data warehouse data retrieval interface can be tested only after the unit testing is confirmed to be passed.

[0185] Step 802: If the data warehouse data retrieval interface test is successful, then publish the data warehouse data retrieval interface in the online environment.

[0186] In one implementation, if the sandbox environment test is confirmed to be successful, a formal release can be requested. After approval by the API administrator, the data warehouse data retrieval interface is released to the online environment.

[0187] Since the data warehouse data retrieval interface needs to be tested by the business side before it can be released to the online environment after configuration, in order to ensure the isolation between the testing and the online formal call links, the interface runtime environment deployed based on the data gateway query service is divided into a sandbox environment and an online environment.

[0188] The sandbox environment can include at least one sandbox cluster, and the online environment can include at least one online cluster. The query service can be deployed across multiple clusters as Java Virtual Machine (JVM) processes. The query service can be deployed in any virtual machine environment, including Kubernetes cloud-native environments and other virtual machine environments. The number of clusters and the number of instances in each cluster can be scaled up or down according to actual business needs, and there are no restrictions here.

[0189] Furthermore, in order to minimize the differences between the sandbox environment and the online environment for the caller and reduce the caller's adaptation costs, this application proposes a request routing scheme, which selects whether to run the data warehouse data retrieval interface in the sandbox environment or the data warehouse data retrieval interface in the online environment through a data gateway.

[0190] In one implementation, the interface development device includes a data gateway. As an example, the data gateway may be a Hypertext Transfer Protocol (HTTP) gateway.

[0191] The data gateway is used to receive data retrieval requests from other devices, which include the domain name of the data retrieval interface. Based on the environment parameters in the data retrieval request, it selects between a sandbox environment and a production environment to call the data warehouse data retrieval interface in the selected interface operating environment. The domain name of the data retrieval interface is obtained by the data gateway after instance registration.

[0192] See Figure 10 The diagram shown is an example of the architecture of a data warehouse data retrieval interface publishing system.

[0193] Figure 10 The system comprises a data gateway, a sandbox environment, and a production environment. The sandbox environment includes sandbox cluster 1 and sandbox cluster 2. The production environment includes production cluster 1 and production cluster 2.

[0194] It's important to note that the cluster information in the sandbox environment and the online environment is a black box to the caller. The data gateway is the sole entry point for the caller's data retrieval requests (i.e., API requests); all sandbox clusters and online clusters obtain their respective data retrieval requests through a unified data gateway.

[0195] In one implementation, the data gateway first registers its instance to obtain the domain name of the data retrieval interface, which can then be invoked by callers. The caller sends a data retrieval request to the data gateway based on this domain name. After receiving data retrieval requests from other devices, the data gateway selects whether to invoke the data retrieval interface of the sandbox environment or the production environment based on the environment parameters in the request.

[0196] In one implementation, the data gateway technology selection chooses Kong Gateway. Kong is a highly available and easily scalable open-source API Gateway project based on OpenResty, which allows configuration of routing policies for downstream service calls. OpenResty is a high-performance web platform based on Nginx and Lua.

[0197] As an example, the environment parameter is Header. If the Header parameter value is "sandbox" and the route identifier "target" is passed, the data retrieval request will be forwarded to the sandbox environment. If the Header parameter value is "online", the data retrieval request will be forwarded to the online environment.

[0198] This allows for the configuration of routing policies based on Kong Gateway, enabling the data gateway to determine the appropriate environment by evaluating the header parameters of data retrieval requests (e.g., HTTP requests). The caller does not need to modify the API request domain name (i.e., the data retrieval interface domain name); they only need to adjust the parameter values ​​passed in the header to switch between sandbox testing and production environment calls.

[0199] In this embodiment, once the data warehouse data retrieval interface is developed and released, it can be displayed in the data gateway API marketplace. Callers from various business lines can browse the API marketplace to determine if a data warehouse data retrieval interface that meets their current needs exists. If a caller determines that a suitable data warehouse data retrieval interface exists, they can apply to the API administrator for usage permissions and the corresponding request key for that interface.

[0200] This approach, based on the API CI / CD process, facilitates a shift from server-side code-based continuous integration to API-based continuous integration. A multi-cluster deployment architecture, encompassing sandbox and production environments, centrally manages and aggregates data warehouse retrieval interfaces. A data gateway unifies the management of data retrieval requests to these interfaces, reducing adaptation costs for API callers and improving integration testing, debugging, and requirement delivery efficiency. Centralized development and management of data warehouse retrieval interfaces via the data gateway enable "develop once, reuse many times," achieving API and data source reuse and avoiding siloed development. This saves manpower and reduces data storage costs.

[0201] Furthermore, after publishing the data warehouse data retrieval interface, the data warehouse data retrieval interface can also be updated.

[0202] See Figure 11 The diagram shows a flowchart of a method for updating a data warehouse data retrieval interface. The specific implementation process of this method includes:

[0203] Step 1000: Monitor the configuration update information of the data warehouse data retrieval interface.

[0204] The configuration update information refers to the parameter updates for the data warehouse data retrieval interface. The configuration update information includes at least one of the following: parameters of the data warehouse data retrieval interface, SQL configuration information, and generated JSON.

[0205] Step 1001: In response to the existence of configuration update information, update the data warehouse data retrieval interface based on the configuration update information.

[0206] In one implementation, when performing step 1001, the following steps may be adopted:

[0207] S10011: If the storage root path of the configuration update information is determined to represent the sandbox environment, then the data warehouse data retrieval interface in the sandbox environment is updated.

[0208] Optionally, the storage root path in the configuration update information can be named according to the interface runtime environment for subsequent differentiation. In this way, the updated data warehouse data retrieval interface can be selected through the storage root path in the configuration update information.

[0209] S10012: If the storage root path for configuration update information is determined to represent the online environment, then the data warehouse data retrieval interface in the online environment is updated.

[0210] Specifically, interface development equipment can also include management services and coordination services. Combined with... Figure 10 The API publishing system also includes management services and coordination services (Zookeeper).

[0211] Specifically, when the data warehouse data retrieval interface is updated, the configuration update information for the data warehouse data retrieval interface in each interface's operating environment is uniformly distributed through the management service.

[0212] In one implementation, a Zookeeper listener is first registered through the query service. Then, when upgrading (i.e., updating) the data warehouse retrieval interface, developers can use the management service to distribute the metadata (i.e., configuration update information) of the data warehouse retrieval interface to the metadata node in Zookeeper. The interface runtime environment communicates with Zookeeper through the query service and listens for metadata change events through the Zookeeper Watch mechanism.

[0213] As an example, the storage root path for the sandbox environment is ` / dataservice / sandbox`, while the storage root path for the production environment is ` / dataservice / prd`. The sandbox environment listens to API nodes under ` / dataservice / sandbox`. The production environment listens to API nodes under ` / dataservice / prd`. When metadata changes, each query service instance re-fetches the metadata of a specific API node under the listening path and updates it to memory. This upgrades the data warehouse data retrieval interface, enabling it to handle new API requests.

[0214] Furthermore, the data warehouse data retrieval interface can be taken offline via API shutdown information.

[0215] In one implementation, the API offline information of the data warehouse data retrieval interface is monitored; the API offline information is used to indicate that the data warehouse data retrieval interface is offline; in response to the existence of API offline information, the data warehouse data retrieval interface is controlled to go offline based on the API offline information.

[0216] It should be noted that the release of data warehouse data retrieval interfaces is also achieved through management services and coordination services.

[0217] In one implementation, the API publishing information of the data warehouse data retrieval interface is monitored; the API publishing information is used to publish the data warehouse data retrieval interface; in response to the existence of API publishing information, the data warehouse data retrieval interface is published based on the API publishing information.

[0218] In this embodiment, based on the same principle as updating the data warehouse data retrieval interface through configuration update information, the data warehouse data retrieval interface is taken offline through API offline information, and the data warehouse data retrieval interface is published through API publishing information, which will not be elaborated here.

[0219] In this embodiment, if changes are needed to the API in the online environment, online configuration upgrades can be performed. The entire CI / CD cycle of the API only involves changes to metadata, which originate from configuration changes. Developers do not need to package and deploy the entire server-side code in the corresponding physical environment; they only need to deploy both the sandbox and production environments simultaneously, configure the web interface, and manage the release of the data warehouse data retrieval interface in the sandbox and production environments, as well as distribute metadata to the corresponding environments, thus achieving upgrades to the data warehouse data retrieval interface. A data gateway centrally manages data warehouse data retrieval interface requests sent to different interface runtime environments, and a management service centrally manages configuration update information for data warehouse data retrieval interfaces in different runtime environments. This achieves full lifecycle management of the data warehouse data retrieval interface, solving the problems of long integration testing processes and high update costs associated with data warehouse data retrieval interfaces.

[0220] Furthermore, to meet the ad-hoc query needs of different scenarios, data warehouse technology (Extract-Transform-Load, ETL) is typically used to import data from the data warehouse into different data sources to provide services for business-side queries. Therefore, the data gateway needs to have the ability to query multiple types and versions of data sources. The traditional development model typically involves importing multiple driver JAR packages via Maven into an open-source framework (e.g., Spring Boot) project, ultimately building the main project JAR package. When querying a data source, a data source connection needs to be created using the driver for each data source. However, this approach has a problem: due to the large variety and versions of data sources, importing multiple data source JAR packages results in a bloated and large project, easily leading to conflicts when different versions of the same data source JAR are imported. When faced with multiple conflicting version dependencies, only one can be selected, which can cause some data sources to fail to connect. Therefore, in this embodiment, the following improvements have been made to the data source configuration and connection.

[0221] In one implementation, the interface development device further includes an SQL adapter; the SQL adapter is used to convert the structured objects in the data warehouse data retrieval interface into SQL statements based on the input parameters in the data retrieval request sent by other devices.

[0222] This is because there are slight differences in the SQL syntax across various data sources. Therefore, each data source plugin has a corresponding SQL adapter. The SQL adapters for different plugins can be the same or different. Specifically, when the API executes query logic, the SQL adapter converts the structured object into the SQL statement corresponding to the data source.

[0223] In one embodiment, the interface development device further includes a driver routing table; the driver routing table registers drivers corresponding to each database; the driver is used to establish a data source connection between the interface development device and the database corresponding to the data retrieval request; the data source connection is used for the data warehouse data retrieval interface to retrieve data from the corresponding database.

[0224] The data source plugin is a JAR file that integrates a driver, an SQL customization module, and a connection management module.

[0225] In one implementation, after the plugin is loaded, the plugin's driver is registered in a driver routing table in memory, and the driver routing table is stored using a local cache (e.g., Guava cache). As an example, the key corresponding to the driver in the driver routing table can be the driver type plus the version number, and the value is the driver object.

[0226] In one implementation, the driver corresponding to each database is loaded by a class loader; there is a one-to-one correspondence between each driver and each type of loader; the class loader is generated by inheriting the class loader base class and overriding the loadClass method.

[0227] Specifically, the driver for the data source of the SQL query can be loaded through the Driver class (i.e., the class loader), and the data source connection between the interface development device and the data source can be established through the driver in accordance with the Java Database Connectivity (JDBC) standard.

[0228] It's important to note that in traditional methods, the data gateway's server is developed using the Java language. When loading Java classes, a parent delegation model is typically used. Dependencies imported in Maven are usually loaded by the application class loader (AppClassLoader). Since classes with the same name are only loaded once, if different versions of the same driver package are imported, the driver class with the same name will only be loaded once, making it impossible for multiple versions of driver classes to coexist, thus leading to class "conflicts."

[0229] To overcome the limitations of class loading, this embodiment of the application defines a custom class loader and generates a class loader by inheriting from the class loader base class and overriding the loadClass method. This breaks the parent delegation model and prevents the driver program from being loaded by AppClassLoader. In this way, each data source plugin has its own corresponding class loader, allowing classes from different versions of plugin JARs to be loaded into the application, thus resolving the aforementioned conflict.

[0230] In one embodiment, the interface development device further includes a storage management plugin library; the storage management plugin library stores at least one plugin containing a driver; the storage path of the plugin is generated based on the corresponding database type name and data source version number.

[0231] Optionally, the storage management plugin library can use a distributed file system (Hadoop Distributed File System, HDFS).

[0232] As an example, the root storage path for each plugin is consistent, starting with / data-plugin. The naming convention for subpaths within each plugin is .. / {data source type name} / {data source version number}. For instance, the plugin for MySQL 8.0 would have the storage path / data-plugin / MySQL / 8.0 / .

[0233] In this way, the storage management plugin library can be used to manage the storage of various plugins in a unified manner, so that the data gateways of each cluster can use the plugins.

[0234] In one implementation, the interface development device further includes HikariCP; HikariCP is used to manage data source connections established for each database.

[0235] Considering the time and performance overhead involved in creating data source connections, a connection pool is used to uniformly manage the data source connections established by the driver for each data source. In this embodiment, given HikariCP's high performance and reliable stability, HikariCP is used as a connection pool. HikariCP is used to initialize each created data source connection and to uniformly manage the connections for each data source. Furthermore, HikariCP can be used to perform periodic security checks (i.e., health checks) on each data source. This achieves the goal of data source connection reuse and also reduces performance overhead.

[0236] In this embodiment, HikariCP is used as an example of a connection pool for illustration. In actual applications, any other connection pool software can be used, and no restrictions are imposed here.

[0237] In one embodiment, the interface development equipment further includes a database management center; the database management center is used to register and manage various databases.

[0238] See Figure 11 for an example diagram of a database or data warehouse query process. The following section will combine... Figure 12The driver loading process is described below. Multiple plugins are managed centrally through a storage management plugin library, each containing a driver. Multiple class loaders load the corresponding drivers from the storage management plugin library and register them in the driver routing table in memory. When the data warehouse data retrieval interface is called, the data source type and version number of the data retrieval request are determined. Based on this data source type and version number, the driver object (i.e., the driver corresponding to the data source) is retrieved from the driver routing table, and then a data source connection is established with the requested database through this driver object. The databases include a data warehouse (Hive) and MySQL.

[0239] In this embodiment, the data source connection driver is plugged in and a driver routing table is established, which can support multiple versions and types of data sources. This makes the types of data sources available for various business scenarios richer and more flexible.

[0240] Furthermore, if it is necessary to support the adaptation of new data sources, only plugin development is required, which is low-cost and fast. After the plugin passes the test, the plugin can be managed uniformly through the plugin library, achieving the goal of developing once and sharing globally.

[0241] See Figure 13 The diagram shown is an example of the architecture of a data warehouse data retrieval interface development system. Figure 13 The system includes a traffic entry layer, a business logic layer, an execution engine layer, a plugin routing layer, a connection data source, an API configuration service, and a relational database management system (RDBMS).

[0242] The traffic entry layer includes rate limiting and circuit breaking components, gateway authentication, and call auditing. The business logic layer includes request parameter parsing, Mybatis, and metadata caching. The execution engine layer includes connection pool caching, plugin adapters, and User-Defined Functions (UDF) modules. The plugin layer includes plugins for multiple databases. The data source layer includes multiple databases. The API configuration service includes API information management, API lifecycle management, permission management, and Zookeeper.

[0243] In one implementation, the traffic ingress layer receives data retrieval requests from the upstream system, authenticates the requests via gateway authentication, and then sends the authenticated requests to the business logic layer. The business logic layer analyzes the input parameters of the data retrieval requests using a request parameter analysis module, obtains the data warehouse data retrieval interface through a metadata cache, and sends the parsed data retrieval request parameters and the data warehouse data retrieval interface to the execution engine layer. The execution engine layer uses a plugin adapter to perform code conversion on the data retrieval request parameters and the data warehouse data retrieval interface to obtain the data warehouse data retrieval interface code. Based on the data retrieval request parameters, it determines whether a data source connection for the corresponding database exists in the connection pool cache. If so, it directly obtains the established data source connection from the connection pool cache; otherwise, it establishes a data source connection with the database in the data retrieval request through the corresponding plugin in the plugin layer, and executes the data warehouse data retrieval interface code to retrieve data from the database in the data retrieval request through this data source connection.

[0244] Furthermore, if it is determined that the data warehouse data retrieval interface needs to be updated, the API developer will send configuration update information to ZooKeeper through API information management. The metadata cache obtains the configuration update information through ZooKeeper and updates the data warehouse data retrieval interface based on this configuration update information.

[0245] Optionally, the data warehouse data retrieval interface can be applied to multiple scenarios such as data products, report analysis, and client calls. The data warehouse data retrieval interface has a short response time and a high success rate. By making the data warehouse data retrieval interface a unified traffic entry point for the data warehouse, the convenience and efficiency of data retrieval for business systems are greatly improved.

[0246] In this embodiment, a configurable data warehouse data retrieval interface development process is implemented based on SQL structured objects. Furthermore, a data warehouse data retrieval interface marketplace addresses the issues of low development efficiency and poor reusability. Moreover, an API CI / CD process based on metadata distribution resolves the problems of lengthy data API delivery processes and high update costs. Further, the data source connection driver is pluggable, and a driver routing table is established, resolving the issues of high cost, incompatibility with multiple data source versions, and chaotic data source connection management when adapting data warehouse data retrieval interfaces to multiple data sources.

[0247] Exemplary media

[0248] After introducing the development method for a data warehouse data retrieval interface according to an exemplary embodiment of this application, the following will refer to... Figure 14 The storage medium of an exemplary embodiment of this application will be described.

[0249] Figure 14The computer-readable storage medium 1400 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this application.

[0250] In exemplary embodiments of this application, the computer-readable storage medium 1400 may be a readable signal medium or a readable storage medium. It stores a program product capable of implementing the methods described above. In some possible implementations, various aspects of this application may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this application according to various exemplary embodiments.

[0251] More specific examples of the computer-readable storage medium 1400 in this application may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0252] In this application, the computer-readable storage medium 1400 may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0253] Optionally, the program code contained on the computer-readable storage medium 1400 may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0254] In some application scenarios, program code for performing the operations of this application can be written in any combination of one or more programming languages. These programming languages ​​include object-oriented programming languages—such as Java and C++—as well as conventional procedural programming languages—such as C or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0255] Exemplary device

[0256] Based on the same inventive concept, this application also provides a development apparatus for a data warehouse data retrieval interface. Since the principle of the above apparatus and device in solving the problem is similar to that of a development method for a data warehouse data retrieval interface, the implementation of the above apparatus can refer to the implementation of the method, and the repeated parts will not be described again.

[0257] See Figure 15 The diagram shown is a structural block diagram of a development apparatus for a data warehouse data retrieval interface according to an embodiment of this application. In some embodiments, the development apparatus for a data warehouse data retrieval interface exemplified in this application includes:

[0258] The acquisition unit 1501 is used to respond to the developer's configuration operation on each parameter configuration item in the data retrieval interface configuration interface and obtain the parameter value of each parameter.

[0259] The generation unit 1502 is used to generate a data warehouse data retrieval interface based on the correspondence between the data retrieval module and the parameters, as well as the parameter values ​​of each parameter. The correspondence is generated after the data warehouse data retrieval interface code is decomposed according to SQL syntax rules. The data warehouse data retrieval interface is used for callers to call after it is published.

[0260] In one embodiment, the generation unit 1502 is used for:

[0261] Based on the parameter values ​​of the corresponding parameters of the data retrieval module, generate the structured object corresponding to the data retrieval module;

[0262] Generate a data warehouse data retrieval interface based on structured objects.

[0263] In one implementation, the structured object includes at least one of the following objects:

[0264] Result set objects, table objects, conditional filter objects, aggregation objects, pagination objects, and sorting objects;

[0265] The parameters corresponding to the result set object include a list of returned result parameters; the parameters corresponding to the database table object include at least one of the following: database, data table, and database schema; the parameters corresponding to the condition filter object include at least one of the following: query condition input parameters, class attribute parameters, and calculation parameters; the parameters corresponding to the aggregation object include aggregation parameters; the parameters corresponding to the pagination object include at least one of the following: limit query parameters and offset query parameters; the parameters corresponding to the sorting object include at least one of the following: sorting type parameters and aggregate sorting parameters.

[0266] In one embodiment, the generating unit 1502 is further configured to:

[0267] In the API runtime environment, publish the data warehouse data retrieval API.

[0268] In one implementation, the interface operating environment includes a sandbox environment and an online environment;

[0269] The generating unit 1502 is also used for:

[0270] In a sandbox environment, publish the data warehouse data retrieval interface;

[0271] If the data warehouse data retrieval interface test is successful, then publish the data warehouse data retrieval interface in the online environment.

[0272] In one embodiment, the interface development device includes a data gateway;

[0273] The data gateway is used to receive data retrieval requests sent by other devices, which include the domain name of the data retrieval interface. Based on the environment parameters in the data retrieval request, it selects between a sandbox environment and a production environment to call the data warehouse data retrieval interface in the selected interface operating environment.

[0274] The domain name for the data retrieval interface is obtained after the data gateway registers an instance.

[0275] In one embodiment, the interface development device also includes an SQL adapter;

[0276] The SQL adapter is used to convert structured objects in the data warehouse data retrieval interface into SQL statements based on the input parameters in the data retrieval requests sent by other devices.

[0277] In one embodiment, the interface development device also includes a driver routing table;

[0278] The driver routing table contains the drivers corresponding to each database.

[0279] The driver is used to establish a data source connection between the interface development device and the database corresponding to the data retrieval request;

[0280] The data source connection is used to enable the data warehouse data retrieval interface to retrieve data from the corresponding database.

[0281] In one embodiment, the interface development device further includes a storage management plugin library; the storage management plugin library stores at least one plugin containing a driver.

[0282] The plugin's storage path is generated based on the corresponding database type name and data source version number.

[0283] In one implementation, the driver corresponding to each database is loaded by a class loader; there is a one-to-one correspondence between each driver and each type of loader; the class loader is generated by inheriting the class loader base class and overriding the loadClass method.

[0284] In one embodiment, the interface development device further includes HikariCP;

[0285] HikariCP is used to manage data source connections established for each database.

[0286] In one embodiment, the interface development equipment further includes a database management center;

[0287] The database management center is used for registering and managing various databases.

[0288] In one embodiment, the generating unit 1502 is further configured to:

[0289] Listen for configuration update information of the data warehouse data retrieval interface; the configuration update information refers to the parameter updates in the data warehouse data retrieval interface.

[0290] In response to the existence of configuration update information, the data warehouse data retrieval interface is updated based on the configuration update information.

[0291] In one embodiment, the generating unit 1502 is further configured to:

[0292] If the storage root path for configuration update information is determined to represent the sandbox environment, then the data warehouse data retrieval interface in the sandbox environment is updated.

[0293] If the storage root path for configuration update information is determined to represent the online environment, then the data warehouse data retrieval interface in the online environment will be updated.

[0294] The development method for a data warehouse data retrieval interface in this application includes: responding to the developer's configuration operations for each parameter configuration item in the data retrieval interface configuration interface, obtaining the parameter values ​​of each parameter; generating the data warehouse data retrieval interface based on the correspondence between the data retrieval module and the parameters, and the parameter values ​​of each parameter; the correspondence is generated after decomposing the data warehouse data retrieval interface code according to the syntax rules of Structured Query Language (SQL); the data warehouse data retrieval interface is used for invocation by callers after deployment. This simplifies the cumbersome operations of developing a data warehouse data retrieval interface and improves the efficiency of data warehouse data retrieval interface development.

[0295] Exemplary electronic devices

[0296] Having introduced the development method, medium, and apparatus for a data warehouse data retrieval interface according to exemplary embodiments of this application, the following references... Figure 16 An electronic device according to an exemplary embodiment of this application will be described.

[0297] Figure 16 The electronic device 1600 shown is merely an example and should not impose any limitations on the functionality and scope of the embodiments of this application.

[0298] like Figure 16 As shown, the electronic device 1600 is presented in the form of a general electronic device. The components of the electronic device 1600 may include, but are not limited to: at least one processing unit 1610, at least one storage unit 1620, and a bus 1630 connecting different system components (including storage unit 1620 and processing unit 1610).

[0299] The storage unit stores program code, which can be executed by the processing unit 1610, causing the processing unit 1610 to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this application.

[0300] In some embodiments, the processing unit 1610 may execute the above embodiments.

[0301] Storage unit 1620 may include readable media in the form of volatile storage units, such as random access memory (RAM) 16201 and / or cache memory 16202, and may further include read-only memory (ROM) 16203.

[0302] Storage unit 1620 may also include a program / utility 16204 having a set (at least one) program module 16205, such program module 16205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0303] The bus 1630 may include a data bus, an address bus, and a control bus.

[0304] Electronic device 1600 can also communicate with one or more external devices 1640 (e.g., keyboard, pointing device, Bluetooth device, etc.) via input / output (I / O) interface 1650. Furthermore, electronic device 1600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 1660. Figure 16 As shown, network adapter 1660 communicates with other modules of electronic device 1600 via bus 1630. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 1600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0305] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause an electronic device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this application.

[0306] It should be noted that although several units / modules or sub-units / modules of the development apparatus for the data warehouse data retrieval interface are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0307] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0308] While the spirit and principles of this application have been described with reference to several specific embodiments, it should be understood that this application is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A development method for a data warehouse data retrieval interface, applied to an interface development device, characterized in that, include: In response to the developer's configuration operations for each parameter configuration item in the data retrieval interface configuration interface, obtain the parameter values ​​of each parameter; Based on the correspondence between the data retrieval module and the parameters, and the parameter values ​​of each parameter, a data warehouse data retrieval interface is generated; the correspondence is generated by decomposing the data warehouse data retrieval interface code according to the syntax rules of Structured Query Language (SQL); the data warehouse data retrieval interface is used for callers after being published. The step of generating a data warehouse data retrieval interface based on the correspondence between the data retrieval module and the parameters, and the parameter values ​​of each parameter, includes: generating a structured object corresponding to the data retrieval module based on the parameter values ​​of the parameters corresponding to the data retrieval module; and generating the data warehouse data retrieval interface based on the structured object.

2. The method according to claim 1, characterized in that, The structured object includes at least one of the following objects: Result set objects, table objects, conditional filter objects, aggregation objects, pagination objects, and sorting objects; The parameters corresponding to the result set object include a list of returned result parameters; the parameters corresponding to the database table object include at least one of the following: database, data table, and database schema; the parameters corresponding to the condition filtering object include at least one of the following: query condition input parameters, class attribute parameters, and calculation parameters; the parameters corresponding to the aggregation object include aggregation parameters; the parameters corresponding to the pagination object include at least one of the following: limit query parameters and offset query parameters; the parameters corresponding to the sorting object include at least one of the following: sorting type parameters and aggregate sorting parameters.

3. The method according to claim 1 or 2, characterized in that, The method further includes: In the interface runtime environment, publish the data warehouse data retrieval interface.

4. The method according to claim 3, characterized in that, The interface operating environment includes a sandbox environment and an online environment; The process of publishing the data warehouse data retrieval interface in the interface runtime environment includes: In the sandbox environment, publish the data warehouse data retrieval interface; If the data warehouse data retrieval interface test is successful, then the data warehouse data retrieval interface will be published in the online environment.

5. The method according to claim 4, characterized in that, The interface development equipment includes a data gateway; The data gateway is used to receive data retrieval requests sent by other devices, which include the domain name of the data retrieval interface, and to select between the sandbox environment or the online environment based on the environment parameters in the data retrieval request, so as to call the data warehouse data retrieval interface in the selected interface running environment. The domain name of the data retrieval interface is obtained after the data gateway registers an instance.

6. The method according to claim 1, characterized in that, The interface development equipment also includes an SQL adapter; The SQL adapter is used to convert the structured objects in the data warehouse data retrieval interface into SQL statements based on the input parameters in the data retrieval requests sent by other devices.

7. The method according to claim 5, characterized in that, The interface development device also includes a driver routing table; The driver routing table contains the drivers corresponding to each database. The driver program is used to establish a data source connection between the interface development device and the database corresponding to the data retrieval request; The data source connection is used to enable the data warehouse data retrieval interface to retrieve data from the corresponding database.

8. The method according to claim 7, characterized in that, The interface development device also includes a storage management plugin library; the storage management plugin library stores at least one plugin containing a driver. The storage path of the plugin is generated based on the corresponding database type name and data source version number.

9. The method according to claim 7, characterized in that, The drivers for each database are loaded by class loaders; there is a one-to-one correspondence between each driver and each class loader; the class loaders are generated by inheriting from the class loader base class and overriding the loadClass method.

10. The method according to claim 7, characterized in that, The interface development equipment also includes HikariCP; HikariCP is used to manage data source connections established for each database.

11. The method according to claim 7, characterized in that, The interface development equipment also includes a database management center; The database management center is used for registering and managing various databases.

12. The method according to claim 3, characterized in that, The method further includes: Monitor the configuration update information of the data warehouse data retrieval interface; the configuration update information refers to the parameter update information in the data warehouse data retrieval interface. In response to the existence of configuration update information, the data warehouse data retrieval interface is updated based on the configuration update information.

13. The method according to claim 12, characterized in that, The step of updating the data warehouse data retrieval interface based on the configuration update information includes: If the storage root path of the configuration update information is determined to represent the sandbox environment, then the data warehouse data retrieval interface in the sandbox environment is updated. If the storage root path of the configuration update information is determined to represent the online environment, then the data warehouse data retrieval interface in the online environment is updated.

14. A development apparatus for a data warehouse data retrieval interface, characterized in that, Applied to interface development equipment, the device includes: The acquisition unit is used to respond to the developer's configuration operations on each parameter configuration item in the data retrieval interface configuration interface and to obtain the parameter value of each parameter. The generation unit is used to generate a data warehouse data retrieval interface based on the correspondence between the data retrieval module and the parameters, as well as the parameter values ​​of each parameter. The correspondence is generated by decomposing the data warehouse data retrieval interface code according to the syntax rules of the Structured Query Language (SQL). The data warehouse data retrieval interface is used for callers to invoke after being published. The generation unit is used to: generate a structured object corresponding to the data retrieval module based on the parameter values ​​of the corresponding parameters of the data retrieval module; and generate the data warehouse data retrieval interface based on the structured object.

15. The apparatus according to claim 14, characterized in that, The structured object includes at least one of the following objects: Result set objects, table objects, conditional filter objects, aggregation objects, pagination objects, and sorting objects; The parameters corresponding to the result set object include a list of returned result parameters; the parameters corresponding to the database table object include at least one of the following: database, data table, and database schema; the parameters corresponding to the condition filtering object include at least one of the following: query condition input parameters, class attribute parameters, and calculation parameters; the parameters corresponding to the aggregation object include aggregation parameters; the parameters corresponding to the pagination object include at least one of the following: limit query parameters and offset query parameters; the parameters corresponding to the sorting object include at least one of the following: sorting type parameters and aggregate sorting parameters.

16. The apparatus according to claim 14 or 15, characterized in that, The generation unit is also used for: In the interface runtime environment, publish the data warehouse data retrieval interface.

17. The apparatus according to claim 16, characterized in that, The interface operating environment includes a sandbox environment and an online environment; The generation unit is also used for: In the sandbox environment, publish the data warehouse data retrieval interface; If the data warehouse data retrieval interface test is successful, then the data warehouse data retrieval interface will be published in the online environment.

18. The apparatus according to claim 17, characterized in that, The interface development equipment includes a data gateway; The data gateway is used to receive data retrieval requests sent by other devices, which include the domain name of the data retrieval interface, and to select between the sandbox environment or the online environment based on the environment parameters in the data retrieval request, so as to call the data warehouse data retrieval interface in the selected interface running environment. The domain name of the data retrieval interface is obtained after the data gateway registers an instance.

19. The apparatus according to claim 14, characterized in that, The interface development equipment also includes an SQL adapter; The SQL adapter is used to convert the structured objects in the data warehouse data retrieval interface into SQL statements based on the input parameters in the data retrieval requests sent by other devices.

20. The apparatus according to claim 18, characterized in that, The interface development device also includes a driver routing table; The driver routing table contains the drivers corresponding to each database. The driver program is used to establish a data source connection between the interface development device and the database corresponding to the data retrieval request; The data source connection is used to enable the data warehouse data retrieval interface to retrieve data from the corresponding database.

21. The apparatus according to claim 20, characterized in that, The interface development device also includes a storage management plugin library; the storage management plugin library stores at least one plugin containing a driver. The storage path of the plugin is generated based on the corresponding database type name and data source version number.

22. The apparatus according to claim 20, characterized in that, The drivers for each database are loaded by class loaders; there is a one-to-one correspondence between each driver and each class loader; the class loaders are generated by inheriting the class loader base class and overriding the loadClass device.

23. The apparatus according to claim 20, characterized in that, The interface development equipment also includes HikariCP; HikariCP is used to manage data source connections established for each database.

24. The apparatus according to claim 20, characterized in that, The interface development equipment also includes a database management center; The database management center is used for registering and managing various databases.

25. The apparatus according to claim 16, characterized in that, The generation unit is also used for: Monitor the configuration update information of the data warehouse data retrieval interface; the configuration update information refers to the parameter update information in the data warehouse data retrieval interface. In response to the existence of configuration update information, the data warehouse data retrieval interface is updated based on the configuration update information.

26. The apparatus according to claim 25, characterized in that, The step of updating the data warehouse data retrieval interface based on the configuration update information includes: If the storage root path of the configuration update information is determined to represent the sandbox environment, then the data warehouse data retrieval interface in the sandbox environment is updated. If the storage root path of the configuration update information is determined to represent the online environment, then the data warehouse data retrieval interface in the online environment is updated.

27. An electronic device, characterized in that, include: processor; as well as A memory storing computer instructions for causing the processor to perform the method according to any one of claims 1 to 13.

28. A computer-readable storage medium, characterized in that, The computer contains computer instructions for causing the computer to perform the method according to any one of claims 1 to 13.

Images