Online data interface configuration publishing method, device, equipment and medium

Abstract

The invention provides an online data interface configuration publishing method and device, equipment and a medium, and the method comprises the steps: displaying an interface application page through a human-computer interaction interface to configure interface parameters, determining an online configuration table based on the parameters, authorizing to access tenants after the verification is passed, and allocating resources. A rule expression engine, a configuration constraint framework, a micro-service framework and a storage component query engine are used for interface packaging, and finally a workflow scheduling engine is used for issuing, updating and offline the interfaces in batches at regular time. The publishing efficiency and flexibility of the online data interface can be improved.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular, to a method, apparatus, device and medium for configuring and publishing an online data interface. Background Art

[0002] With the booming development of big data technology, various data storage components have emerged like mushrooms after rain, each showing unique advantages in different business scenarios and meeting the increasingly diverse data storage needs of enterprises. Against this background, the importance of data has become increasingly prominent. It is not only the key basis for enterprise decision-making but also the core support for the efficient operation of business systems. Therefore, improving the online data service ability to ensure that data can quickly and accurately support business systems has become an important direction for current technological development.

[0003] Facing the urgent demand of business systems for online data service ability, how to quickly build an efficient and flexible data service supply system based on the characteristics of various data and their related components has become the core ability that enterprises must accelerate to improve. This requires not only a deep understanding of the performance characteristics and application scenarios of different data storage components but also the ability to closely integrate these components with the data service process to form a complete closed-loop from data access, processing, storage to service. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide a method, apparatus, device and medium for configuring and publishing an online data interface, which improves the efficiency and flexibility of publishing the online data interface.

[0005] The technical solution of the embodiments of the present application is implemented as follows:

[0006] In a first aspect, the embodiments of the present application provide a method for configuring and publishing an online data interface, the method including:

[0007] Responding to an online interface application request, displaying an application page through a human-computer interaction interface; wherein, the application page is used to configure interface parameters required for the online interface;

[0008] Responding to a submission request for the application page, determining an online configuration table corresponding to the online interface based on the interface parameters;

[0009] Obtaining an audit result of the online configuration table, if the audit result is passed, authorizing and allocating resources to an access tenant of the online interface, and performing interface configuration encapsulation based on the online configuration table and a preset rule expression engine, configuration constraint framework, microservice framework, and query engines corresponding to different storage components to obtain a target interface;

[0010] Performing workflow scheduling on the target interface.

[0011] In a second aspect, an on-line data interface configuration and release device is further provided in an embodiment of the present application. The device includes:

[0012] A display module, configured to display an application page through a human-computer interaction interface in response to an on-line interface application request; wherein, the application page is used to configure interface parameters required for the on-line interface;

[0013] A determination module, configured to determine an on-line configuration table corresponding to the on-line interface based on the interface parameters in response to a submission request for the application page;

[0014] An encapsulation module, configured to obtain an audit result of the on-line configuration table. If the audit result is passed, authorize and allocate resources to an access tenant of the on-line interface, and perform interface configuration encapsulation based on the on-line configuration table, a preset rule expression engine, a configuration constraint framework, a microservice framework, and query engines corresponding to different storage components to obtain a target interface;

[0015] A scheduling module, configured to perform workflow scheduling on the target interface.

[0016] In a third aspect, an electronic device is further provided in an embodiment of the present application, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to execute the on-line data interface configuration and release method according to any one of the first aspect.

[0017] In a fourth aspect, a computer-readable storage medium is further provided in an embodiment of the present application. A computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, it executes the on-line data interface configuration and release method according to any one of the first aspect.

[0018] The embodiments of the present application have the following beneficial effects:

[0019] Through an intuitive human-computer interaction interface, users can easily configure interface parameters and automatically generate an on-line configuration table based on these parameters. After a strict audit process, once the audit is passed, the system will quickly authorize and allocate resources to the access tenant of the on-line interface. The embodiments of the present application integrate a rule expression engine, a configuration constraint framework, a microservice framework, and query engines for different storage components, realizing efficient configuration encapsulation of the interface, thereby obtaining a target interface with complete functions. Finally, through fine workflow scheduling of the target interface, stable release and timely update of the interface are ensured, greatly optimizing the user experience and improving the overall service quality. Description of the Drawings

[0020] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and thus should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other relevant drawings can also be obtained based on these drawings.

[0021] Figure 1 is a schematic flowchart of steps S101 - S104 provided by an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of the online interface configuration and release principle provided by an embodiment of the present application;

[0023] Figure 3 is a schematic flowchart of steps S301 - S302 provided by an embodiment of the present application;

[0024] Figure 4 is a schematic flowchart of steps S401 - S402 provided by an embodiment of the present application;

[0025] Figure 5 is a schematic flowchart of steps S501 - S503 provided by an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of the encapsulation principle provided by an embodiment of the present application;

[0027] Figure 7 is a schematic structural diagram of an online data interface configuration and release device provided by an embodiment of the present application;

[0028] Figure 8 is a schematic diagram of the composition structure of an electronic device provided by an embodiment of the present application. Specific Embodiments

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purposes of illustration and description, and are not used to limit the protection scope of the present application. Additionally, it should be understood that the schematic drawings are not drawn to the physical scale. The flowcharts used in the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations in the flowchart may not be implemented in sequence, and steps without logical context relationships may be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of the present application.

[0030] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments. Generally, the components of the embodiments of the present application described and illustrated herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0032] In the following description, the terms "first / second / third" are only used to distinguish similar objects and do not represent a specific order for the objects. It is understood that "first / second / third" can be interchanged with a specific order or sequence when allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] It should be noted that the term "including" will be used in the embodiments of the present application to indicate the presence of the features stated thereafter, but does not exclude the addition of other features.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used herein are for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0035] See Figure 1 , Figure 1 is a schematic flowchart of steps S101-S104 of the online data interface configuration publishing method provided by the embodiments of the present application, and will be described in conjunction with Figure 1 the steps S101-S104 shown.

[0036] In step S101, in response to an online interface application request, an application page is displayed through a human-computer interaction interface; wherein, the application page is used to configure the interface parameters required for the online interface.

[0037] Please see Figure 2 , Figure 2 is a schematic diagram of the online interface configuration publishing principle provided by the embodiments of the present application, as Figure 2As shown, when a request for an online interface application is received, the system first displays an application page through a human-computer interaction interface (such as a web page, application interface, etc.). The main function of this application page is to enable users to configure various interface parameters required for the online interface. These parameters include, but are not limited to, interface name, access path, request method (such as GET, POST, etc.), request parameter type, return data type, etc.

[0038] In step S102, in response to a submission request for the application page, an online configuration table corresponding to the online interface is determined based on the interface parameters.

[0039] Here, after the user fills in the information on the application page and submits it, the system will determine a corresponding online configuration table based on the interface parameters input by the user. This online configuration table is the basis for subsequent processes such as interface configuration, review, and release.

[0040] In step S103, the review result of the online configuration table is obtained. If the review result is passed, authorization and resource allocation are performed for the access tenant of the online interface, and interface configuration encapsulation is performed based on the online configuration table and a preset rule expression engine, configuration constraint framework, microservice framework, and query engines corresponding to different storage components to obtain a target interface.

[0041] Here, the system will obtain the review result of the online configuration table. This review process is completed by humans or an automated system, aiming to ensure the correctness, security, and compliance of the interface configuration. If the review result is passed, the system will proceed to the next step; if not, the user needs to reconfigure or modify the configuration.

[0042] After the review is passed, the system will authorize the access tenant of the online interface to ensure that only authorized tenants can access the interface. At the same time, the system will allocate necessary resources for the interface, such as database connections, server resources, etc. Next, the system will perform configuration encapsulation on the interface according to the online configuration table and a preset rule expression engine, configuration constraint framework, microservice framework, and query engines corresponding to different storage components. This process is a key step in converting interface parameters, rules, constraints, etc. into executable interface services. After encapsulation, the obtained target interface already has the ability to process requests and return results.

[0043] In step S104, workflow scheduling is performed on the target interface.

[0044] Finally, the system will perform workflow scheduling on the target interface. This means that the system will automatically schedule the execution timing, execution order, etc. of the interface according to preset workflows or rules. Workflow scheduling ensures the efficient and orderly operation of the interface, and also improves the overall performance and stability of the system.

[0045] The above method realizes the full automation of the interface from application, configuration, review to release and scheduling. This method not only improves the efficiency and accuracy of interface configuration, but also enhances the security and stability of the system.

[0046] In some embodiments, referring to Figure 3 , Figure 3 is a schematic flow chart of steps S301 - S302 provided by an embodiment of the present application. The application page includes at least one interface parameter among the purpose of the interface, access tenant, cluster type, cluster number, library, table, query engine, common query conditions, query fields, processing logic, access user volume, and expected concurrency. Before submitting the application page, the method further includes steps S301 - S302, which will be described in combination with each step.

[0047] In step S301, configuration verification is performed on the interface parameters in the application page. The configuration verification includes at least integrity verification and the configuration constraint framework verification.

[0048] In step S302, if the verification fails, the corresponding item that fails the verification is marked on the application page, and the relevant verification information of this item is displayed.

[0049] Here, the application page is not only used to display and receive the interface parameters input by the user, but also these parameters will participate in the subsequent configuration and release of the interface. The application page includes the following interface parameters:

[0050] Purpose of the interface: Describes the main functions and uses of the interface.

[0051] Access tenant: Specifies which tenants can access the interface.

[0052] Cluster type: Indicates the type of cluster on which the interface will be deployed (such as physical cluster, cloud cluster, etc.).

[0053] Cluster number: If there are multiple clusters in the system, it is necessary to specify on which cluster the interface will be deployed.

[0054] Library: Specifies which database the interface will access.

[0055] Table: In the specified database, which table the interface will access.

[0056] Query engine: Indicates which query engine the interface will use (such as MySQL query engine, Elasticsearch query engine, etc.).

[0057] Common query conditions: Lists the common query conditions of the interface for subsequent optimization and quick query.

[0058] Query fields: Specify which fields of data the interface will return.

[0059] Processing logic: Describe how the interface processes data and returns results after receiving a request.

[0060] Estimated number of accessing users: Estimate how many users will access this interface.

[0061] Estimated concurrency: Estimate the maximum number of concurrent requests that the interface needs to process.

[0062] Before submitting the application page, the system will perform configuration verification on the interface parameters entered by the user to ensure the correctness and integrity of the parameters. The verification process includes:

[0063] Integrity verification: Check whether all required fields are filled. If a required field is not filled by the user, the verification fails.

[0064] Configuration constraint framework verification: According to the preset configuration constraint framework, check whether the parameters entered by the user conform to specific rules or constraints. For example, some fields can only accept data in a specific format (such as date format, number range, etc.).

[0065] If the verification fails, the system will mark the corresponding item that fails the verification on the application page and display the relevant verification information for that item. This helps users quickly locate the problem and make modifications. For example, if the user enters a non-existent cluster number in the "Cluster Number" field, the system will display an error prompt next to this field, informing the user that "The cluster number does not exist. Please re-enter."

[0066] Through the above method, with the configuration verification and feedback mechanism, the system can ensure that the interface parameters submitted by the user are correct and complete, thereby improving the success rate of interface configuration and release.

[0067] In some embodiments, refer to Figure 4 , Figure 4 is the process schematic diagram of steps S401 - S402 provided by the embodiment of the present application. The response is for the submission request of the application page, and based on the interface parameters, determine the online configuration table corresponding to the online interface, which can be implemented through steps S401 - S402, and will be described in combination with each step.

[0068] In step S401, extract the interface parameters from the application page and perform format conversion processing on the interface parameters.

[0069] In step S402, determine the preset online configuration table template, and fill the interface parameters into the configuration table template to obtain the connection configuration table.

[0070] Here, when the user fills in all the necessary interface parameters on the application page and submits them, the system first needs to extract these parameters from the application page. The extraction process involves parsing the form data entered by the user and converting it into a format recognizable within the system.

[0071] Next, the system will perform format conversion processing on the extracted interface parameters. This is because the parameters entered by the user exist in multiple formats (such as strings, numbers, dates, etc.), and the system needs these parameters to be stored or processed in a specific format internally. For example, if the user enters a date, the system needs to convert it into a string in the "YYYY-MM-DD" format for subsequent processing.

[0072] After obtaining the interface parameters after format conversion, the system needs to determine a preset online configuration table template. This template is a pre-defined data structure used to store all the information required for interface configuration. The template contains multiple fields, and each field corresponds to a specific parameter in the interface configuration.

[0073] Next, the system will fill the format-converted interface parameters into this configuration table template. This process involves assigning parameter values to the corresponding fields in the template and ensuring that each field is filled correctly. If a parameter does not have a corresponding field in the template, or a field in the template is required but the user does not provide the corresponding parameter value, the system will throw an error or prompt the user to make corrections.

[0074] In the above manner, the system can generate a complete online configuration table based on the interface parameters submitted by the user. This configuration table will serve as the basis for subsequent processes such as interface configuration, review, and release, ensuring that the interface can work as expected by the user.

[0075] In some embodiments, the review results at least include interface review results and data review results. The interface review results are used to judge at least one of the interface purpose, cluster type, number of interface users, and expected concurrency volume to determine whether it meets the business requirements; the data review is used to judge at least one of the cluster type, cluster number, library, table, common query conditions, query fields, and processing logic, confirm the library and table information, whether there are existing indexes for common query fields, whether the query fields are included, and whether the processing logic is correct. If a rule expression is included, it will be verified.

[0076] Here, the interface review results mainly focus on whether the business requirements of the interface are met. It will judge the following aspects:

[0077] Interface purpose: Check whether the design of the interface conforms to the business requirements, that is, whether the functions provided by the interface meet the requirements of the business scenario.

[0078] Cluster type: Evaluate which type of cluster is more suitable for deploying the interface to ensure the performance and stability of the interface.

[0079] Number of interface users: Estimate the number of users the interface will serve in order to allocate sufficient resources for the interface and avoid performance bottlenecks.

[0080] Expected concurrency: Evaluate the maximum number of concurrent requests the interface needs to handle to ensure that the system can still operate stably under high concurrency.

[0081] By passing the interface review, the system can ensure that the interface design meets the business requirements and has sufficient performance and stability to support the development of the business.

[0082] The data review results focus more on the correctness of data processing. It will judge the following aspects:

[0083] Cluster type, cluster number: Confirm whether the cluster type and cluster number that the interface will access are correct to ensure the accuracy and security of data access.

[0084] Library, table: Verify whether the database and tables that the interface will access exist and whether there are access permissions.

[0085] Common query conditions, query fields: Check whether the common query conditions are reasonable, whether the query fields contain the required data, and whether these fields already have indexes to improve query efficiency.

[0086] Processing logic: Evaluate whether the processing logic of the interface is correct, including whether operations such as data filtering, conversion, and aggregation meet the business requirements.

[0087] Rule expression verification: If the interface contains rule expressions (such as regular expressions, SQL statements, etc.), then verification is required to ensure the correctness and security of the expressions.

[0088] By passing the data review, the system can ensure that the interface can correctly access and process data, avoiding risks such as data errors or data leakage.

[0089] In some embodiments, refer to Figure 5 , Figure 5 is a schematic flowchart of steps S501 - S503 provided by an embodiment of the present application. The target interface is obtained by performing interface configuration encapsulation based on the online configuration table and a preset rule expression engine, configuration constraint framework, microservice framework, and query engines corresponding to different storage components, and can be implemented through steps S501 - S503, which will be described in combination with each step.

[0090] In step S501, configuration information is read from the online configuration table.

[0091] In step S502, determine the storage components to be invoked and the corresponding query engines based on the configuration information, and implement the interface logic based on the rule expression engine and the configuration constraint framework.

[0092] In step S503, integrate the implemented interface logic into the microservice framework and register it as a callable service to obtain the target interface.

[0093] Here, first, the system reads the configuration information of the interface from the online configuration table. This information includes the basic attributes of the interface (such as interface name, access path, etc.), storage component information (such as database type, table name, etc.), query conditions (such as common query fields, query logic, etc.), and processing logic (such as data conversion, aggregation rules, etc.). This information is the basis for subsequent interface encapsulation.

[0094] Based on the read configuration information, the system determines the storage components to be invoked by the interface and their corresponding query engines. For example, if the interface needs to access a MySQL database, the system will select a MySQL query engine. Then, the system uses the rule expression engine to parse and process the complex logic in the query conditions (such as regular expression matching, condition combination, etc.), and uses the configuration constraint framework to ensure that the interface logic conforms to the preset rules and constraints (such as data format verification, access permission control, etc.).

[0095] During the process of implementing the interface logic, the system generates corresponding SQL statements or query requests according to the configuration information, and invokes the corresponding storage components and query engines to execute these requests. At the same time, the system also processes the query results, and performs operations such as data conversion and aggregation on the data according to the processing logic in the configuration information to meet the business requirements.

[0096] After implementing the interface logic, the system integrates it into the microservice framework. The microservice framework is an architectural pattern for building and deploying scalable and maintainable microservices. By integrating the interface logic into the microservice framework, the system can register it as a callable service, enabling other services or clients to call this interface through network requests.

[0097] During the process of registering the service, the system assigns a unique identifier to the interface (such as service name, interface path, etc.), and configures the corresponding access permissions and load balancing policies. In this way, when other services or clients initiate requests, the system can find the corresponding interface according to the identifier in the request and call its implementation logic to process the request.

[0098] After the above steps, the system will finally obtain a target interface. This target interface is a service that encapsulates specific business logic and data access capabilities and can be invoked through network requests. It has characteristics such as high availability, scalability, and maintainability, and can meet business requirements and adapt to the development and changes of the business.

[0099] In some embodiments, the storage component includes HBase, Elasticsearch, ClickHouse, Kudu, HDFS / Hive, data lake Paimon / Hudi, StarRocks / Doris, distributed cache Redis, relational database storage Mysql, Oracle, EverDB;

[0100] The query engine includes Phoenix JDBC query encapsulation, HBase API interface encapsulation, Elasticsearch API interface query encapsulation, ClickHouse interface encapsulation, Impala interface encapsulation, Presto / Trino interface encapsulation, StarRocks / Doris general interface encapsulation, Mysql general interface encapsulation, Oracle general interface encapsulation, EverDB general interface encapsulation, distributed cache Redis general interface encapsulation;

[0101] The query engine accesses the corresponding storage component through the distributed cache application;

[0102] The microservice framework is the Spring-Boot microservice framework, and the Spring-Boot microservice framework is used to package the target interface into an executable JAR file.

[0103] The storage component covers various types of data storage systems, including:

[0104] HBase: A distributed and scalable big data storage system built on Hadoop HDFS, suitable for processing large-scale structured data.

[0105] Elasticsearch: A Lucene-based search engine that supports functions such as full-text search, structured search, and real-time analysis.

[0106] ClickHouse: A columnar database management system (DBMS) for online analytical processing (OLAP), suitable for quickly querying and analyzing large amounts of data.

[0107] Kudu: A distributed storage system that focuses on high-throughput and low-latency data access and is suitable for real-time data analysis scenarios.

[0108] HDFS / Hive: The Hadoop Distributed File System (HDFS) and Hive (a data warehouse software) together provide big data storage and query capabilities.

[0109] Data Lake Paimon / Hudi: A data lake is a centralized repository for storing massive amounts of data, supporting various types of analysis; Paimon and Hudi are specific data lake solutions or formats.

[0110] StarRocks / Doris: Both are high-performance, real-time analytical databases suitable for building data warehouses and data marts.

[0111] Distributed Cache Redis: A high-performance distributed in-memory database commonly used in scenarios such as caching and session storage.

[0112] Relational Database Storage Mysql, Oracle, EverDB: These are all widely used relational database management systems that support functions such as transaction processing and data integrity constraints.

[0113] Query engines correspond to the above storage components and provide various interface encapsulation methods, including:

[0114] Phoenix JDBC Query Encapsulation: Java Database Connectivity (JDBC) query encapsulation for accessing HBase.

[0115] HBase API Interface Encapsulation: Directly call the HBase API interface for querying. Elasticsearch API Interface Query Encapsulation: Use the API interface provided by Elasticsearch for querying.

[0116] ClickHouse Interface Encapsulation: Encapsulates the query interface of ClickHouse.

[0117] Impala Interface Encapsulation: Impala is an open-source massively parallel processing (MPP) query engine, and its query interface is encapsulated.

[0118] Presto / Trino Interface Encapsulation: Presto (now Trino) is a distributed SQL query engine, and its query interface is encapsulated.

[0119] StarRocks / Doris General Interface Encapsulation: Encapsulates the general query interfaces of StarRocks and Doris.

[0120] Mysql General Interface Encapsulation: Encapsulates the general query interface of MySQL.

[0121] Oracle General Interface Encapsulation: Encapsulates the general query interfaces of the Oracle database.

[0122] EverDB General Interface Encapsulation: Encapsulates the general query interfaces of EverDB.

[0123] Distributed Cache Redis General Interface Encapsulation: Encapsulates the general query and cache operation interfaces of Redis.

[0124] These query engines execute query operations by means of a distributed cache application (such as Redis) or directly accessing the corresponding storage components. The corresponding relationship between the storage components and the query engines is as Figure 6 shown.

[0125] Interface applicants can select appropriate storage types and query engines according to storage characteristics and interface requirements;

[0126] Please continue to refer to Figure 6 . Whether to add a cache can be selected for the interface configuration. By setting the cache, the storage IO pressure and query performance can be reduced to a certain extent;

[0127] The development framework is based on the Spring-Boot microservices framework. Spring-Boot is an open-source tool based on the Spring framework, aiming to simplify the development and deployment of Java applications. The interface application can be packaged into an executable JAR file, and developers only need to run a command to start the application, which is more suitable for the development of interface applications; combined with the SpringCloud Gateway service gateway to implement functions such as request routing, permission control, and traffic limiting, and alternatives such as Zuul can be sought according to needs; combined with Spring Cloud Consul to implement service registration and discovery, and cooperate with Pod online and offline operations to achieve seamless rolling updates of container service interfaces, and alternatives such as Eureka can be sought according to needs.

[0128] As a microservices architecture solution based on Spring Boot, Spring Cloud has functions such as configuration management, service discovery, circuit breakers, and intelligent routing, with rich functions, but the encapsulation complexity will be relatively high;

[0129] The application of the distributed cache Redis can be very effective in relieving the pressure on the storage engine and the computing engine in certain scenarios. For example, since reports, dashboards, online interfaces, etc. are generally accessed by different services, and some of the access volumes are relatively large. For example, for the reports viewed by customer service, the reports will be refreshed and accessed simultaneously by different customer services. And currently, the general data ingestion tasks usually have a parameter batch interval of 60s and a batch size of 50,000 / 100,000. Frequent refreshing within a minute will not only cause computing pressure and read pressure on the ClickHouse cluster, but also the returned results are the same. Therefore, the query results will be cached in the Redis cluster, and the cache time is set to 60s, which can effectively relieve the cluster pressure.

[0130] The Alluxio data platform is located between the computing and storage systems. The Alluxio caching mechanism can improve the efficiency of data retrieval, and can provide views for the workloads on the data platform at all stages of the data workflow. Regardless of where the data is located, this platform can provide high-performance data access, simplify data engineering, improve GPU utilization, and reduce cloud computing and storage costs.

[0131] The rule expression engine Aviator is mainly applied in data processing scenarios. For example, since the stored data may not necessarily meet the flexibility of queries and the flexibility of interface return data, combined with rule expressions, flexible rule expressions can be configured by interface applicants for enhancement.

[0132] In some embodiments, the workflow scheduling for the target interface includes:

[0133] Regularly publish, update or take offline the target interface in batches;

[0134] When the target interface is taken offline, the authorized permissions are recycled.

[0135] Here, workflow scheduling refers to the orderly management and execution of a series of tasks or operations to ensure that they are carried out in a predetermined order and schedule. In the scenario of the embodiments of the present application, workflow scheduling is applied to the management of the target interface to achieve batch publishing, updating and taking offline of the interface.

[0136] Publishing: Publish the newly developed or updated interface to the production environment so that it can provide services externally. Usually includes code compilation, packaging, deployment, and related configuration updates.

[0137] Updating: Upgrade or modify the published interface to fix bugs, add new features or optimize performance. The update operation usually needs to be carried out without affecting the existing services.

[0138] Taking offline: Remove the interface from the production environment and stop providing services externally. This is because the interface has been replaced, the business requirements have changed, or there are security risks.

[0139] By performing these operations in batches at regular intervals, it can be ensured that changes to the interface can be made within a controllable time window, reducing the impact on the business and improving work efficiency.

[0140] When the target interface goes offline, the authorized permissions need to be revoked. This is to ensure that after the interface no longer provides services, the relevant access permissions are also revoked in a timely manner to avoid potential security risks.

[0141] In summary, the embodiments of the present application have the following beneficial effects:

[0142] (1) Simplify the interface configuration process: Users can easily configure interface parameters through the human-computer interaction interface without having to deeply understand the underlying technical details. Automatically process the format conversion of interface parameters and generate configuration tables, reducing the complexity and error rate of manual operations.

[0143] (2) Enhance the flexibility and scalability of interface configuration: Support multiple interface parameter configurations to meet diverse business needs. Can flexibly adjust interface configurations according to business changes to achieve rapid iteration and optimization of the interface.

[0144] (3) Improve the accuracy and efficiency of interface auditing: Through the dual guarantees of interface auditing and data auditing, ensure that interface configurations comply with business requirements and data specifications. The automated auditing process reduces the workload of manual auditing and improves the auditing efficiency.

[0145] (4) Achieve efficient encapsulation and release of interfaces: Use technical means such as regular expression engines, configuration constraint frameworks, microservice frameworks, and query engines to achieve rapid encapsulation and integration of interface logic. Support packaging the target interface into an executable JAR file for easy deployment and running in different environments.

[0146] (5) Optimize the workflow scheduling of interfaces: Regularly batch release, update, or take offline interfaces, reducing the impact on the business and improving the stability and reliability of the system. Timely revoke the authorized permissions when the interface goes offline to ensure the security and compliance of the system.

[0147] In summary, this online data interface configuration and release method brings significant effects and advantages to the management and release of data interfaces by simplifying the process, enhancing flexibility, improving auditing efficiency, achieving efficient encapsulation and release, optimizing workflow scheduling, and enhancing user experience and service quality.

[0148] Based on the same inventive concept, an on-line data interface configuration and release device corresponding to the on-line data interface configuration and release method in the first embodiment is further provided in the embodiments of the present application. Since the principle of problem-solving of the device in the embodiments of the present application is similar to the above on-line data interface configuration and release method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0149] As Figure 7 shown, Figure 7 is a schematic structural diagram of an on-line data interface configuration and release device 700 provided in the embodiments of the present application. The on-line data interface configuration and release device 700 includes:

[0150] A display module 701, configured to display an application page through a human-computer interaction interface in response to an on-line interface application request; wherein, the application page is used to configure interface parameters required for the on-line interface;

[0151] A determination module 702, configured to determine an on-line configuration table corresponding to the on-line interface based on the interface parameters in response to a submission request for the application page;

[0152] An encapsulation module 703, configured to obtain an audit result of the on-line configuration table. If the audit result is passed, authorize and allocate resources to the access tenant of the on-line interface, and perform interface configuration encapsulation based on the on-line configuration table, a preset rule expression engine, a configuration constraint framework, a microservice framework, and query engines corresponding to different storage components to obtain a target interface;

[0153] A scheduling module 704, configured to perform workflow scheduling on the target interface.

[0154] Those skilled in the art should understand that Figure 7 the implementation functions of the various units in the on-line data interface configuration and release device 700 shown can be understood with reference to the relevant descriptions of the foregoing on-line data interface configuration and release method. Figure 7 The functions of the various units in the on-line data interface configuration and release device 700 shown can be implemented by a program running on a processor, or can be implemented by specific logic circuits.

[0155] In a possible implementation manner, the application page includes at least one interface parameter among interface purpose, access tenant, cluster type, cluster number, library, table, query engine, common query conditions, query fields, processing logic, access user volume, and expected concurrency volume. Before submitting the application page, the display module 701 further includes:

[0156] Performing configuration verification on the interface parameters in the application page, where the configuration verification includes at least integrity verification and configuration constraint framework verification;

[0157] If the verification fails, mark the corresponding item that fails the verification on the application page and display the relevant verification information for that item.

[0158] In a possible implementation, the determination module 702 responds to a submission request for the application page and determines the online configuration table corresponding to the online interface based on the interface parameters, including:

[0159] Extract the interface parameters from the application page and perform format conversion processing on the interface parameters;

[0160] Determine a preset online configuration table template and fill the interface parameters into the configuration table template to obtain the connection configuration table.

[0161] In a possible implementation, the audit result at least includes an interface audit result and a data audit result. The interface audit result is used to judge at least one of the interface usage, cluster type, interface user volume, and expected concurrency volume to determine whether it meets the business requirements; the data audit is used to judge at least one of the cluster type, cluster number, library, table, common query conditions, query fields, and processing logic, confirm the library and table information, whether there is an index for common query fields, whether the query fields are included, and whether the processing logic is correct. If a rule expression is included, perform verification.

[0162] In a possible implementation, the encapsulation module 703 performs interface configuration encapsulation based on the online configuration table and a preset rule expression engine, configuration constraint framework, microservice framework, and query engines corresponding to different storage components to obtain a target interface, including:

[0163] Read the configuration information from the online configuration table;

[0164] Based on the configuration information, determine the storage components to be called and the corresponding query engines, and implement the interface logic based on the rule expression engine and configuration constraint framework;

[0165] Integrate the implemented interface logic into the microservice framework and register it as a callable service to obtain the target interface.

[0166] In a possible implementation, the storage components include HBase, Elasticsearch, ClickHouse, Kudu, HDFS / Hive, data lake Paimon / Hudi, StarRocks / Doris, distributed cache Redis, relational database storage Mysql, Oracle, EverDB;

[0167] The query engine includes Phoenix JDBC query encapsulation, HBase API interface encapsulation, Elasticsearch API interface query encapsulation, ClickHouse interface encapsulation, Impala interface encapsulation, Presto / Trino interface encapsulation, StarRocks / Doris general interface encapsulation, Mysql general interface encapsulation, Oracle general interface encapsulation, EverDB general interface encapsulation, and distributed cache Redis general interface encapsulation;

[0168] The query engine accesses the corresponding storage components through the distributed cache application;

[0169] The microservice framework is the Spring-Boot microservice framework, and the Spring-Boot microservice framework is used to package the target interface into an executable JAR file.

[0170] In a possible implementation, the scheduling module 704 performs workflow scheduling on the target interface, including:

[0171] Regularly publish, update, or take offline the target interface in batches;

[0172] When the target interface is taken offline, the authorized permissions are recycled.

[0173] The above online data interface configuration publishing device has the following beneficial effects:

[0174] (1) Simplify the interface configuration process: Users can easily configure interface parameters through the human-computer interaction interface without having to deeply understand the underlying technical details. Automatically process the format conversion of interface parameters and generate configuration tables, reducing the complexity and error rate of manual operations.

[0175] (2) Enhance the flexibility and scalability of interface configuration: Support multiple interface parameter configurations to meet diverse business needs. Can flexibly adjust interface configurations according to business changes to achieve rapid iteration and optimization of interfaces.

[0176] (3) Improve the accuracy and efficiency of interface auditing: Through the dual guarantees of interface auditing and data auditing, ensure that interface configurations meet business requirements and data specifications. The automated auditing process reduces the workload of manual auditing and improves the auditing efficiency.

[0177] (4) Achieve efficient encapsulation and publishing of interfaces: Utilize technical means such as rule expression engines, configuration constraint frameworks, microservice frameworks, and query engines to achieve rapid encapsulation and integration of interface logic. Support packaging the target interface into an executable JAR file for easy deployment and operation in different environments.

[0178] (5) Optimize the workflow scheduling of the interface: Regularly batch publish, update, or take offline the interface, reducing the impact on the business and improving the stability and reliability of the system. When the interface is taken offline, the authorized permissions are promptly recycled to ensure the security and compliance of the system.

[0179] In summary, this online data interface configuration and publishing method brings significant effects and advantages to the management and publishing of data interfaces by simplifying the process, enhancing flexibility, improving the review efficiency, achieving efficient encapsulation and publishing, optimizing the workflow scheduling, and enhancing the user experience and service quality.

[0180] As Figure 8 shown, Figure 8 is a schematic structural diagram of the electronic device 800 provided by an embodiment of the present application. The electronic device 800 includes:

[0181] A processor 801, a storage medium 802, and a bus 803. The storage medium 802 stores machine-readable instructions executable by the processor 801. When the electronic device 800 runs, the processor 801 communicates with the storage medium 802 through the bus 803. The processor 801 executes the machine-readable instructions to perform the steps of the online data interface configuration and publishing method described in the embodiments of the present application.

[0182] In actual application, the various components in the electronic device 800 are coupled together through the bus 803. It can be understood that the bus 803 is used to realize the connection and communication between these components. In addition to the data bus, the bus 803 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 8 all the various buses are labeled as the bus 803.

[0183] The above electronic device has the following beneficial effects:

[0184] (1) Simplify the interface configuration process: Users can easily configure interface parameters through the human-computer interaction interface without having to deeply understand the underlying technical details. Automatically process the format conversion of interface parameters and generate configuration tables, reducing the complexity and error rate of manual operations.

[0185] (2) Enhance the flexibility and scalability of interface configuration: Support various interface parameter configurations to meet diverse business needs. Can flexibly adjust the interface configuration according to business changes to achieve rapid iteration and optimization of the interface.

[0186] (3) Improve the accuracy and efficiency of interface review: Through the double guarantee of interface review and data review, ensure that the interface configuration meets business requirements and data specifications. The automated review process reduces the workload of manual review and improves the review efficiency.

[0187] (4) Achieve efficient encapsulation and release of interfaces: Utilize technical means such as regular expression engines, configuration constraint frameworks, microservice frameworks, and query engines to achieve rapid encapsulation and integration of interface logic. Support packaging the target interface into an executable JAR file for convenient deployment and operation in different environments.

[0188] (5) Optimize the workflow scheduling of interfaces: Timed batch release, update, or take offline of interfaces reduces the impact on the business and improves the stability and reliability of the system. When an interface is taken offline, authorized permissions are promptly recycled to ensure the security and compliance of the system.

[0189] In summary, this online data interface configuration and release method brings significant effects and advantages to the management and release of data interfaces by simplifying the process, enhancing flexibility, improving review efficiency, achieving efficient encapsulation and release, optimizing workflow scheduling, and enhancing user experience and service quality.

[0190] The embodiments of this application also provide a computer-readable storage medium storing executable instructions that, when executed by at least one processor 801, implement the online data interface configuration and release method described in the embodiments of this application.

[0191] In some embodiments, the storage medium may be a ferromagnetic random access memory (FRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD ROM), etc.; it may also be various devices including one or any combination of the above memories.

[0192] In some embodiments, the executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0193] As an example, executable instructions may or may not correspond to files in a file system, and may be stored as part of a file that stores other programs or data. For example, they may be stored in one or more scripts in a HyperText Markup Language (HTML) document, stored in a single file dedicated to the program under discussion, or stored in multiple collaborative files (e.g., files that store one or more modules, subroutines, or code segments).

[0194] As an example, executable instructions can be deployed to execute on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed across multiple locations and interconnected by a communication network.

[0195] The above computer-readable storage medium has the following beneficial effects:

[0196] (1) Simplify the interface configuration process: Users can easily configure interface parameters through the human-computer interaction interface without having to deeply understand the underlying technical details. Automatically process the format conversion of interface parameters and generate configuration tables, reducing the complexity and error rate of manual operations.

[0197] (2) Enhance the flexibility and scalability of interface configuration: Support multiple interface parameter configurations to meet diverse business needs. Can flexibly adjust interface configurations according to business changes to achieve rapid iteration and optimization of interfaces.

[0198] (3) Improve the accuracy and efficiency of interface auditing: Through the dual guarantee of interface auditing and data auditing, ensure that interface configurations meet business requirements and data specifications. The automated auditing process reduces the workload of manual auditing and improves auditing efficiency.

[0199] (4) Achieve efficient encapsulation and release of interfaces: Use technical means such as regular expression engines, configuration constraint frameworks, microservice frameworks, and query engines to achieve rapid encapsulation and integration of interface logic. Support packaging the target interface into an executable JAR file for easy deployment and running in different environments.

[0200] (5) Optimize the workflow scheduling of interfaces: Schedule the batch release, update, or offline of interfaces at regular intervals, reducing the impact on the business and improving the stability and reliability of the system. Recover authorized permissions in a timely manner when an interface is taken offline to ensure the security and compliance of the system.

[0201] In summary, this online data interface configuration and release method brings significant effects and advantages to the management and release of data interfaces by simplifying the process, enhancing flexibility, improving auditing efficiency, achieving efficient encapsulation and release, optimizing workflow scheduling, and enhancing user experience and service quality.

[0202] In several embodiments provided by the present application, it should be understood that the disclosed methods and electronic devices can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed with each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0203] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0204] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0205] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0206] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for publishing online data interface configuration, characterized in that: The method comprises: In response to the online interface application request, an application page is displayed through the human-computer interaction interface; wherein the application page is used to configure the interface parameters required for the online interface; In response to a submission request for the application page, determining an online configuration table corresponding to the online interface based on the interface parameters; Obtaining the audit result of the online configuration table, if the audit result is passed, authorizing and allocating resources to the tenant accessing the online interface, and performing interface configuration encapsulation based on the online configuration table and a preset rule expression engine, a configuration constraint framework, a microservice framework, and query engines corresponding to different storage components to obtain a target interface; Performing workflow scheduling on the target interface.

2. The method according to claim 1, characterized in that The application page includes at least one interface parameter of interface purpose, access tenant, cluster type, cluster number, library, table, query engine, common query conditions, query fields, processing logic, access user quantity, and estimated concurrent quantity. Before submitting the application page, the method further includes: Performing configuration verification on the interface parameters in the application page, wherein the configuration verification at least includes integrity verification and configuration constraint framework verification; If the verification fails, the corresponding item that fails the verification is marked on the application page, and the relevant verification information of the item is displayed.

3. The method according to claim 1, characterized in that The responding to the submission request of the application page, determining the online configuration table corresponding to the online interface based on the interface parameter, includes: Extracting the interface parameters from the application page and performing format conversion processing on the interface parameters; A preset online configuration table template is determined, and the interface parameters are filled into the configuration table template to obtain the connection configuration table.

4. The method according to claim 1, characterized in that: The audit results include at least interface audit results and data audit results. The interface audit results are used to judge at least one of the interface purpose, cluster type, interface user volume, and expected concurrency to determine whether business needs are met; the data audit is used to judge at least one of the cluster type, cluster number, library, table, common query conditions, query fields, and processing logic, confirm library and table information, whether common query fields have indexes, whether query fields are included, whether the processing logic is correct, and verify if regular expressions are included.

5. The method according to claim 1, characterized in that The interface configuration encapsulation based on the online configuration table and the preset rule expression engine, configuration constraint framework, microservice framework, and query engines corresponding to different storage components to obtain the target interface includes: Read configuration information from the online configuration table; Determine the storage component and the corresponding query engine to be called based on the configuration information, and implement the interface logic based on the rule expression engine and the configuration constraint framework; The implemented interface logic is integrated into the microservice framework, registered as a callable service, and the target interface is obtained.

6. The method according to claim 1, characterized in that The storage components include HBase, Elasticsearch, ClickHouse, Kudu, HDFS / Hive, data lake Paimon / Hudi, StarRocks / Doris, distributed cache Redis, relational database storage Mysql, Oracle, EverDB; The query engine includes Phoenix JDBC query encapsulation, HBase API interface encapsulation, Elasticsearch API interface query encapsulation, ClickHouse interface encapsulation, Impala interface encapsulation, Presto / Trino interface encapsulation, StarRocks / Doris general interface encapsulation, Mysql general interface encapsulation, Oracle general interface encapsulation, EverDB general interface encapsulation, and distributed cache Redis general interface encapsulation; The query engine accesses the corresponding storage component through a distributed cache application; The microservice framework is a Spring-Boot microservice framework, and the Spring-Boot microservice framework is used to package the target interface into an executable JAR file.

7. The method according to claim 1, characterized in that The performing workflow scheduling on the target interface includes: Regularly publish, update or take offline the target interfaces in batches; When the target interface goes offline, the authorized permissions are reclaimed.

8. An online data interface configuration publishing device, characterized in that: The device comprises: A display module, used to respond to an online interface application request and display an application page through a human-computer interaction interface; wherein the application page is used to configure interface parameters required for the online interface; A determination module, configured to respond to a submission request for the application page and determine an online configuration table corresponding to the online interface based on the interface parameters; The encapsulation module is used to obtain the audit result of the online configuration table. If the audit result is passed, the access tenant of the online interface is authorized and resources are allocated, and the interface configuration encapsulation is performed based on the online configuration table and the preset rule expression engine, configuration constraint framework, microservice framework, and query engines corresponding to different storage components to obtain the target interface; The scheduling module is used to schedule the workflow of the target interface.

9. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to execute the online data interface configuration publishing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the online data interface configuration publishing method according to any one of claims 1 to 7 is executed.

Images