Trino-based multi-data-source integrated data service management method, platform, equipment and medium

Abstract

The invention discloses a Trino-based multi-data-source integrated data service management method, platform, equipment and medium, belongs to the technical field of data processing, and aims to solve the technical problem of how to create standardized data services based on different data sources and support real-time monitoring and flow limiting of the data services at the same time. According to the technical scheme, multi-data-source access is carried out, wherein unified access and management are carried out on various data sources including a relational database, a non-relational database, a data warehouse and a big data platform based on a Trino connector, and cross-data-source dynamic query is achieved; data service creation and release: based on the SQL query capability of Trino, providing a standardized data service template, encapsulating data processing logic into reusable data service, and simplifying development and maintenance of the data service; client authentication and authorization: automatically distributing a unique key for each client for subsequent authentication and authorization; and data service monitoring and current limiting: carrying out real-time monitoring and current limiting on the data service through a built-in monitoring component.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data service management method, platform, device and medium based on Trino multi-data source integration. Background Art

[0002] Data source refers to the source that provides data for analysis, processing or storage. With the rapid development of big data technology, the amount of data has increased dramatically and the data sources have become diversified. Enterprises need a platform that can uniformly manage and efficiently process data. Traditional data service management platforms often cannot meet real-time requirements, and are inefficient when processing multiple data sources and have high maintenance costs.

[0003] Therefore, how to create standardized data services based on different data sources while supporting real-time monitoring and flow control of data services is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The technical task of the present invention is to provide a data service management method, platform, device and medium based on Trino's multi-data source integration to solve the problem of how to create standardized data services based on different data sources while supporting real-time monitoring and flow control of data services.

[0005] The technical task of the present invention is achieved in the following way: a data service management method based on Trino multi-data source integration, the method is as follows:

[0006] Access to multiple data sources: Trino-based connectors provide unified access and management for multiple data sources including relational databases, non-relational databases, data warehouses, and big data platforms, enabling dynamic queries across data sources.

[0007] Data service creation and publishing: Based on Trino's SQL query capabilities, it provides standardized data service templates, encapsulates data processing logic into reusable data services, and simplifies the development and maintenance of data services;

[0008] Client authentication and authorization: Automatically assign a unique key to each client for subsequent authentication and authorization;

[0009] Data service monitoring and throttling: Real-time monitoring and throttling of data services are performed through built-in monitoring components.

[0010] As a preferred option, when multiple data sources are accessed, a unified interface is used to shield the read and write differences of various heterogeneous data sources, achieve unified access and management of multiple data sources, and ensure the accuracy and stability of data source access through data source configuration wizards and automated testing tools.

[0011] Preferably, the standardized data service template includes a SQL-based template and a table-based template;

[0012] Among them, SQL templates allow users to write customized SQL query statements and encapsulate complex data processing logic;

[0013] Directly operate on the specified data table based on the table template to simplify the development process of data services. When creating data services, you can also configure the service parameters of query timeout and result cache. After creation, the data service will be published to the unified data service API market for other systems or applications to call, realizing efficient reuse and unified management of interfaces.

[0014] Preferably, for each published data service API, the corresponding person in charge authorizes other clients, and only authorized clients can successfully call the corresponding interface to ensure data security and compliance.

[0015] As a preferred option, data service monitoring and current limiting are as follows:

[0016] Monitor the call status of each data service API in real time, and ensure that the person in charge of the data service API can intuitively obtain the operation status of the interface through graphical reports and real-time alarms, and promptly discover and handle potential problems; the call status of each data service includes key indicators such as the number of calls, response time and error rate;

[0017] Set a flow control-based current limiting strategy, which allows the person in charge of the data service API to set a current limiting setting for the interface based on the actual access traffic and server load, such as QPS limit or concurrent number limit, to ensure the stability and availability of the interface.

[0018] More preferably, the method also uses logging and auditing to track and record the calling history of the interface, providing strong support for troubleshooting.

[0019] A data service management platform based on Trino for multi-data source integration, the platform includes:

[0020] Multi-data source integration module, used to achieve unified connection and management of multiple data sources based on Trino connectors;

[0021] Data service creation and management module, which is used to create data service APIs based on standard data service templates and Trino's SQL query function, and encapsulate complex data processing logic into reusable API interfaces;

[0022] Data service monitoring module, which is used to monitor and limit the flow of data services in real time based on built-in monitoring components;

[0023] Among them, the data service monitoring module includes:

[0024] The data service monitoring submodule is used to monitor the call status of each data service API in real time, and ensure that the person in charge of the data service API can intuitively obtain the operation status of the interface through graphical reports and real-time alarms, and promptly discover and handle potential problems; the call status of each data service includes key indicators such as the number of calls, response time and error rate;

[0025] The data service current limiting submodule is used to set a current limiting strategy based on flow control, that is, to allow the person in charge of the data service API to set current limiting settings such as QPS limit or concurrent number limit on the interface according to the actual access traffic and server load, to ensure the stability and availability of the interface.

[0026] Preferably, the standardized data service template includes a SQL-based template and a table-based template;

[0027] Among them, SQL templates allow users to write customized SQL query statements and encapsulate complex data processing logic;

[0028] Directly operate on the specified data table based on the table template to simplify the development process of data services. When creating data services, you can also configure the service parameters of query timeout and result cache. After creation, the data service will be published to the unified data service API market for other systems or applications to call, realizing efficient reuse and unified management of interfaces.

[0029] An electronic device comprising: a memory and at least one processor;

[0030] Wherein, the memory stores a computer program;

[0031] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the data service management method based on Trino multi-data source integration as described above.

[0032] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the data service management method based on Trino multi-data source integration as described above.

[0033] The data service management method, platform, device and medium based on Trino multi-data source integration of the present invention have the following advantages:

[0034] (1) The present invention improves the utilization efficiency and flexibility of data services and optimizes data storage and access, meeting the needs of modern enterprises for data service management;

[0035] (ii) The present invention can realize the creation of standardized data services based on different data sources, and support real-time monitoring and flow control of data services;

[0036] (III) The multi-data source integration module of the present invention can shield the read and write differences of various heterogeneous data sources and realize unified access and management of various data sources;

[0037] (iv) The data service creation and management module of the present invention can create a standard reusable API interface to improve query efficiency and reduce resource consumption;

[0038] (V) The data service monitoring module of the present invention can perform real-time monitoring and flow control on data services. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present invention is further described below in conjunction with the accompanying drawings.

[0040] Attached Figure 1 A flowchart of a data service management method for multi-data source integration based on Trino;

[0041] Attached Figure 2 This is an architecture diagram of the data service management platform for multi-data source integration based on Trino. DETAILED DESCRIPTION

[0042] The data service management method, platform, device and medium based on Trino multi-data source integration of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments of the specification.

[0043] Embodiment 1:

[0044] As attached Figure 1 As shown, this embodiment provides a data service management method based on Trino multi-data source integration, the method is as follows:

[0045] S1. Access to multiple data sources: Trino-based connectors provide unified access and management for multiple data sources including relational databases, non-relational databases, data warehouses, and big data platforms, enabling dynamic queries across data sources.

[0046] S2. Data service creation and publishing: Based on Trino's SQL query capabilities, it provides standardized data service templates, encapsulates data processing logic into reusable data services, and simplifies the development and maintenance of data services;

[0047] S3, Client Authentication and Authorization: Automatically assign a unique key to each client for subsequent authentication and authorization;

[0048] S4. Data service monitoring and flow control: Real-time monitoring and flow control of data services are performed through built-in monitoring components.

[0049] When multiple data sources are accessed in step S1 of this embodiment, a unified interface is used to shield the read and write differences of multiple heterogeneous data sources, achieve unified access and management of multiple data sources, and ensure the accuracy and stability of data source access through a data source configuration wizard and automated testing tools.

[0050] The standardized data service template in step S2 of this embodiment includes a SQL-based template and a table-based template;

[0051] Among them, SQL templates allow users to write customized SQL query statements and encapsulate complex data processing logic;

[0052] Directly operate on the specified data table based on the table template to simplify the development process of data services. When creating data services, you can also configure the service parameters of query timeout and result cache. After creation, the data service will be published to the unified data service API market for other systems or applications to call, realizing efficient reuse and unified management of interfaces.

[0053] In this embodiment, for each published data service API, the corresponding person in charge authorizes other clients, and only authorized clients can successfully call the corresponding interface to ensure data security and compliance.

[0054] The data service monitoring and flow limiting in step S4 of this embodiment are specifically as follows:

[0055] S401. Monitor the call status of each data service API in real time, and ensure that the person in charge of the data service API can intuitively obtain the operation status of the interface through graphical reports and real-time alarms, and promptly discover and handle potential problems; the call status of each data service includes key indicators such as the number of calls, response time and error rate;

[0056] S402. Set a flow control-based current limiting strategy, that is, allow the data service API person in charge to set a current limiting setting for the interface such as QPS limit or concurrent number limit according to the actual access traffic and server load, so as to ensure the stability and availability of the interface.

[0057] This embodiment also uses logging and auditing methods to track and record the call history of the interface, providing strong support for problem troubleshooting.

[0058] Embodiment 2:

[0059] This embodiment provides a data service management platform based on Trino's multi-data source integration, which includes:

[0060] Multi-data source integration module, used to achieve unified connection and management of multiple data sources based on Trino connectors;

[0061] Data service creation and management module, which is used to create data service APIs based on standard data service templates and Trino's SQL query function, and encapsulate complex data processing logic into reusable API interfaces;

[0062] Data service monitoring module, which is used to monitor and limit the flow of data services in real time based on built-in monitoring components;

[0063] Among them, the data service monitoring module includes:

[0064] The data service monitoring submodule is used to monitor the call status of each data service API in real time, and ensure that the person in charge of the data service API can intuitively obtain the operation status of the interface through graphical reports and real-time alarms, and promptly discover and handle potential problems; the call status of each data service includes key indicators such as the number of calls, response time and error rate;

[0065] The data service current limiting submodule is used to set a current limiting strategy based on flow control, that is, to allow the person in charge of the data service API to set current limiting settings such as QPS limit or concurrent number limit on the interface according to the actual access traffic and server load, to ensure the stability and availability of the interface.

[0066] The standardized data service templates in this embodiment include SQL-based templates and table-based templates;

[0067] Among them, SQL templates allow users to write customized SQL query statements and encapsulate complex data processing logic;

[0068] Directly operate on the specified data table based on the table template to simplify the development process of data services. When creating data services, you can also configure the service parameters of query timeout and result cache. After creation, the data service will be published to the unified data service API market for other systems or applications to call, realizing efficient reuse and unified management of interfaces.

[0069] Embodiment 3:

[0070] An embodiment of the present invention further provides an electronic device, comprising: a memory and a processor;

[0071] Wherein, the memory stores computer-executable instructions;

[0072] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the data service management method based on Trino multi-data source integration in any embodiment of the present invention.

[0073] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0074] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0075] Embodiment 4:

[0076] This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the data service management method based on Trino multi-data source integration in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0077] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0078] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0079] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0080] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data service management method based on Trino's multi-data source integration, characterized in that: The method is as follows: Access to multiple data sources: Trino-based connectors provide unified access and management for multiple data sources including relational databases, non-relational databases, data warehouses, and big data platforms, enabling dynamic queries across data sources. Data service creation and publishing: Based on Trino's SQL query capabilities, it provides standardized data service templates, encapsulates data processing logic into reusable data services, and simplifies the development and maintenance of data services; Client authentication and authorization: Automatically assign a unique key to each client for subsequent authentication and authorization; Data service monitoring and throttling: Real-time monitoring and throttling of data services are performed through built-in monitoring components.

2. The data service management method based on Trino multi-data source integration according to claim 1 is characterized in that: When multiple data sources are accessed, a unified interface is used to shield the read and write differences of various heterogeneous data sources, achieve unified access and management of multiple data sources, and ensure the accuracy and stability of data source access through the data source configuration wizard and automated testing tools.

3. The data service management method based on Trino multi-data source integration according to claim 1 is characterized in that: Standardized data service templates include SQL-based templates and table-based templates; Among them, SQL templates allow users to write customized SQL query statements and encapsulate complex data processing logic; Directly operate on the specified data table based on the table template to simplify the development process of data services. When creating data services, you can also configure the service parameters of query timeout and result cache. After creation, the data service will be published to the unified data service API market for other systems or applications to call, realizing efficient reuse and unified management of interfaces.

4. The data service management method based on Trino multi-data source integration according to claim 1 is characterized in that: For each published data service API, the corresponding person in charge authorizes other clients. Only authorized clients can successfully call the corresponding interface to ensure data security and compliance.

5. The data service management method based on Trino multi-data source integration according to claim 1 is characterized in that: Data service monitoring and current limiting are as follows: Monitor the call status of each data service API in real time, and ensure that the person in charge of the data service API can intuitively obtain the operation status of the interface through graphical reports and real-time alarms, and promptly discover and handle potential problems; the call status of each data service includes key indicators such as the number of calls, response time and error rate; Set a flow control-based current limiting strategy, which allows the person in charge of the data service API to set a current limiting setting for the interface based on the actual access traffic and server load, such as QPS limit or concurrency limit.

6. The data service management method based on Trino multi-data source integration according to any one of claims 1 to 5, characterized in that: This method also uses logging and auditing to track and record the call history of the interface, providing strong support for troubleshooting.

7. A data service management platform based on Trino's multi-data source integration, characterized in that: The platform includes: Multi-data source integration module, used to achieve unified connection and management of multiple data sources based on Trino connectors; Data service creation and management module, which is used to create data service APIs based on standard data service templates and Trino's SQL query function, and encapsulate complex data processing logic into reusable API interfaces; Data service monitoring module, which is used to monitor and limit the flow of data services in real time based on built-in monitoring components; Among them, the data service monitoring module includes: The data service monitoring submodule is used to monitor the call status of each data service API in real time, and ensure that the person in charge of the data service API can intuitively obtain the operation status of the interface through graphical reports and real-time alarms, and promptly discover and handle potential problems; the call status of each data service includes key indicators such as the number of calls, response time and error rate; The data service current limiting submodule is used to set a current limiting strategy based on flow control, that is, to allow the person in charge of the data service API to set current limiting settings for the interface such as QPS limit or concurrent number limit according to the actual access traffic and server load.

8. The data service management platform based on Trino multi-data source integration according to claim 7, characterized in that: Standardized data service templates include SQL-based templates and table-based templates; Among them, SQL templates allow users to write customized SQL query statements and encapsulate complex data processing logic; Directly operate on the specified data table based on the table template to simplify the development process of data services. When creating data services, you can also configure the service parameters of query timeout and result cache. After creation, the data service will be published to the unified data service API market for other systems or applications to call, realizing efficient reuse and unified management of interfaces.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the data service management method based on Trino multi-data source integration as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the data service management method based on Trino multi-data source integration as described in any one of claims 1 to 6.

Images