Index collection method and device, equipment and storage medium

Abstract

The invention provides an index collection method and device, equipment and a storage medium, and the method comprises the steps: obtaining an index ID, loading the service configuration information of an index according to the index ID and an ESQL service, carrying out the grammar analysis of the service configuration information through an ESQL index grammar analyzer, obtaining a grammar analysis result, and carrying out the grammar analysis of the service configuration information according to the grammar analysis result, according to the method, the target result is obtained from the target database, the target result is converted to obtain the index result, the ESQL service is adopted and combined with the ESQL index grammar parser to participate in the determination process of the index result, optimization and upgrading are carried out compared with traditional SQL, and expansibility and compatibility are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a method, device, equipment, and storage medium for collecting metrics. Background Art

[0002] With the continuous development of big data technology, data analysis has become increasingly important. Management needs to assist in decision-making based on data analysis, products need to iterate according to the results of data analysis, operations need to evaluate the effectiveness of activities through data analysis, etc. The use and analysis of data both rely on metrics. Based on the storage requirements of massive data and the efficiency requirements of data processing, big data platforms usually adopt distributed databases. It is difficult to quickly and efficiently respond to the collection and processing of big data metrics proposed by operators in multiple data sources under a distributed database.

[0003] Existing database service processes perform query syntax checks on the input SQL, parse objects such as tables, indexes, and views involved in the statement, and check the names and related structures of these objects against the data dictionary to confirm whether fields, tables, views, etc. exist in the database. However, the traditional SQL execution method is too single, the R & D investment in human resources is high, and the business encapsulation, maintainability, scalability, availability, and compatibility are limited. For example, when switching the database or framework of the system or upgrading the business, a large number of SQL scripts need to be written or modified to meet or apply to different business scenarios. Summary of the Invention

[0004] Embodiments of this application provide a method, device, equipment, and storage medium for collecting metrics to solve at least one problem existing in the related art. The technical solutions are as follows: In a first aspect, an embodiment of this application provides a method for collecting metrics, including: Obtain a metric ID; Load the service configuration information of the metric according to the metric ID and the ESQL service; Perform syntax parsing on the service configuration information through an ESQL metric syntax parser to obtain a syntax parsing result; According to the syntax parsing result, obtain a target result from the target database and convert the target result to obtain a metric result.

[0005] In one implementation, the step of loading the service configuration information of the metric according to the metric ID and the ESQL service includes: In response to the metric ID, start the ESQL service; When starting the ESQL service, load the service configuration information of the metric corresponding to the metric ID through the ESQL service. The service configuration information includes a general SQL script and a plugin configuration.

[0006] In one embodiment, the syntax parsing of the service configuration information by the ESQL metric syntax parser to obtain the syntax parsing result includes: Through the ESQL metric syntax parser, perform priority processing and operator parsing on the general SQL script respectively to obtain the priority processing result and the operator parsing result; Through the ESQL metric syntax parser, perform custom plug-in processing according to the plug-in configuration to determine the custom plug-in processing result.

[0007] In one embodiment, the performing, by the ESQL metric syntax parser, priority processing and operator parsing on the general SQL script respectively to obtain the priority processing result and the operator parsing result includes: Through the ESQL metric syntax parser, identify the priority symbols in the general SQL script to determine the order of each sub-function and the script of the general SQL script, and obtain the priority processing result; Through the ESQL metric syntax parser, perform value assignment, binding, and function statistics on the character information in the general SQL script to obtain the operator parsing result.

[0008] In one embodiment, the performing, by the ESQL metric syntax parser, custom plug-in processing according to the plug-in configuration to determine the custom plug-in processing result includes: Through the ESQL metric syntax parser, identify the custom plug-ins registered by the special identifier recognizer of the plug-in configuration; Call the custom functions built in the custom plug-ins to calculate the custom plug-in processing result.

[0009] In one embodiment, the service configuration information further includes data source information, and the obtaining of the target result from the target database according to the syntax parsing result includes: Convert the syntax parsing result into an executable query statement; Send the query statement to the proxy interface; Through the proxy interface, determine the target execution interface according to the data source information, and use the target execution interface to transmit the query statement to the corresponding database to determine the target result.

[0010] In one embodiment, the service configuration information further includes a data format, and the converting of the target result to obtain the metric result includes: Convert the target result according to the data format to obtain the metric result.

[0011] In a second aspect, an embodiment of the present application provides an index collection device, including: An acquisition module, configured to acquire an index ID; A loading module, configured to load service configuration information of an index according to the index ID and an ESQL service; A parsing module, configured to perform syntax parsing on the service configuration information through an ESQL index syntax parser to obtain a syntax parsing result; A conversion module, configured to obtain a target result from a target database according to the syntax parsing result, and convert the target result to obtain an index result.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory, where instructions are stored in the memory, and the instructions are loaded and executed by the processor to implement the method in any one of the above aspects.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the method in any one of the above aspects is implemented.

[0014] The beneficial effects in the above technical solutions at least include: By acquiring an index ID, loading service configuration information of the index according to the index ID and an ESQL service, performing syntax parsing on the service configuration information through an ESQL index syntax parser to obtain a syntax parsing result, obtaining a target result from a target database according to the syntax parsing result, and converting the target result to obtain an index result, using the ESQL service and combining with the ESQL index syntax parser to participate in the determination process of the index result, the traditional SQL is optimized and upgraded, and the scalability and compatibility are improved.

[0015] The above summary is only for the purpose of the specification and is not intended to be limiting in any way. In addition to the above-described illustrative aspects, embodiments, and features, further aspects, embodiments, and features of the present application will be readily apparent by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In the drawings, unless otherwise specified, the same reference numerals throughout the several views denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed according to the present application and should not be regarded as limiting the scope of the present application.

[0017] Figure 1 It is a schematic flowchart of the steps of an index collection method according to an embodiment of the present application; Figure 2 Schematic flowchart of an index collection method according to an embodiment of the present application; Figure 3 Block diagram of the structure of an index collection device according to an embodiment of the present application; Figure 4 Block diagram of the structure of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0018] In the following, only some exemplary embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature rather than restrictive.

[0019] Referring to Figure 1 , a flowchart of an index collection method according to an embodiment of the present application is shown. The index collection method may at least include steps S100 - S400: S100. Obtain an index ID.

[0020] S200. Load service configuration information of the index according to the index ID and the ESQL service.

[0021] S300. Perform syntax parsing on the service configuration information through an ESQL index syntax parser to obtain a syntax parsing result.

[0022] S400. Obtain a target result from a target database according to the syntax parsing result, and convert the target result to obtain an index result.

[0023] The index collection method according to the embodiment of the present application can be executed by an electronic control unit, a controller, a processor, etc. of a terminal such as a computer, a mobile phone, a tablet, a vehicle-mounted terminal, etc., or can also be executed by a cloud server, for example, executed by a system in the cloud server or the terminal.

[0024] The technical solution of the embodiment of the present application, by obtaining an index ID, loading service configuration information of the index according to the index ID and the ESQL service, performing syntax parsing on the service configuration information through an ESQL index syntax parser to obtain a syntax parsing result, obtaining a target result from a target database according to the syntax parsing result, and converting the target result to obtain an index result, adopts the ESQL service and combines with the ESQL index syntax parser to participate in the process of determining the index result, which is optimized and upgraded compared with the traditional SQL, and improves the scalability and compatibility.

[0025] Such as Figure 2As shown, in one embodiment, the system has an ESQL data metric collection executor, which is responsible for providing a unified metric collection execution entry to the caller. The caller can input a metric ID, and the obtained metric ID is transmitted to the ESQL metric configuration loading service (hereinafter referred to as the ESQL service) of the system.

[0026] In one embodiment, step S200 includes steps S210 - S220: S210. In response to the metric ID, start the ESQL service.

[0027] Optionally, the ESQL service starts the ESQL service in response to the metric ID.

[0028] S220. When starting the ESQL service, load the service configuration information of the metric corresponding to the metric ID through the ESQL service.

[0029] In the embodiments of the present application, when starting the ESQL service, the service configuration information of the metric corresponding to the metric ID is actively loaded through the ESQL service and persistently stored in the cache and the database. Optionally, the service configuration information includes but is not limited to general SQL scripts, plugin configurations, data source information (such as the affiliated database), data formats, etc.

[0030] In one embodiment, step S300 includes steps S310 - S320: S310. Through the ESQL metric syntax parser, perform priority processing (priority judgment) and operator parsing on the general SQL script respectively to obtain the priority processing result and the operator parsing result.

[0031] Optionally, S310 includes steps S3101 and S3102: S3101. Through the ESQL metric syntax parser, identify the priority symbols in the general SQL script, determine the order of each sub - function and the script of the general SQL script, and obtain the priority processing result.

[0032] In the embodiments of the present application, service configuration information is input into an ESQL metric syntax parser. The ESQL metric syntax parser adopts the syntax format of the C-like language and provides a complete set of algorithms to parse, convert, and format the service configuration information. Specifically, the ESQL metric syntax parser identifies the priority symbols in the general SQL script, determines the order of each sub-function and the script in the general SQL script, and obtains the priority processing result, so as to know which sub-functions and scripts are executed first. For example, the priority symbols include operators such as curly braces and parentheses. For the composite expression (a + 1 - b), it is calculated step by step using "priority" and "associativity". Different "priorities" are assigned to different operators, and the operations with higher priorities are executed first, and then the operations with lower priorities are executed. If the priorities are the same, the execution order is determined according to the "associativity". The priority of multiplication and division is higher than that of addition and subtraction. For operations of the same level, they are from left to right, and the content in parentheses is regarded as a whole, etc.

[0033] S3102. Through the ESQL metric syntax parser, assign values, bind, and perform function statistics on the character information in the general SQL script to obtain the operator parsing result.

[0034] Optionally, the character information includes link operators such as AND and OR, and operators such as assignment (a = b), binding (a: param), compound assignment operator (a = a + x), increment (++), decrement (--), function statistics, etc. Through the ESQL metric syntax parser, assign values, bind, and perform function statistics on the character information in the general SQL script, so as to obtain the operator parsing result.

[0035] S320. Through the ESQL metric syntax parser, perform custom plug-in processing according to the plug-in configuration to determine the custom plug-in processing result.

[0036] Optionally, S320 includes steps S3201 - S3202: S3201. Through the ESQL metric syntax parser, identify the registered custom plug-ins according to the special identifier in the plug-in configuration.

[0037] Optionally, through the ESQL metric syntax parser, identify the already registered custom plug-ins according to the special identifier in the plug-in configuration. It should be noted that the custom plug-ins can include one or several plug-ins.

[0038] S3202. Invoke the custom functions built in the custom plug-ins to calculate the custom plug-in processing result.

[0039] Optionally, the custom function built into the custom plug-in is called through the ESQL metric syntax parser to calculate the processing result of the custom plug-in based on the function input parameters in the general SQL script. For example, the custom function is the esqlCountSum(sqlparam) statistical function. The specific business data value is statistically calculated according to the statistical function and the function input parameters, which is the processing result of the custom plug-in, and is used as the condition or result of the SQL query statement corresponding to the business metric for output.

[0040] It should be noted that in the embodiments of the present application, the corresponding plug-ins can be registered in advance according to actual needs, and associated with the plug-in configurations corresponding to the metric IDs to meet the development needs of users. Once the metric ID is determined, the registered plug-in can be found according to the plug-in configuration and calculations can be performed using the custom function, improving the extensibility performance.

[0041] In one implementation manner, in step S400, obtaining the target result from the target database according to the syntax parsing result includes steps S410 - S430: S410. Convert the syntax parsing result into an executable query statement.

[0042] Optionally, through the atomic general ESQL executor in the system, the priority processing result, operator parsing result, custom plug-in processing result of the syntax parsing result, and can also include functions (sub-functions) and parameter values in the general SQL script are converted into an executable query plan, such as an (SQL) query statement.

[0043] S420. Send the query statement to the proxy interface.

[0044] Then, the query statement is sent to the proxy interface (interface proxy) to avoid directly accessing the databases of each service and shield the differences in accessing the underlying databases externally.

[0045] S430. Through the proxy interface, determine the target execution interface according to the data source information, and use the target execution interface to transmit the query statement to the corresponding database to determine the target result.

[0046] In the embodiments of the present application, through the proxy interface, the target database and the target execution interface of the target database are determined according to the data source information, such as DWD or ADS. The query statement is transmitted to the corresponding database (i.e., the target database) through the target execution interface for execution to determine the target result.

[0047] In one implementation, the system also provides an ESQL result adaptation service, which offers rich result set formats such as XML, JSON, XLS, DOC, PDF, TXT, binary data stream, etc. When the ESQL result adaptation service receives the target result, it determines the target format to be converted according to the data format. For example, if it is JSON, it will convert the target result to obtain the metric result in JSON format, and the ESQL data metric collection executor outputs the final metric result and returns it to the caller.

[0048] Through the method of the embodiments of the present application, by using the parsing and operation technologies of the general ESQL scripting language, a mechanism and method for dynamically collecting and processing data metrics under a distributed database of a big data platform are provided. It decouples the metric extraction in the big data business operation field from the functions of the big data platform base, supports the dynamic configuration and dynamic execution of the metric system in big data projects, hands over the formulation of complex and variable data metrics to business operation personnel for management, saves a large amount of time and effort, enhances the expansion ability of the platform base, and improves the operation efficiency of the big data platform and the effectiveness of data governance. Secondly, for the existing simple SQL queries, a large amount of personalized code is required to complete the expression of this business scenario, which cannot meet the increasingly complex business system support requirements of the transportation and logistics industry and cannot be flexibly configured. The present application uses ESQL technology to solve the above pain points, meet the requirements of adapting to complex business scenarios and flexible configuration, etc., so that the support for business systems has further improvements in scalability, flexibility, and efficiency.

[0049] Refer to Figure 3 , which shows the structural block diagram of the metric collection device according to an embodiment of the present application. The device may include: An acquisition module, configured to acquire a metric ID; A loading module, configured to load the service configuration information of the metric according to the metric ID and the ESQL service; A parsing module, configured to perform syntax parsing on the service configuration information through an ESQL metric syntax parser to obtain a syntax parsing result; A conversion module, configured to obtain a target result from the target database according to the syntax parsing result and convert the target result to obtain a metric result.

[0050] The functions of the modules in each device of the embodiments of the present application can refer to the corresponding descriptions in the above method, and will not be elaborated here.

[0051] Refer to Figure 4, which shows a structural block diagram of an electronic device according to an embodiment of the present application. The electronic device includes: a memory 310 and a processor 320. Instructions that can run on the processor 320 are stored in the memory 310, and the processor 320 loads and executes the instructions to implement the index collection method in the above embodiment. Among them, the number of the memory 310 and the processor 320 can be one or more.

[0052] In one implementation, the electronic device further includes a communication interface 330, which is used to communicate with external devices and perform data interaction and transmission. If the memory 310, the processor 320, and the communication interface 330 are implemented independently, the memory 310, the processor 320, and the communication interface 330 can be interconnected through a bus and complete communication with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, Figure 4 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0053] Optionally, in specific implementation, if the memory 310, the processor 320, and the communication interface 330 are integrated on a chip, the memory 310, the processor 320, and the communication interface 330 can complete communication with each other through an internal interface.

[0054] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the index collection method provided in the above embodiment.

[0055] An embodiment of the present application further provides a chip, which includes a processor for calling and running instructions stored in a memory from the memory, so that a communication device installed with the chip executes the method provided in the embodiment of the present application.

[0056] An embodiment of the present application further provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, the output interface, the processor, and the memory are connected through an internal connection path. The processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0057] It should be understood that the above-mentioned processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the advanced RISC machines (ARM) architecture.

[0058] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM).

[0059] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another.

[0060] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0061] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" can explicitly or implicitly include at least one of these features. In the description of the present application, "a plurality" means two or more, unless otherwise specifically defined.

[0062] Any process or method description shown in the flowchart or described in other ways herein can be understood to represent a module, segment, or part of code including one or more executable instructions for implementing a specific logical function or process. And the scope of the preferred embodiments of the present application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed.

[0063] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing a logical function, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in connection with these instruction execution systems, apparatus, or devices.

[0064] It should be understood that each part of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the method in the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0065] In addition, each functional unit in various embodiments of the present application can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk, an optical disc, etc.

[0066] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of various changes or substitutions, and these should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for collecting indicators, characterized in that, including: Obtain an indicator ID; Load the service configuration information of the indicator according to the indicator ID and the ESQL service; Perform syntax parsing on the service configuration information through an ESQL indicator syntax parser to obtain a syntax parsing result; Obtain a target result from a target database according to the syntax parsing result, and convert the target result to obtain an indicator result.

2. The index acquisition method according to claim 1, characterized in that: The loading of the service configuration information of the indicator according to the indicator ID and the ESQL service includes: In response to the indicator ID, start the ESQL service; When starting the ESQL service, load the service configuration information of the indicator corresponding to the indicator ID through the ESQL service, and the service configuration information includes a general SQL script and a plugin configuration.

3. The index acquisition method according to claim 2, wherein: The performing of syntax parsing on the service configuration information through an ESQL indicator syntax parser to obtain a syntax parsing result includes: Perform priority processing and operator parsing on the general SQL script respectively through an ESQL indicator syntax parser to obtain a priority processing result and an operator parsing result; Perform custom plugin processing according to the plugin configuration through an ESQL indicator syntax parser to determine a custom plugin processing result.

4. The index acquisition method according to claim 3, characterized in that: The performing of priority processing and operator parsing on the general SQL script respectively through an ESQL indicator syntax parser to obtain a priority processing result and an operator parsing result includes: Identify the priority symbols in the general SQL script through an ESQL indicator syntax parser to determine the order of each sub-function and the script of the general SQL script, and obtain a priority processing result; Perform assignment, binding, and function statistics on the character information in the general SQL script through an ESQL indicator syntax parser to obtain an operator parsing result.

5. The index acquisition method according to claim 3, wherein: The performing of custom plugin processing according to the plugin configuration through an ESQL indicator syntax parser to determine a custom plugin processing result includes: Identify the custom plugins registered according to the special identifier recognizer of the plugin configuration through an ESQL indicator syntax parser; Call the custom functions built in the custom plugins to calculate the custom plugin processing result.

6. The index acquisition method according to any one of claims 1-5, characterized in that: The service configuration information further includes data source information, and the obtaining of the target result from the target database according to the syntax parsing result includes: Convert the syntax parsing result into an executable query statement; Send the query statement to a proxy interface; Through the proxy interface, determine a target execution interface according to the data source information, and use the target execution interface to transmit the query statement to the corresponding database to determine the target result.

7. The index acquisition method according to any one of claims 1-5, characterized in that: The service configuration information further includes a data format, and the converting of the target result to obtain an indicator result includes: Convert the target result according to the data format to obtain an indicator result.

8. An index acquisition device, characterized in that, including: An obtaining module for obtaining an indicator ID; A loading module for loading the service configuration information of the indicator according to the indicator ID and the ESQL service; A parsing module, configured to perform syntax parsing on the service configuration information through an ESQL metric syntax parser to obtain a syntax parsing result; A conversion module, configured to obtain a target result from a target database according to the syntax parsing result, and convert the target result to obtain a metric result.

9. An electronic device, characterized in that, Comprising: A processor and a memory, wherein instructions are stored in the memory, and the instructions are loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, in which a computer program is stored, and when the computer program is executed, the method according to any one of claims 1-7 is implemented.

Images