Query method and system for multiple non-relational databases

Abstract

The invention discloses a query method and system for multiple non-relational databases, and the method aims at the query method and system for the multiple non-relational databases, and comprises the steps: analyzing an obtained query statement in a statement analysis layer; different query requests for different non-relational databases are generated based on the analysis result; and sending different query requests to a data access interface, so as to communicate with the non-relational database corresponding to each query request through the unified data access interface of the data access layer to obtain a query result. Through the unified query statement, joint query and calculation of data in a plurality of NoSQL databases are realized, and the problems of non-unified query languages, low data processing efficiency and the like of oSQL databases are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of database query, and in particular to a query method and system for multiple non-relational databases. Background Art

[0002] NoSQL refers to non-relational databases. NoSQL databases were created to address the challenges brought about by large-scale data sets with multiple data types, especially the difficulties in big data applications.

[0003] In the related art, when implementing queries on different types of NoSQL databases, it is necessary to implement the queries for different types of databases respectively, which results in low query efficiency and consumes a lot of resources. Summary of the invention

[0004] The main purpose of the present invention is to provide a query method and system for multiple non-relational databases to solve the technical problems existing in the related technologies.

[0005] In order to achieve the above-mentioned purpose, according to a first aspect of the present invention, a method is provided, which includes parsing the obtained query statement in the statement parsing layer, and generating different query requests for different non-relational databases based on the parsing results; sending the different query requests to the data access interface, so as to communicate with the non-relational databases corresponding to each query request through the unified data access interface of the data access layer to obtain the query results.

[0006] Optionally, the method further comprises invoking a computing engine to process the query result to obtain a result set corresponding to the query statement.

[0007] Optionally, after obtaining the query statement, the query statement is parsed, and different query requests for different non-relational databases are generated based on the parsing results, including: splitting the character string in the query statement into multiple tokens, and parsing each token into a corresponding expression node; encapsulating the expression node into a syntax abstract tree; traversing the syntax abstract tree, and classifying different nodes to generate a final node tree; converting the nodes in the node tree into Java classes, and loading and compiling the Java classes to generate instance objects, wherein each node corresponds to its own Java class generation method.

[0008] Optionally, classifying different nodes to generate a final node tree includes: classifying Select nodes, flow table nodes, Join nodes, and Where nodes to generate a final node tree.

[0009] According to a second aspect of the present invention, a query system for multiple non-relational databases is provided, including a parsing layer, in which the query statements obtained are parsed, and different query requests for different non-relational databases are generated based on the parsing results; a data access layer, which communicates with the non-relational databases corresponding to each query request through a unified data access interface to obtain the query results.

[0010] Optionally, the system further includes a computing engine for processing the query result to obtain a result set corresponding to the query statement.

[0011] Optionally, after obtaining the query statement, the query statement is parsed, and different query requests for different non-relational databases are generated based on the parsing results, including: splitting the character string in the query statement into multiple tokens, and parsing each token into a corresponding expression node; encapsulating the expression node into a syntax abstract tree; traversing the syntax abstract tree, and classifying different nodes to generate a final node tree; converting the nodes in the node tree into Java classes, and loading and compiling the Java classes to generate instance objects, wherein each node corresponds to its own Java class generation method.

[0012] Optionally, classifying different nodes to generate a final node tree includes: classifying Select nodes, flow table nodes, Join nodes, and Where nodes to generate a final node tree.

[0013] According to a third aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute any one of the methods described in the first aspect.

[0014] According to a fourth aspect of the present invention, there is provided an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the method described in any one implementation of the first aspect.

[0015] This embodiment is a query method and system for multiple non-relational databases, wherein the method includes parsing the query statements obtained at the statement parsing layer, and generating different query requests for different non-relational databases based on the parsing results; sending different query requests to the data access interface, so as to communicate with the non-relational databases corresponding to each query request through the unified data access interface of the data access layer to obtain the query results. Through the unified query statement, the joint query and calculation of data in multiple NoSQL databases are realized, solving the problems of inconsistent query language of oSQL databases and low data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is a flow chart of a query method for multiple non-relational databases according to an embodiment of the present invention;

[0018] Figure 2 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0020] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] According to an embodiment of the present invention, a query method for multiple non-relational databases is provided, such as Figure 1 As shown, it includes the following steps 101 to 103:

[0023] Step 101: Parse the acquired query statement at the statement parsing layer, and generate different query requests for different non-relational databases based on the parsing results.

[0024] In this step, the query language is designed in advance. Users can write query statements according to the specified syntax of the query language, that is, the query statements are written according to the specified syntax. The syntax structure is basically compatible with the MySQL syntax. When jointly querying different nosql databases, the left join idea is used for association. The syntax structure is as follows, which includes:

[0025] Select module, flow table module, association table module, where condition module, among which, the Select module table uses the computing engine to execute the select module according to the data returned by the flow table or association module to obtain the final output data.

[0026] Stream table module: The full name attribute in the stream table module is used to trigger whether the fql task is executed, and it also represents a dynamic table in the real-time stream. To some extent, the difference from the table in the database is that the stream table data does not need to be obtained through the association module, but directly obtained from the real-time stream. This is the most efficient way to associate data because it is already in memory.

[0027] where module: defines the logical filtering function of the entire fql.

[0028] Association module: defines the function of obtaining data from which database in what way. For example, it can be defined through the joinType field, as shown in the following example:

[0029] :ES JOIN returns unique data based on es primary key query

[0030] |ES AGGR JOIN queries based on conditions and uses ES aggregation functions to aggregate within ES and return unique data

[0031] |ES UNIQUE JOIN queries based on conditions and returns unique data;

[0032] |ES LIST JOIN returns an array object based on the query conditions;

[0033] |REDIS JOIN pattern key to redis hash structure data. Return unique data;

[0034] |HBASE JOIN queries hbase based on the primary key and returns unique data;

[0035] |HBASE PREFIX JOIN returns an array object based on the prefix query;

[0036] |MEMEORY JOIN joins unique data from memory.

[0037] Furthermore, a Where condition module may be included to implement logic verification.

[0038] In this step, the categories of different non-relational databases include but are not limited to lasticsearch, HBase, Redis, etc.

[0039] As an optional implementation method of this embodiment, after obtaining the query statement, the query statement is parsed, and different query requests for different non-relational databases are generated based on the parsing results, including: splitting the string in the query statement into multiple tokens, and parsing each token into a corresponding expression node; encapsulating the expression node into a syntax abstract tree; traversing the syntax abstract tree, and classifying different nodes to generate a final node tree; converting the nodes in the node tree into Java classes, and loading and compiling the Java classes to generate instance objects, wherein each node corresponds to its own Java class generation method.

[0040] In this optional implementation, after reading the query string, it is split into tokens, parsed to generate corresponding expression nodes, and then encapsulated to generate a syntax abstract tree. The visitor pattern is used to traverse the syntax abstract tree and classify it according to the above four modules to generate the final node tree.

[0041] In the node tree, each operation or expression is a node. For example, the association table is an association node. The association node includes the association table node, the association condition node, the optional sorting node, etc. They form a tree in the form of a binary tree. Finally, the node needs to be converted into a physical execution plan (Java class). Each node has its own code generation method. Finally, you only need to call the node's code generation method to get the final code (Java class).

[0042] Furthermore, the Java class is loaded and compiled to generate instance objects, which are cached in the memory. Once the corresponding event flow triggers the corresponding instance, the corresponding logic will be executed to obtain the final desired result.

[0043] As an optional implementation of this embodiment, classifying different nodes to generate the final node tree includes: classifying Select nodes, flow table nodes, Join nodes, and Where nodes to generate the final node tree.

[0044] In this optional implementation, the Select node is a required node, which mainly defines the execution logic, such as the selected fields and the logical results after various functions are applied. It expresses the final output result. The stream table node is a required node, which is generated for real-time processing. The event that triggers the current execution in real-time calculation is the stream table. The stream is encapsulated in the required format upstream, including the required data and related attributes to help calculation. The Join node is not a required node. Only when the data in the stream table is not sufficient to meet the logical needs, it may be necessary to associate data from the library. At this time, the Join node will be generated. The association node includes the association table node, the association condition node, and the optional sorting node. The Where node is not a required node to implement logical verification of the calculation logic.

[0045] Step 102: Send different query requests to the data access interface to communicate with the non-relational database corresponding to each query request through the unified data access interface of the data access layer to obtain the query result.

[0046] In this step, the data access layer is responsible for communicating with different NoSQL databases to obtain query results. A unified interface layer is provided in the data access layer to support query operations on multiple databases. In order to ensure flexibility and scalability, this layer not only covers the current mainstream database query interface, but also has an extension mechanism designed to easily integrate support for other data systems.

[0047] As an example, there is a waybill information table in the real-time stream, as shown in Table 1. Now the waybill information needs to be expanded. The user information needs to be obtained from the user table, as shown in Table 2, and the user's department information needs to be obtained from the department table, as shown in Table 2. The user table is stored in es, and the department information is stored in redis.

[0048] Table 1

[0049]

[0050] Table 2

[0051] User table (Es) User_id User_name Depart_id

[0052] Table 3

[0053] Department table (Redis) Depart_id Depart_name Additional Information

[0054] The following example briefly illustrates how to query data from different data sources simultaneously in a flow:

[0055] Select module:

[0056] select

[0057] a.order_id,

[0058] a.user_id,

[0059] b.user_name,

[0060] b.depart_id,

[0061] c.depart_name,

[0062] a.other_info,

[0063] Stream table module, where order_info is the current data in the stream. This data triggers the execution of this instance:

[0064] from order_info a,

[0065] Association table module, es joinuser_info: tells the fql engine to get a piece of data from the es user_info index using the user_id as the primary key. redis joindepart_info: tells the fql engine to get department information from the redis hash structure based on the department id:

[0066] es join user_info b on a.user_id=b.user_id,

[0067] redis join depart_info c on b.depart_id=c.depart_id,

[0068] Where condition module: Where a.statues>3and b.status=1. The meaning is: where module, logical filtering function, means that only when the corresponding status meets the condition, the corresponding task will be executed.

[0069] As an optional implementation manner of this embodiment, the method further includes calling a computing engine to process the query result to obtain a result set corresponding to the query statement.

[0070] In this optional implementation, after querying the results of different non-relational databases, based on the data integration and conversion mechanism, the data structures and data formats of different NoSQL databases are automatically identified and parsed, and converted into a unified internal data representation, which is to shield the data formats returned by the different underlying NoSQL databases from the computing engine and unify them into a row format, which is a truct structure containing field names, types, notes, and values, reducing the complexity of the preprocessing of the computing engine. Through the computing engine, using preset data conversion functions and operators, users are supported to perform various calculations and conversion operations on the query results to obtain the final result set.

[0071] This embodiment provides a query language that can realize unified query and calculation of data in different NoSQL databases (such as Elasticsearch, HBase, Redis, etc.). In this way, users only need to write one query statement to realize joint query and calculation of data in multiple NoSQL databases, which greatly improves data processing efficiency and convenience. It effectively solves the problems of inconsistent query language and low data processing efficiency of traditional NoSQL databases.

[0072] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0073] According to an embodiment of the present invention, a query system for multiple non-relational databases is also provided, including a parsing layer, which parses the acquired query statements at the statement parsing layer and generates different query requests for different non-relational databases based on the parsing results; a data access layer, which communicates with the non-relational databases corresponding to each query request through a unified data access interface to obtain query results.

[0074] As an optional implementation of this embodiment, the system further includes a computing engine for processing the query result to obtain a result set corresponding to the query statement.

[0075] As an optional implementation method of this embodiment, after obtaining the query statement, the query statement is parsed, and different query requests for different non-relational databases are generated based on the parsing results, including: splitting the string in the query statement into multiple tokens, and parsing each token into a corresponding expression node; encapsulating the expression node into a syntax abstract tree; traversing the syntax abstract tree, and classifying different nodes to generate a final node tree; converting the nodes in the node tree into Java classes, and loading and compiling the Java classes to generate instance objects, wherein each node corresponds to its own Java class generation method.

[0076] As an optional implementation of this embodiment, classifying different nodes to generate the final node tree includes: classifying Select nodes, flow table nodes, Join nodes, and Where nodes to generate the final node tree.

[0077] An embodiment of the present invention provides an electronic device, such as Figure 2 As shown, the electronic device includes one or more processors 31 and a memory 32. Figure 2 A processor 31 is taken as an example.

[0078] The controller may further include: an input device 33 and an output device 34 .

[0079] The processor 31, the memory 32, the input device 33 and the output device 34 may be connected via a bus or other means. Figure 2 The example of connecting through bus is taken in the following.

[0080] The processor 31 may be a central processing unit (CPU). The processor 31 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips. A general-purpose processor may be a microprocessor or the processor may be any conventional processor.

[0081] The memory 32 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the control method in the embodiment of the present invention. The processor 31 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 32, that is, the enterprise data exchange method based on email in the above method embodiment is realized.

[0082] The memory 32 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of a processing device operated by the server, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 32 may optionally include a memory remotely arranged relative to the processor 31, and these remote memories may be connected to a network connection device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0083] The input device 33 can receive input digital or character information, and generate key signal input related to user settings and function control of the processing device of the server. The output device 34 can include display devices such as a display screen.

[0084] One or more modules are stored in the memory 32, and when executed by one or more processors 31, the execution is as follows: Figure 1 The method shown.

[0085] Those skilled in the art can understand that the implementation of all or part of the processes in the above-mentioned embodiment method is a program that can be completed by instructing related hardware through a computer program, which can be stored in a computer-readable storage medium. When the program is executed, it can include the process of the embodiment of the above-mentioned method. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0086] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A query method for multiple non-relational databases, characterized in that: include: Parse the query statements obtained at the statement parsing layer, and generate different query requests for different non-relational databases based on the parsing results; Different query requests are sent to the data access interface to communicate with the non-relational database corresponding to each query request through the unified data access interface of the data access layer to obtain the query results.

2. The query method for multiple non-relational databases according to claim 1, characterized in that: The method further includes calling a computing engine to process the query result to obtain a result set corresponding to the query statement.

3. The query method for multiple non-relational databases according to claim 1, characterized in that: After obtaining the query statement, the query statement is parsed, and different query requests for different non-relational databases are generated based on the parsing results, including: Split the string in the query statement into multiple tokens, and parse each token into the corresponding expression node; Encapsulating the expression nodes into a syntax abstract tree; Traversing the syntax abstract tree, and classifying different nodes to generate a final node tree; The nodes in the node tree are converted into Java classes, and the Java classes are loaded and compiled to generate instance objects, wherein each node corresponds to its own Java class generation method.

4. The query method for multiple non-relational databases according to claim 3, characterized in that: The final node tree generated by classifying different nodes includes: The final node tree is generated by classifying the Select nodes, flow table nodes, Join nodes, and Where nodes.

5. A query system for multiple non-relational databases, characterized in that: include: The parsing layer parses the query statements obtained at the statement parsing layer and generates different query requests for different non-relational databases based on the parsing results; The data access layer communicates with the non-relational database corresponding to each query request through a unified data access interface to obtain the query results.

6. The query system for multiple non-relational databases according to claim 5, characterized in that: The system also includes a computing engine that processes the query results to obtain a result set corresponding to the query statement.

7. The query system for multiple non-relational databases according to claim 5, characterized in that: After obtaining the query statement, the query statement is parsed, and different query requests for different non-relational databases are generated based on the parsing results, including: Split the string in the query statement into multiple tokens, and parse each token into the corresponding expression node; Encapsulating the expression nodes into a syntax abstract tree; Traversing the syntax abstract tree, and classifying different nodes to generate a final node tree; The nodes in the node tree are converted into Java classes, and the Java classes are loaded and compiled to generate instance objects, wherein each node corresponds to its own Java class generation method.

8. The query system for multiple non-relational databases according to claim 7, characterized in that: The final node tree generated by classifying different nodes includes: The final node tree is generated by classifying the Select nodes, flow table nodes, Join nodes, and Where nodes.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 4.

10. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the method described in any one of claims 1 to 4.

Images