General data query method and device applied to different databases

By applying common data query methods on different databases, using language recognition tools to dynamically compile filtering rules as expression objects and compile them into actual query conditions, the problem of different database query methods is solved, and operators can quickly and accurately filter internal test users The ability to improve the flexibility and scalability of data filtering.

CN118964420APending Publication Date: 2024-11-15WUHAN JINGCHEN INTELLIGENT IDENTIFICATION TECH CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202410991471.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Different database query methods are different, making it difficult for operators to quickly and accurately filter multiple internal test users under different conditions.

Method used

Provide a general data query method applied to different databases. Through preset filtering rules and language recognition tools, filtering rules are dynamically compiled into expression objects, and compiled into a specific database based on the actual query conditions of user requests to determine whether the user request is Matches the filtering rule.

Benefits of technology

Implement flexible data filtering and rule matching functions. Operators only need to use familiar SQL syntax to describe rules, and the system converts them into actual query operations suitable for different data sources, realizing highly configurable and scalable data filtering methods. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964420A_ABST
    Figure CN118964420A_ABST
Patent Text Reader

Abstract

The invention relates to a general data query method and device applied to different databases, and belongs to the technical field of data query. The method comprises the steps that according to a preset screening condition, a screening rule is configured in the background; dynamically compiling the screening rule into an expression object by adopting a preset language recognition tool; acquiring a user request, and compiling the expression object into an actual query condition of a specific database by adopting a data source based on the user request; and determining whether the user request is matched with the screening rule or not according to the actual query condition. The problem that in the prior art, due to the fact that the query modes of different databases are different, it is difficult for an operator to quickly and accurately screen a plurality of internal test users under different conditions is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data query, and in particular to a universal data query method and device applied to different databases. Background Art

[0002] In order to quickly verify and smoothly launch new functions, a group of users need to be selected for internal testing. Operations personnel hope to filter users entering the internal test based on a variety of conditions, such as filtering by user age group. This type of data is usually stored in the user information database; sometimes it is necessary to filter based on the version of the APP that the user is using. This type of data can only be generated when the user initiates a request, that is, the user's real-time status data is stored in memory; sometimes it is necessary to filter based on whether the user has used a certain hardware. This type of data is stored in a cheaper log database. Different databases have different query methods, and operations personnel cannot understand the usage rules of all databases, so it is difficult to accurately and quickly filter multiple internal test users under different conditions. Summary of the invention

[0003] In view of this, it is necessary to provide a general data query method and device applied to different databases to solve the problem in the prior art that different databases have different query methods, which makes it difficult for operators to quickly and accurately screen multiple internal test users under different conditions.

[0004] In order to solve the above problems, the present invention provides a universal data query method applied to different databases, comprising: Configure the filtering rules in the background according to the preset filtering conditions; Use the preset language recognition tool to dynamically compile the screening rules into expression objects; Obtain user requests, and based on the user requests, use data sources to compile expression objects into actual query conditions for a specific database; Determine whether the user request matches the filtering rules based on the actual query conditions.

[0005] In a possible implementation, the screening rule includes a matching condition and a result definition, and the screening rule is a statement using a SQL-like syntax.

[0006] In a possible implementation, the dynamically compiling the screening rules into expression objects using a preset language recognition tool includes: Antlr4 package is used to parse the filtering rules of SQL-like language to obtain the syntax tree; Generate an expression object based on the parsed syntax tree; The general data query method applied to different databases also includes: The generated expression object is serialized into JSON format and stored in persistent storage.

[0007] In a possible implementation, the expression object includes at least a constant expression, a column name expression, and an operation expression.

[0008] In a possible implementation, compiling the expression object into an actual query condition of a specific database using a data source based on a user request includes: Extract compiled expression objects based on user requests; According to different filtering rules, the corresponding data source plug-in is determined, and the expression object is passed to the compile(AbstractExpr expr) method of the data source plug-in; Based on the compile(AbstractExpr expr) method, all types of expressions are recursively processed to obtain the actual query conditions.

[0009] In a possible implementation, determining whether the user request matches the screening rule according to the actual query condition includes: Determine whether the user request meets the filtering rules based on the execution pair returned by the compile(AbstractExpr expr) method; If it meets the requirements, the function corresponding to the result condition is returned.

[0010] In a possible implementation, the function corresponding to the result condition is to determine whether the user matches.

[0011] In a second aspect, the present invention further provides a general data query device applied to different databases, comprising: The rule configuration module is used to configure the filtering rules in the background according to the preset filtering conditions; A compilation module, used to dynamically compile the screening rules into expression objects using a preset language recognition tool; Obtain user requests, and based on the user requests, use data sources to compile expression objects into actual query conditions for a specific database; Determine whether the user request matches the filtering rules based on the actual query conditions.

[0012] In a third aspect, the present invention further provides an electronic device, comprising: a processor and a memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the general data query method applied to different databases as described above are implemented.

[0013] In a fourth aspect, the present invention also provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the general data query method applied to different databases as described above.

[0014] The beneficial effects of the present invention are as follows: first, according to the preset screening conditions, the screening rules are configured in the background; then, the screening rules are dynamically compiled into expression objects using the preset language recognition tool; and the user request is obtained, and based on the user request, the expression object is compiled into the actual query conditions of a specific database using the data source; through dynamic parsing and compilation and through the data source plug-in architecture to support queries of different types of databases, flexible data screening and rule matching functions are realized. The operator only needs to use the familiar SQL syntax to describe the rules, and the system is responsible for converting them into actual query operations applicable to different data sources, and according to the actual query conditions, it determines whether the user request matches the screening rules, thereby realizing a highly configurable and scalable data screening method. Furthermore, for the operator, only need to learn how to use a filter, can easily filter users, without having to worry about what certain conditions are related to and what methods should be used to judge users. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG1 is a method flow chart of an embodiment of a general data query method applied to different databases provided by the present invention; Figure 2 for Figure 1 A method flow chart of an embodiment of step S102; Figure 3 The present invention provides a general data query method for different databases, a schematic diagram of the structure of an embodiment of the antlr4 package parsing syntax tree; Figure 4 is a schematic diagram of an embodiment of a universal data query device applied to different databases provided by the present invention; Figure 5 It is a schematic diagram of the operating environment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0016] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0017] A specific embodiment of the present invention discloses a general data query method applicable to different databases. Figure 1 ,include: S101. Configure screening rules in the background according to preset screening conditions; S102, using a preset language recognition tool to dynamically compile the screening rules into an expression object; S103, obtaining a user request, and compiling the expression object into an actual query condition of a specific database using a data source based on the user request; S104: Determine whether the user request matches the screening rule based on the actual query condition.

[0018] In this embodiment, first, according to the preset filtering conditions, the filtering rules are configured in the background; then, the preset language recognition tool is used to dynamically compile the filtering rules into expression objects; and the user request is obtained, and based on the user request, the expression object is compiled into the actual query conditions of a specific database using the data source; through dynamic parsing and compilation and through the data source plug-in architecture to support queries of different types of databases, flexible data filtering and rule matching functions are realized. The operator only needs to use the familiar SQL syntax to describe the rules, and the system is responsible for converting them into actual query operations applicable to different data sources, and determining whether the user request matches the filtering rules based on the actual query conditions, thereby realizing a highly configurable and scalable data filtering method. For the operator, only need to learn how to use a filter, can easily filter users, without having to worry about what certain conditions are related to and what methods should be used to judge users.

[0019] It should be noted that the preset filtering conditions are a set of rules or conditions pre-set in the system to determine the system's behavior or results when processing user requests. These conditions may include various logical judgments, data attribute requirements or other constraints, the purpose of which is to filter, process or respond to user requests based on these conditions. Specifically, they include: Logical judgment: For example, a user must meet both condition A and condition B to access a specific function.

[0020] Data attribute requirements: For example, only users who are over 18 years old and have passed email verification can perform certain operations.

[0021] Permission control: For example, only administrator-level users can view or modify specific categories of data.

[0022] Time limits: For example, only requests submitted within a certain time period will be processed.

[0023] Geographic location requirements: For example, only users in a certain region can enjoy a particular service or offer.

[0024] The screening conditions can be customized and adjusted according to the system requirements and business scenarios to ensure that the system meets the expected logic and security requirements when processing user requests.

[0025] Filtering rules are a way to specifically define and configure these preset filtering conditions. Operations personnel or system administrators can define these rules on the system's backend configuration interface to determine what kind of requests meet which conditions and how the system should respond or process when the conditions are met.

[0026] It should be noted that the filter provided to the operator uses a SQL-like language, which can support combinations of multiple query conditions such as greater than, less than, interval, logical operators, etc. It is understandable that other filters can also be used in other embodiments.

[0027] Furthermore, the system is extended through "data source plug-ins" to prepare for new query data sources that are added later. For example, if you need to support MongoDB data sources, you need to write corresponding plug-ins, which are responsible for converting SQL-like filters into MongoDB query statements. One function supports operators to create multiple rules, each rule will call the corresponding plug-in, each rule has its own supported conditions, and operators choose to create appropriate rules.

[0028] In some embodiments, the screening rule includes a matching condition and a result definition, and the screening rule is a statement using a SQL-like syntax.

[0029] In this embodiment, the working mode of "rules" is like that of gateway routing rules, which consists of "matches" and "results". When a user's request arrives, if the "match" is satisfied, the function specified by the user's "result" is returned. The "match" part consists of multiple statements using SQL syntax. The operator only needs to use SQL syntax to describe the expected user range, without worrying about whether the queried field is in the database, in memory, or from the information carried in the user request.

[0030] Furthermore, operators can create multiple rules, each of which can call a specific plug-in. Each plug-in implements query statement conversion compatible with a specific database, so each rule can select the appropriate plug-in to execute the filter as needed.

[0031] In some embodiments, the preset language recognition tool is used to dynamically compile the screening rules into expression objects, see Figure 2 ,include: S201, using Antlr4 package to parse the screening rules of SQL-like language to obtain a syntax tree; S202, generating an expression object according to the parsed syntax tree; The general data query method applied to different databases also includes: S203: Serialize the generated expression object into JSON format and store it in persistent storage.

[0032] In this embodiment, the precompiled expression object refers to the internal data structure obtained after using Antlr4 to parse the SQL screening rules in the configuration phase. These objects are serialized and saved in the system's persistent storage so that actual query operations can be dynamically generated at runtime. The precompiled expression objects include column name expressions, constant expressions, and operation expressions, which describe the structure and logic of the SQL WHERE clause.

[0033] It should be noted that the filter parsing uses the Antlr4 component. Antlr4 (Another Tool for Language Recognition) is a language recognition tool for lexical parsing and grammatical parsing, and has community-maintained grammars of various languages, including SQL, so it can be used out of the box without writing additional grammar files.

[0034] It is understandable that in other embodiments, other parsing components may also be used.

[0035] In a specific embodiment, for example, select * from t_user where appversion>'6.0.0' and (region='zh_CN' or lang='zh_CN') , the syntax tree obtained by parsing with antlr4 package is as follows Figure 3 shown.

[0036] We only need to care about the where part, which is the expr:11 tree. The expr:11 tree is an operation expression that contains the left value expr:9 (appversion='6.0.0'), the operator and, and the right value expr:14 expression; and expr:14 is a subtree consisting of the left value operation expression region='zh_CN', the operator or, and the right value operation expression lang='zh_CN'.

[0037] The definition of the operation expression Operation class is as follows: public class Operation extends AbstractExpr { private String operator; private AbstractExpr left; private AbstractExpr right; } The definition of the column name expression ColumnName class is as follows: public class ColumnName extends AbstractExpr{ private String schemaName; private String tableName; private String columnName; } The definition of the constant expression LiteralValue class is as follows: public class LiteralValue extends AbstractExpr { private String value; private String valueType; } Parse expr:9 into an Operation expression object, whose left value is a column name expression (ColumnName), right value is a constant expression (LiteralValue), and the operator is ">". Finally, the JSON after serialization of expr:11 expression is: { "@type": "Operation", "operator": ">", "left": { "@type": "ColumnName", "schema_name": "", "table_name": "", "column_name": "appversion" }, "right": { "@type": "LiteralValue", "value": "'6.0.0'", "value_type": "string" } } expr:14 is also parsed into an Operation object, whose operator is OR, and both the left and right values ​​are an Operation expression. The parsing of the left and right subtrees is the same as above, and finally two Operation objects are obtained, which are nested into the Operation object corresponding to expr:14. Finally, expr:14 is serialized into JSON: [ { "@type": "Operation", "operator": "or", "left": { "@type": "Operation", "operator": "=", "left": { "@type": "ColumnName", "schema_name": "", "table_name": "", "column_name": "region" }, "right": { "@type": "LiteralValue", "value": "'zh_CN'", "value_type": "string" } }, "right": { "@type": "Operation", "operator": "=", "left": { "@type": "ColumnName", "schema_name": "", "table_name": "", "column_name": "lang" }, "right": { "@type": "LiteralValue", "value": "'zh_CN'", "value_type": "string" } } } ] The final result of the analysis is the Operation object corresponding to expr:11. This object is serialized into JSON and stored in the database together with the rules.

[0038] In some embodiments, the step of compiling the expression object into an actual query condition of a specific database using a data source based on a user request includes: Extract compiled expression objects based on user requests; According to different filtering rules, the corresponding data source plug-in is determined, and the expression object is passed to the compile(AbstractExpr expr) method of the data source plug-in; Based on the compile(AbstractExpr expr) method, all types of expressions are recursively processed to obtain the actual query conditions.

[0039] In this embodiment, the user's real-time status is packaged into a memory object, and the SQL-like filter is dynamically compiled into a query object at runtime, and the user's real-time status is regarded as a "row" to determine whether the user meets the conditions for entering the internal test at this time. The user's existing behavior or log is stored in various databases. The SQL-like filter will be translated into a query statement of the object database at runtime and sent to the corresponding database for query to determine whether the user meets the conditions for entering the internal test.

[0040] It should be noted that each data source plug-in is responsible for converting the filter rules defined in the system into query statements or operations for a specific database. For example, different plug-ins are required to handle different databases (such as in-memory databases, MySQL, MongoDB, etc.), and data source plug-ins are extended to support different types of databases (such as in-memory databases, MySQL, MongoDB, etc.). Each data source plug-in is responsible for converting SQL-like filter rules into query statements for the corresponding database. For example, for MongoDB, the plug-in may create a MongoDB query object, taking the Criteria.where() syntax of Spring Data MongoDB as an example.

[0041] It should be noted that the compile(AbstractExpr expr) method is an abstract method defined in the data source plug-in interface or abstract class. At runtime, the system will select the appropriate plug-in based on the user's operation or request, and call the compile method of the plug-in, passing the pre-compiled expression object as a parameter to this method. The compile(AbstractExpr expr) method recursively processes all types of expressions. For different expression types (such as ColumnName, Operation), different data source plug-ins will have corresponding processing logic. For example, for the Operation operator object, the plug-in may recursively parse the left value and the right value, and then create the corresponding query operation based on the operator (such as the processing of the greater than operator).

[0042] In a specific embodiment, a compile(AbstractExpr expr) method is first defined to process all types of expressions. For example, to process a ColumnName expression, the "Memory Data Source Plugin" creates an AccessFieldOperation object to access the value of a specified field from a memory object; the "MySQL Data Source Plugin" directly outputs a statement fragment in the MySQL dialect, enclosing the field with `; the "MongoDB Data Source Plugin" creates a query object, and in Spring Data MongoDB, it is Criteria.where(field name).

[0043] If the compile(AbstractExpr expr) method encounters an Operation operator object, it needs to recursively call compile(AbstractExpr expr) to parse the left value and the right value respectively, and then process the operator. Taking the greater than operator as an example, the "Memory Data Source Plugin" will create a Gt operation object, which implements Function<Object, Boolean> When performing a match, the object in memory is passed to the Gt object method to obtain a Boolean value; the "MySQL data source plug-in" directly splices a ">" symbol; the "MongoDB data source plug-in" calls the gt method of the Criteria object.

[0044] Since the subtree is parsed recursively, an execution object is obtained after the expression parsing at the top level is returned. Finally, the execution methods of these execution objects are called. For example, the execution object obtained by the "memory data source plug-in" will return a Boolean value after being called, that is, whether the user matches; after the "MySQL data source plug-in" is executed, several rows of results are obtained to determine whether the user matches.

[0045] Based on the above-mentioned general data query method applied to different databases, the embodiment of the present invention also provides a general data query device applied to different databases, please refer to Figure 4 , including: a rule configuration module 410 , a compilation module 420 , an actual query condition compilation module 430 and a query matching module 440 .

[0046] The rule configuration module 410 is used to configure the screening rules in the background according to the preset screening conditions; A compiling module 420, for dynamically compiling the screening rules into expression objects using a preset language recognition tool; The actual query condition compiling module 430 obtains the user request and compiles the expression object into the actual query condition of the specific database based on the user request using the data source; The query matching module 440 determines whether the user request matches the screening rule according to the actual query condition.

[0047] like Figure 5 As shown, based on the above-mentioned general data query method applied to different databases, the present invention also provides an electronic device, which can be a computing electronic device such as a mobile terminal, a desktop computer, a notebook, a palm computer, and a server. The electronic device includes a processor 510, a memory 520, and a display 530. Figure 5 Only some components of the electronic device are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0048] In some embodiments, the memory 520 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. In other embodiments, the memory 520 may also be an external storage electronic device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 520 may also include both an internal storage unit of the electronic device and an external storage electronic device. The memory 520 is used to store application software and various types of data installed in the electronic device, such as program codes for installing the electronic device. The memory 520 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a general data query program 540 applied to different databases is stored on the memory 520, and the general data query program 540 applied to different databases can be executed by the processor 510, thereby realizing the general data query method applied to different databases in each embodiment of the present application.

[0049] In some embodiments, the processor 510 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes stored in the memory 520 or process data, such as executing a general data query method applied to different databases.

[0050] In some embodiments, the display 530 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 530 is used to display information of the general data query electronic device applied to different databases and to display a visual user interface. The components 510-530 of the electronic device communicate with each other via a system bus.

[0051] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0052] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A general data query method applied to different databases, characterized in that: include: Configure the filtering rules in the background according to the preset filtering conditions; Use the preset language recognition tool to dynamically compile the screening rules into expression objects; Obtain user requests, and based on the user requests, use data sources to compile expression objects into actual query conditions for a specific database; Determine whether the user request matches the filtering rules based on the actual query conditions.

2. The universal data query method applied to different databases according to claim 1, characterized in that: The screening rule includes a matching condition and a result definition, and the screening rule is a statement using a SQL-like syntax.

3. The universal data query method applied to different databases according to claim 2, characterized in that: The method of using a preset language recognition tool to dynamically compile the screening rules into an expression object includes: Antlr4 package is used to parse the filtering rules of SQL-like language to obtain the syntax tree; Generate an expression object based on the parsed syntax tree; The general data query method applied to different databases also includes: The generated expression object is serialized into JSON format and stored in persistent storage.

4. The universal data query method applied to different databases according to claim 3, characterized in that: The expression object includes at least a constant expression, a column name expression and an operation expression.

5. The universal data query method applied to different databases according to claim 2, characterized in that: The method of using a data source to compile an expression object into an actual query condition of a specific database based on a user request includes: Extract compiled expression objects based on user requests; According to different filtering rules, the corresponding data source plug-in is determined, and the expression object is passed to the compile(AbstractExpr expr) method of the data source plug-in; Based on the compile(AbstractExpr expr) method, all types of expressions are recursively processed to obtain the actual query conditions.

6. The universal data query method applied to different databases according to claim 5, characterized in that: Determining whether the user request matches the screening rule according to the actual query condition includes: Determine whether the user request meets the filtering rules based on the execution pair returned by the compile(AbstractExpr expr) method; If it meets the requirements, the function corresponding to the result condition is returned.

7. The universal data query method applied to different databases according to claim 6, characterized in that: The function corresponding to the result condition is to determine whether the user matches.

8. A general data query device applied to different databases, characterized in that: include: The rule configuration module is used to configure the filtering rules in the background according to the preset filtering conditions; A compilation module, used to dynamically compile the screening rules into expression objects using a preset language recognition tool; The actual query condition compilation module is used to obtain the user request and compile the expression object into the actual query condition of a specific database based on the user request using the data source; The query matching module is used to determine whether the user request matches the screening rules based on the actual query conditions.

9. An electronic device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the processor implements the steps of the general data query method applied to different databases as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the general data query method applied to different databases as described in any one of claims 1-7.

Citation Information

Patent Citations

  • Query sentence processing device and query sentence processing method

    CN102253984A

  • A cross-data-source SQL compiling device and method

    CN109408580A

  • General DSL query method under RESTful architecture

    CN114356954A

  • Database query method and system, medium and computer

    CN116501763A

  • Unified SQL query method oriented to heterogeneous data sources

    CN117093599A