Data query method, system, storage medium and electronic device

By registering the syntax analysis interface in the Spark system and converting the SQL query statement to Elasticsearch native operators, the problem of poor performance in Spark query Elasticsearch data is solved, and efficient data query and fusion analysis are achieved.

CN113342818BActive Publication Date: 2025-08-05BEIJING XUEZHITU NETWORK TECH
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202110718294.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-28
Publication Date
2025-08-05
Estimated Expiration
2041-06-28

AI Technical Summary

Technical Problem

When using Spark to query Elasticsearch data, the performance of the prior art is not ideal and cannot achieve the query efficiency of the native interface. Especially in the case of massive data, insufficient memory will affect the query analysis performance.

Method used

By registering the syntax analysis interface in the Spark system, obtaining Elasticsearch mapping information, creating temporary tables, and converting the operators of the SQL query statement into operators of the Elasticsearch native query interface, data fusion and analysis query are realized.

Benefits of technology

It realizes the same query performance as the Elasticsearch native interface, reduces the time and technical cost of the ETL process, and supports data query methods for non-professional and technical personnel.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113342818B_ABST
    Figure CN113342818B_ABST
Patent Text Reader

Abstract

This application discloses a data query method, system, storage medium, and electronic device. The data query method includes: a registration step: registering the obtained syntax analysis interface in the Spark system; a mapping information acquisition step: acquiring the mapping information in the Elasticsearch system through the index name in the Spark system; a temporary table creation step: creating a temporary table in the Spark system based on the mapping information; a conversion step: converting the operators in the SQL query statement into query operators in the DSL of the Elasticsearch native query interface; a query step: using the SQL query statement to query the data in the Elasticsearch system; and a fusion step: using the Spark system to fuse the Elasticsearch data with data from other systems, and then performing analysis and query through the Spark system. Using this solution, it is possible to use SQL to query data in the Elasticsearch system based on Spark.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of data query, and in particular relates to a data query method, system, storage medium and electronic equipment. Background Art

[0002] With the growing mobile internet population and the digital transformation of enterprises, mobile devices and IoT devices generate massive amounts of data daily, leading to an increasing demand for data analysis. Data analysis systems rely on database systems. Different data types (structured, semi-structured, text, logs) are stored and analyzed using different database systems, including relational databases, document databases, and text retrieval databases. Over time, an enterprise typically finds that different types of data are scattered across different database systems. These data are closely interconnected and require input into the actual analysis process. The traditional approach involves migrating data from multiple database systems into a big data system (such as Hadoop) according to specific rules, requiring significant time and effort to perform ETL (the process of extracting, transforming, and loading data from the source to the target). This undoubtedly imposes significant time and technical costs on enterprises seeking to rapidly analyze, model, and explore the value of their data.

[0003] Elasticsearch is a distributed search engine database that many companies use for text retrieval and data analysis. If a large amount of enterprise data is scattered across Elasticsearch and other database systems, and both require data retrieval and analysis (for example, Elasticsearch stores factual data while MySQL stores dimensional data), this can become a major pain point for the company, as mentioned above.

[0004] Spark is a distributed computing engine that provides interfaces for connecting to multiple external data sources. Using Spark, you can integrate Elasticsearch with other data sources for analytical queries, eliminating the time-consuming ETL process and saving businesses significant time and technical costs. Furthermore, Spark supports SQL (Structured Query Language), a common standard for databases and data warehouses, and can be used to query data in Elasticsearch systems. This makes it a more user-friendly data query and analysis method for non-technical professionals (product managers, data analysts).

[0005] Currently, there are tools available on the market that use Spark to query Elasticsearch data, but their performance is suboptimal and cannot match the performance of querying Elasticsearch using native interfaces. These tools use Spark to query Elasticsearch by storing all the data from a specific Elasticsearch index in memory at once, filtering and aggregating the data in memory. This approach can lead to memory constraints when querying massive amounts of data. If the memory size is insufficient to store the data, Spark will store the data on disk, severely impacting subsequent query and analysis performance. Summary of the Invention

[0006] The embodiments of the present application provide a data query method, system, storage medium, and electronic device to at least solve the problem that existing data query methods consume a lot of time and energy for data ETL work.

[0007] The present invention provides a data query method, which includes:

[0008] Registration step: register the obtained syntax analysis interface into the Spark system;

[0009] Steps for obtaining mapping information: Obtain mapping information from the Spark system to the Elasticsearch system through the index name;

[0010] Temporary table creation step: creating a temporary table in the Spark system according to the mapping information;

[0011] Conversion step: Convert the operators in the SQL query statement into query operators in the DSL of the Elasticsearch native query interface;

[0012] Query step: Use SQL query statements to query the data in the Elasticsearch system;

[0013] Fusion step: After fusing the Elasticsearch data with data from other systems using the Spark system, analysis and query are performed through the Spark system.

[0014] The above data query method, wherein the registration step includes:

[0015] File generation steps: write syntax analysis to generate files;

[0016] Idiom grammar analysis interface generation step: using the grammar analysis to generate a grammar analysis interface, and registering the grammar analysis interface in the Spark system.

[0017] In the above data query method, the step of obtaining mapping information includes:

[0018] Connecting step: In the Spark system, connect to the Elasticsearch system using the address and port number of the Elasticsearch system;

[0019] Creation step: obtaining the mapping information of a specific index in the Elasticsearch system through the index name.

[0020] In the above data query method, the fusion step includes:

[0021] Mounting step: Mounting the abstract syntax tree specific to the Elasticsearch query to the Join node of the global abstract syntax tree through the Spark system;

[0022] Analysis and query step: After fusing the Elasticsearch data with other system data according to the abstract syntax tree through the Spark system, analysis and query are performed through the Spark system.

[0023] The present invention also provides a data query system, which includes:

[0024] A registration module registers the obtained syntax analysis interface into the Spark system;

[0025] A mapping information acquisition module, wherein the mapping information acquisition module acquires mapping information in the Elasti csearch system through the index name in the Spark system;

[0026] A temporary table creation module, wherein the temporary table creation module creates a temporary table in the Spark system according to the mapping information;

[0027] A conversion module converts operators in SQL query statements into query operators in the DSL of the Elasticsearch native query interface;

[0028] A query module, wherein the query module uses SQL query statements to query data in the Elasticsearch system;

[0029] A fusion module, which uses the Spark system to fuse the Elasticsearch data with other system data, and then performs analysis and query through the Spark system.

[0030] The above data query system, wherein the registration module includes:

[0031] A file generation unit, wherein the file generation unit writes a syntax analysis and generates a file;

[0032] An idiom grammar analysis interface generation unit is used to generate a grammar analysis interface using the grammar analysis, and register the grammar analysis interface in the Spark system.

[0033] In the above data query system, the module for obtaining mapping information includes:

[0034] A connection unit, wherein the connection unit is connected to the Elasticsearch system in the Spark system using the address and port number of the Elasticsearch system;

[0035] A creation unit is configured to obtain the mapping information of a specific index in the Elasticsearch system through the index name.

[0036] The above data query system, wherein the fusion module includes:

[0037] A mounting unit, which mounts the abstract syntax tree specific to the Elasticsearch query to the Join node of the global abstract syntax tree through the Spark system;

[0038] An analysis and query unit is configured to fuse the Elasticsearch data with other system data according to the abstract syntax tree through the Spark system, and then perform analysis and query through the Spark system.

[0039] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the data query methods when executing the computer program.

[0040] The present invention also provides a storage medium storing a computer program, wherein the program implements any of the above-mentioned data query methods when executed by a processor.

[0041] The beneficial effects of the present invention are:

[0042] This invention belongs to the field of query optimization in information retrieval technology. Using this solution, Spark can be used to query data in an Elasticsearch system using SQL. Because operators are pushed down, performance is comparable to that of the native Elasticsearch interface. Furthermore, data in the Elasticsearch system can be combined with data from other database systems for query analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0044] In the attached figure:

[0045] Figure 1 is a flow chart of the data query method of the present invention;

[0046] Figure 2 1 is a step-by-step flow chart of step S1 of the present invention;

[0047] Figure 3 is a step-by-step flow chart of step S2 of the present invention;

[0048] Figure 4 is a step-by-step flow chart of step S6 of the present invention;

[0049] Figure 5 It is a structural diagram of the data query system of the present invention;

[0050] Figure 6 is a framework diagram of an electronic device according to an embodiment of the present invention;

[0051] Figure 7 It is an abstract syntax tree conversion flowchart. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0053] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0054] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0055] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0056] The present invention is described in detail below with reference to the various embodiments shown in the accompanying drawings, but it should be noted that these embodiments are not limitations of the present invention, and any equivalent transformations or substitutions in functions, methods, or structures made by ordinary technicians in this field based on these embodiments are all within the scope of protection of the present invention.

[0057] Before elaborating on various embodiments of the present invention, the core inventive concept of the present invention is summarized and elaborated in detail through the following several embodiments.

[0058] Example 1:

[0059] Please refer to Figure 1 , Figure 1 This is a flow chart of the data query method. Figure 1 As shown, the data query method of the present invention includes:

[0060] Registration step S1: registering the obtained syntax analysis interface into the Spark system.

[0061] Specifically, when writing a syntax analysis generation file, in addition to regular SQL operators, it is also necessary to support MATCH (for matching sentences), MATCH_PHRASE (for matching phrases), and OFFSET (for paging) operators; use syntax analysis to generate a syntax analysis interface for analyzing the abstract syntax tree; and register the syntax analysis interface with the Spark system.

[0062] Obtaining mapping information step S2: Obtain mapping information in the Elasticsearch system through the index name in the Spark system.

[0063] Specifically, in the Spark system, use the Elasticsearch address and port number to connect to the Elasticsearch system, and obtain the mapping information of a specific index in Elasticsearch through the index name, including fields, field types, index analyzer types, query analyzer types, and so on.

[0064] Temporary table creation step S3: creating a temporary table in the Spark system according to the mapping information.

[0065] Specifically, after obtaining the mapping information for a specific index from Elasticsearch, Spark uses this information to create a temporary table. During this process, text fields in the Elasticsearch index are mapped to string types in Spark, and nested fields in the Elasticsearch index are mapped to nested types in Spark.

[0066] Conversion step S4: convert the operators in the SQL query statement into query operators of the DSL in the Elasticsearch native query interface.

[0067] Specifically, the operators in the SQL query statement are converted into query operators in the DSL (domain-specific language) in the Elasticsearch native query interface, which includes:

[0068] The Match operator in the SQL query statement is pushed down to Elasticsearch as the match operator in the Elasticsearch query interface DSL;

[0069] The MatchPhrase operator in the SQL query statement is pushed down to Elasticsearch as the match_phrase operator in the Elasticsearch query interface DSL;

[0070] Implemented parsing of the Offset keyword in SQL query statements, pushing it down to Elasticsearch and using it as the from operator in the Elasticsearch query interface DSL to implement paging.

[0071] The Limit operator in the SQL query statement is pushed down to Elasticsearch as the size operator in the Elasticsearch query interface DSL;

[0072] The Like operator in the SQL query statement is pushed down to Elasticsearch as a wildcard operator in the Elasticsearch query interface DSL;

[0073] The Rlike operator in the SQL query statement is pushed down to Elasticsearch as the regexp operator in the Elasticsearch query interface DSL;

[0074] The And and Or operators in the SQL query are pushed down to Elasticsearch as bool operators in the Elasticsearch query interface DSL;

[0075] The EqualTo operator in the SQL query statement is pushed down to Elasticsearch as the term operator in the Elasticsearch query interface DSL;

[0076] The IN operator in the SQL query statement is pushed down to Elasticsearch as the TERMS operator in the Elasticsearch query interface DSL;

[0077] The LessThan and GreaterThan operators in the SQL query are pushed down to Elasticsearch as range operators in the Elasticsearch query interface DSL;

[0078] The Sort operator in the SQL query statement is pushed down to Elasticsearch as the sort operator in the Elasticsearch query interface DSL;

[0079] The Aggregate operator in the SQL query statement is pushed down to Elasticsearch as the agg operator in the Elasticsearch query interface DSL.

[0080] Query step S5: using SQL query statements to query the data in the Elasticsearch system;

[0081] Fusion step S6: After fusing the Elasticsearch data with data from other systems using the Spark system, analysis and query are performed through the Spark system.

[0082] Specifically, by leveraging Spark's data fusion capabilities, the abstract syntax tree specific to the Elasticsearch query is mounted under the Join node of the global abstract syntax tree, enabling fusion analysis and query of Elasticsearch data and other system data.

[0083] Please refer to Figure 2 , Figure 2 is a flowchart of the registration step S1. Figure 2 As shown, the registration step S1 includes:

[0084] File generation step S11: write syntax analysis to generate a file;

[0085] Idiom grammar analysis interface generation step S12: using the grammar analysis to generate a grammar analysis interface, and registering the grammar analysis interface in the Spark system.

[0086] Please refer to Figure 3 , Figure 3 This is a flowchart of step S2 of obtaining mapping information. Figure 3 As shown, the step S2 of obtaining mapping information includes:

[0087] Connecting step S21: In the Spark system, connect to the Elasticsearch system using the address and port number of the Elasticsearch system;

[0088] Creation step S22: obtaining the mapping information of the specific index in the Elasticsearch system through the index name.

[0089] Please refer to Figure 4 , Figure 4 This is the flow chart of the fusion S6. Figure 4 As shown, the fusion step S6 includes:

[0090] Mounting step S61: Mounting the abstract syntax tree specific to the Elasticsearch query to the Join node of the global abstract syntax tree through the Spark system;

[0091] Analysis and query step S62: After fusing the Elasticsearch data and other system data according to the abstract syntax tree through the Spark system, analysis and query are performed through the Spark system.

[0092] Here, we take the SQL query SELECT name FROM es_index WHERE desc MATCH'Minglue Technology' ORDER BY age LIMIT 100 OFFSET 100 as an example. The process of converting the abstract syntax tree in Spark to the Elasticsearch query interface DSL is as follows: Figure 7 shown.

[0093] Example 2:

[0094] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of the data query system of the present invention. Figure 5 A data query system of the present invention includes:

[0095] A registration module registers the obtained syntax analysis interface into the Spark system;

[0096] A mapping information acquisition module, wherein the mapping information acquisition module acquires mapping information in the Elasticsearch system through the index name in the Spark system;

[0097] A temporary table creation module, wherein the temporary table creation module creates a temporary table in the Spark system according to the mapping information;

[0098] A conversion module converts operators in SQL query statements into query operators in the DSL of the Elasticsearch native query interface;

[0099] A query module, wherein the query module uses SQL query statements to query data in the Elasticsearch system;

[0100] A fusion module, which uses the Spark system to fuse the Elasticsearch data with other system data, and then performs analysis and query through the Spark system.

[0101] Wherein, the registration module includes:

[0102] A file generation unit, wherein the file generation unit writes a syntax analysis and generates a file;

[0103] An idiom grammar analysis interface generation unit is used to generate a grammar analysis interface using the grammar analysis, and register the grammar analysis interface in the Spark system.

[0104] Wherein, the mapping information acquisition module includes:

[0105] A connection unit, wherein the connection unit is connected to the Elasticsearch system in the Spark system using the address and port number of the Elasticsearch system;

[0106] A creation unit is configured to obtain the mapping information of a specific index in the Elasticsearch system through the index name.

[0107] The fusion module includes:

[0108] A mounting unit, which mounts the abstract syntax tree specific to the Elasticsearch query to the Join node of the global abstract syntax tree through the Spark system;

[0109] An analysis and query unit is configured to fuse the Elasticsearch data with other system data according to the abstract syntax tree through the Spark system, and then perform analysis and query through the Spark system.

[0110] Example 3:

[0111] Combine Figure 6 As shown, this embodiment discloses a specific implementation of an electronic device. The electronic device may include a processor 81 and a memory 82 storing computer program instructions.

[0112] Specifically, the processor 81 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0113] Among them, the memory 82 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), where the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0114] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0115] The processor 81 implements any one of the data query methods in the above embodiments by reading and executing computer program instructions stored in the memory 82 .

[0116] In some embodiments, the electronic device may further include a communication interface 83 and a bus 80. Figure 6 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0117] The communication interface 83 is used to implement communication between the various modules, devices, units, and / or devices in the embodiments of the present application. The communication interface 83 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0118] The bus 80 includes hardware, software, or both, and couples the components of the electronic device to each other. The bus 80 includes, but is not limited to, at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Bus 80 may include one or more buses, where appropriate. Although embodiments herein describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

[0119] The electronic device can be based on data query to achieve the combination Figure 1-Figure 4 Described method.

[0120] In addition, in conjunction with the data query method in the above embodiment, the present application embodiment can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any of the cross-data query methods in the above embodiment is implemented.

[0121] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] In summary, the present invention provides the following advantages: using this solution, we can query data in the Elasticsearch system using SQL based on Spark. Because operators are pushed down, the performance is comparable to that of the native Elasticsearch interface. Furthermore, we can also perform fusion query analysis on data in the Elasticsearch system and data in other database systems.

[0123] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of the present application. Therefore, the scope of the present invention shall be subject to the scope of the appended claims.

Claims

1. A data query method, characterized in that: include: Registration step: register the obtained syntax analysis interface into the Spark system; Steps for obtaining mapping information: Obtain mapping information from the Spark system to the Elasticsearch system through the index name; Temporary table creation step: creating a temporary table in the Spark system according to the mapping information; Conversion step: Convert the operators in the SQL query statement into query operators in the DSL of the Elasticsearch native query interface; Query step: Use SQL query statements to query the data in the Elasticsearch system; Fusion step: using the Spark system to fuse the Elasticsearch data with data from other systems, and then performing analysis and query through the Spark system; The fusion step includes: a mounting step: mounting the abstract syntax tree specific to the Elasticsearch query to the Join node of the global abstract syntax tree through the Spark system; Analysis and query step: After fusing the Elasticsearch data with other system data according to the abstract syntax tree through the Spark system, analysis and query are performed through the Spark system.

2. The data query method according to claim 1, wherein: The registration steps include: File generation steps: write syntax analysis to generate files; Idiom grammar analysis interface generation step: using the grammar analysis to generate a grammar analysis interface, and registering the grammar analysis interface in the Spark system.

3. The data query method according to claim 2, wherein: The step of obtaining mapping information includes: Connecting step: In the Spark system, connect to the Elasticsearch system using the address and port number of the Elasticsearch system; Creation step: obtaining the mapping information of a specific index in the Elasticsearch system through the index name.

4. A data query system, characterized in that: include: A registration module registers the obtained syntax analysis interface into the Spark system; A mapping information acquisition module, wherein the mapping information acquisition module acquires mapping information in the Elasticsearch system through the index name in the Spark system; A temporary table creation module, wherein the temporary table creation module creates a temporary table in the Spark system according to the mapping information; A conversion module converts operators in SQL query statements into query operators in the DSL of the Elasticsearch native query interface; A query module, wherein the query module uses SQL query statements to query data in the Elasticsearch system; A fusion module, which uses the Spark system to fuse the Elasticsearch data with other system data and then performs analysis and query through the Spark system; the fusion module includes: a mounting unit, which mounts the abstract syntax tree specific to the Elasticsearch query to the Join node of the global abstract syntax tree through the Spark system; An analysis and query unit is configured to fuse the Elasticsearch data with other system data according to the abstract syntax tree through the Spark system, and then perform analysis and query through the Spark system.

5. The data query system according to claim 4, wherein: The registration module includes: A file generation unit, wherein the file generation unit writes a syntax analysis and generates a file; An idiom grammar analysis interface generation unit is used to generate a grammar analysis interface using the grammar analysis, and register the grammar analysis interface in the Spark system.

6. The data query system according to claim 5, characterized in that: The mapping information acquisition module includes: A connection unit, wherein the connection unit is connected to the Elasticsearch system in the Spark system using the address and port number of the Elasticsearch system; A creation unit is configured to obtain the mapping information of a specific index in the Elasticsearch system through the index name.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data query method according to any one of claims 1 to 3 is implemented.

8. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data query method according to any one of claims 1 to 3 is implemented.

Citation Information

Patent Citations

  • Realization method and system for querying elasticsearch

    CN106934062A