SQL parameter-based data import method, device, equipment and medium
By introducing a data import method based on SQL parameters in the Spark environment, the problems of complex Spark code writing and high learning costs are solved. This enables efficient and secure data processing and import into ElasticSearch, improving the accuracy of data processing and the versatility of the program.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING TOPSEC NETWORK SECURITY TECH
- Filing Date
- 2023-11-09
- Publication Date
- 2026-08-04
AI Technical Summary
In existing technologies, Spark code is complex to write, has a high learning cost, poor program versatility, low data processing efficiency and accuracy, and insufficient data import security.
Using an SQL parameter-based approach, temporary views are created and Kafka data is loaded into Elasticsearch in the Spark environment by defining SQL statements. The Spark runtime mechanism is then used to perform data import, enabling data processing and import without requiring specialized coding knowledge.
It reduces learning costs, improves the accuracy and efficiency of data processing, reduces the risk of data errors, and enhances the program's versatility and the security of data import.
Smart Images
Figure CN117473023B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a data import method, apparatus, device, and medium based on SQL parameters. Background Technology
[0002] Kafka cluster is a distributed stream processing platform used for high-throughput, real-time data transmission and processing. Data from Kafka clusters is processed in real time using Spark and then fed into the Elasticsearch distributed search and analytics engine. It is commonly used to build applications such as real-time data processing and search engines.
[0003] The relevant technologies typically use Spark's Structured Streaming application to connect to the Kafka data stream and apply real-time operations to the Kafka data stream. For example, various transformations and operations are used to process the data in real time, and then the processed data is written to Elasticsearch for subsequent search and analysis.
[0004] Among related technologies, Spark, as a big data processing framework, involves complex distributed computing concepts and technologies. Writing and learning Spark code requires professional coding knowledge, consuming significant resources and time, resulting in a high learning curve. Furthermore, when processing different types of data, it is often necessary to rewrite, adjust, or copy previous code, leading to poor program versatility, high maintenance costs, reduced accuracy and efficiency in data processing, and compromised data import security. Summary of the Invention
[0005] In view of this, the present disclosure provides a data import method, apparatus, device, and medium based on SQL parameters, offering a more intelligent and simplified solution for data import in a big data environment. By performing real-time data processing and importing into ElasticSearch based on SQL parameters in a Spark environment, users can process and import different data types without professional coding knowledge, reducing learning costs, greatly improving the accuracy and efficiency of data processing, reducing the risk of data errors, improving the versatility of the program, and enhancing the security of data import in a big data environment.
[0006] In a first aspect, embodiments of this disclosure provide a data import method based on SQL parameters, employing the following technical solution:
[0007] The first SQL statement and the second SQL statement are defined according to the preset configuration information. The first SQL statement is used to create a temporary view in the Spark Session computing engine session. The data source of the temporary view is ElasticSearch. The second SQL statement is used to load the target data of the Kafka cluster into Spark and import the target data in Spark into ElasticSearch through the temporary view.
[0008] The parser converts the first and second SQL statements into program execution logic.
[0009] Extract the statement parameters of the second SQL statement and use the statement parameters as input parameters for the stream task logic plan;
[0010] Create an insert command and start a Spark streaming computation task via the API interface;
[0011] The insertion command is executed using the Spark runtime mechanism to import the target data into the Elasticsearch.
[0012] In some embodiments, the insertion command is executed using the Spark runtime mechanism to import the target data into the Elasticsearch, including:
[0013] Create a custom ElasticSearch data source using Spark's runtime mechanism;
[0014] Based on the preset view rules of the temporary view, the target data is selected from the data source of the Kafka cluster;
[0015] The target data is loaded into Spark in real time using the Spark Streaming application;
[0016] The insert command is executed to transform the target data in Spark and insert it into the Elasticsearch data source.
[0017] In some embodiments, a custom ElasticSearch data source is created using the Spark runtime mechanism, including:
[0018] Locate the META-INF / service directory;
[0019] Add the fully qualified path to the Es Source Provider class to the org.apache.spark.sql.Data Source Register file in the META-INF / service directory;
[0020] Implement the Data Source Register class into the Es Source Provider class;
[0021] In the Data Source Register class, set shortName to insert_es;
[0022] The target data is written to the ElasticSearch data source using the es-hadoop library through the mechanism of the Spark streaming task.
[0023] In some embodiments, before defining the first SQL statement and the second SQL statement according to preset configuration information, the method further includes:
[0024] Set the view configuration parameters of the temporary view, wherein the view configuration parameters include at least one of the connection information between the temporary view and the ElasticSearch, node information, and index name.
[0025] In some embodiments, extracting the statement parameters of the second SQL statement and using the statement parameters as input parameters for the stream task logical plan includes:
[0026] Extract at least one of the Kafka cluster connection information, subscribed topics, and consumer offset from the second SQL statement as the input parameter;
[0027] The input parameters are encapsulated into a KV structure;
[0028] The encapsulated input parameters are written into the streaming task logic plan.
[0029] In some embodiments, the method further includes:
[0030] When the streaming task logic plan is an operation to insert into a table, preset supplementary parameters are written to the ElasticSearch data source, wherein the preset supplementary parameters are used to indicate that the target data is written to the ElasticSearch data source.
[0031] In some embodiments, the parser converts the first SQL statement and the second SQL statement into program execution logic, including:
[0032] The first data source expression in the first SQL statement is matched using regular expressions, and the second data source expression in the second SQL statement is matched using regular expressions.
[0033] Add a first identifier to the first data source name in the first data source expression, and add a second identifier to the second data source name in the second data source expression.
[0034] Secondly, this disclosure also provides a data import device based on SQL parameters, employing the following technical solution:
[0035] The statement definition unit is configured to define a first SQL statement and a second SQL statement according to preset configuration information. The first SQL statement is used to create a temporary view in the Spark Session computing engine session. The data source of the temporary view is ElasticSearch. The second SQL statement is used to load the target data of the Kafka cluster into Spark and import the target data in Spark into ElasticSearch through the temporary view.
[0036] The conversion unit is configured to convert the first SQL statement and the second SQL statement into program execution logic through a parser;
[0037] The extraction unit is configured to extract the statement parameters of the second SQL statement and use the statement parameters as input parameters of the stream task logic plan;
[0038] The creation unit is configured to create insert commands and start Spark streaming computation tasks via the API interface;
[0039] The import unit is configured to execute the insert command using the Spark runtime mechanism to import the target data into the ElasticSearch.
[0040] Thirdly, this disclosure also provides an electronic device that adopts the following technical solution:
[0041] The electronic device includes:
[0042] At least one processor; and,
[0043] A memory communicatively connected to the at least one processor; wherein,
[0044] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform any of the above-described SQL parameter-based data import methods.
[0045] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing computer instructions for causing a computer to execute any of the above-described data import methods based on SQL parameters.
[0046] This disclosure provides a data import method based on SQL parameters. By performing real-time data processing and importing into ElasticSearch based on SQL parameters in a Spark environment, users can process and import different data types without professional coding knowledge, reducing the learning cost. This method can greatly improve the accuracy and efficiency of data processing, reduce the risk of introducing data errors, improve the versatility of the program, and enhance the security of data import in a big data environment.
[0047] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0048] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0049] Figure 1 A flowchart illustrating a data import method based on SQL parameters provided in this embodiment of the disclosure;
[0050] Figure 2 This is a schematic diagram of the structure of a data import device based on SQL parameters provided in an embodiment of the present disclosure;
[0051] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0052] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0053] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0054] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.
[0055] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0056] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.
[0057] Figure 1 This is a flowchart illustrating a data import method based on SQL parameters provided in this disclosure. The method includes the following steps:
[0058] S101. Define the first SQL statement and the second SQL statement according to the preset configuration information. The first SQL statement is used to create a temporary view in the Spark Session computing engine session. The data source of the temporary view is ElasticSearch. The second SQL statement is used to load the target data of the Kafka cluster into Spark and import the target data in Spark into ElasticSearch through the temporary view.
[0059] For example, the first SQL statement is:
[0060] CREATE TEMPORARYVIEW es_xxx(name string, password string) USING org.elasticsearch.spark.sql.DefaultSource OPTIONS("es.batch.write.refresh"="false","es.nodes"="http: / / localhost:23200","es.resource"="http_log*","es.resource.write"="http_log / http_log");
[0061] The second SQL statement is: INSERT INTO xxx select*from kafka("kafka.bootstrap.servers"="localhost:23093","subscribe"="http_log","startingoffsets"="latest").
[0062] In this embodiment, a temporary view named es_http_log is created. This temporary view uses Elasticsearch as the data source. In Spark, the temporary view is equivalent to defining a "virtual table" for accessing data in Elasticsearch.
[0063] S102. The parser converts the first SQL statement and the second SQL statement into program execution logic.
[0064] Optionally, for the first SQL statement and the second SQL statement, implement the SparkSqlParser interface to convert the first SQL statement and the second SQL statement into program execution logic.
[0065] Optionally, the parser translates the first SQL statement and the second SQL statement into program execution logic, including:
[0066] The first data source expression in the first SQL statement is matched using regular expressions, and the second data source expression in the second SQL statement is matched using regular expressions.
[0067] Add a first identifier to the first data source name in the first data source expression, and add a second identifier to the second data source name in the second data source expression.
[0068] For example, regular expressions are used to match the first data source expression `es_xxx` in the first SQL statement, and regular expressions are used to match the second data source expression `kafka` in the second SQL statement. Then, the first data source name in the first data source expression `es_xxx` is replaced by adding a first identifier to the front and back of the first data source name. Similarly, the second data source name in the second data source expression `kafka` is replaced by adding a second identifier to the front and back of the second data source name. During the parsing process, the first and second data source expressions are processed and transformed to make them recognizable by the program.
[0069] For example, the first and second identifiers can be "__", and the converted first SQL statement is: CREATE TEMPORARY VIEW __es_xxx__(name string, password string) USING org.elasticsearch.spark.sql.DefaultSource OPTIONS("es.batch.write.refresh"="false","es.nodes"="http: / / localhost:23200","es.resource"="http_log*","es.resource.write"="http_log / http_log");
[0070] The converted second SQL statement is: INSERT INTO xxx select*from__kafka__("kafka.bootstrap.servers"="localhost:23093","subscribe"="http_log","startingoffsets"="latest").
[0071] S103. Extract the statement parameters of the second SQL statement and use the statement parameters as input parameters for the flow task logic plan.
[0072] In some embodiments, extracting the statement parameters of the second SQL statement and using these statement parameters as input parameters for the stream task logical plan includes:
[0073] Extract at least one of the following from the second SQL statement: Kafka cluster connection information, subscribed topic, and consumer offset, as input parameters;
[0074] Encapsulate the input parameters into a key-value structure;
[0075] Write the encapsulated input parameters into the stream task logic plan.
[0076] S104. Create an insert command and start a Spark streaming task via the API interface.
[0077] S105. Execute the insert command using the Spark runtime mechanism to import the target data into Elasticsearch.
[0078] This disclosure applies to data import from Kafka to Elasticsearch (ES) in big data environments. By introducing SQL parameters, it achieves an efficient and flexible data processing workflow. Through real-time data processing and import into Elasticsearch based on SQL parameters in a Spark environment, users can handle and import different data types without requiring specialized coding knowledge, reducing the learning curve. This significantly improves the accuracy and efficiency of data processing, reduces the risk of data errors, enhances program versatility, and improves the security of data import in big data environments.
[0079] In some embodiments, the Spark runtime mechanism is used to execute insert commands to import target data into Elasticsearch, including:
[0080] Create a custom ElasticSearch data source using Spark's runtime mechanism;
[0081] Based on the preset view rules of the temporary view, the target data is selected from the data source of the Kafka cluster;
[0082] Load target data into Spark in real time using Spark Streaming applications;
[0083] Execute the insert command to transform the target data in Spark and insert it into the Elasticsearch data source.
[0084] In some embodiments, a custom ElasticSearch data source is created using the Spark runtime mechanism, including:
[0085] Locate the META-INF / service directory;
[0086] Add the fully qualified path to the Es Source Provider class to the org.apache.spark.sql.Data Source Register file in the META-INF / service directory;
[0087] Implement the Data Source Register class into the Es Source Provider class;
[0088] In the Data Source Register class, set shortName to insert_es;
[0089] By leveraging the Spark streaming task mechanism, the es-hadoop library is used to write the target data to an ElasticSearch data source.
[0090] Optionally, the above process includes locating the META-INF / service directory in the project, then adding the full path to the Elasticsearch data source provider class to the org.apache.spark.sql.DataSourceRegister file (filename; the Spark runtime mechanism registers the data source based on the package name specified in this file), ensuring that the EsSourceProvider class implements the DataSourceRegister class (a Java language interface), and then setting shortName to "insert_es" (the name of the data source, also an interface property in the DataSourceRegister class) in the Data Source Register class. Finally, using the Spark stream processing task mechanism, the es-hadoop library (an open-source JAR file used for writing data to Elasticsearch in a big data environment) is used to write the data to the Elasticsearch data source.
[0091] In some embodiments, before defining the first SQL statement and the second SQL statement according to preset configuration information, the method further includes:
[0092] Set the view configuration parameters for the temporary view, where the view configuration parameters include at least one of the following: connection information between the temporary view and ElasticSearch, node information, and index name.
[0093] In some embodiments, the method further includes:
[0094] When the streaming task's logical plan involves an insert operation into a table, preset supplementary parameters are written to the Elasticsearch data source. These preset supplementary parameters indicate that the target data is being written to the Elasticsearch data source. By writing these preset supplementary parameters to the Elasticsearch data source, the streaming task's logical plan can be optimized.
[0095] Figure 2 This is a schematic diagram of a data import device based on SQL parameters provided in an embodiment of this disclosure. This disclosure also provides a data import device based on SQL parameters, including:
[0096] Statement definition unit 21 is configured to define a first SQL statement and a second SQL statement according to preset configuration information. The first SQL statement is used to create a temporary view in the Spark Session computing engine session. The data source of the temporary view is ElasticSearch. The second SQL statement is used to load the target data of the Kafka cluster into Spark and import the target data in Spark into ElasticSearch through the temporary view.
[0097] The conversion unit 22 is configured to convert the first SQL statement and the second SQL statement into program execution logic through a parser;
[0098] Extraction unit 23 is configured to extract the statement parameters of the second SQL statement and use the statement parameters as input parameters of the flow task logic plan;
[0099] Create cell 24, which is configured to create insert commands and start Spark streaming computation tasks through the API interface;
[0100] Import unit 25 is configured to use the Spark runtime mechanism to execute insert commands and import the target data into Elasticsearch.
[0101] An electronic device according to embodiments of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, flash memory, etc.
[0102] The processor may be a central processing unit (CPU) or other processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the electronic device to perform all or part of the steps of a data import method based on SQL parameters according to the foregoing embodiments of this disclosure.
[0103] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.
[0104] like Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the electronic device in the embodiment of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0105] like Figure 3 As shown, an electronic device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) or a program loaded from a storage device into random access memory (RAM). The RAM also stores various programs and data required for the operation of the electronic device. The processor, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.
[0106] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow electronic devices to communicate wirelessly or wiredly with other devices (such as edge computing devices) to exchange data. Although Figure 3 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.
[0107] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from ROM. When the computer program is executed by a processor, all or part of the steps of a data import method based on SQL parameters according to embodiments of this disclosure are performed.
[0108] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.
[0109] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions. When these non-transitory computer-readable instructions are executed by a processor, all or part of the steps of a data import method based on SQL parameters according to the foregoing embodiments of the present disclosure are performed.
[0110] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).
[0111] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.
[0112] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0113] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.
[0114] Additionally, as used herein, the “or” used in a list of items beginning with “at least one” indicates a separate list, such that a list of, for example, “at least one of A, B, or C” means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word “exemplary” does not imply that the described example is preferred or better than other examples.
[0115] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.
[0116] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.
[0117] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0118] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.
Claims
1. A data import method based on SQL parameters, characterized in that, include: The first SQL statement and the second SQL statement are defined according to the preset configuration information. The first SQL statement is used to create a temporary view in the Spark Session computing engine session. The data source of the temporary view is ElasticSearch. The second SQL statement is used to load the target data of the Kafka cluster into Spark and import the target data in Spark into ElasticSearch through the temporary view. The parser converts the first and second SQL statements into program execution logic. Extract the statement parameters of the second SQL statement and use the statement parameters as input parameters for the stream task logic plan; Create an insert command and start a Spark streaming computation task via the API interface; The insertion command is executed using the Spark runtime mechanism to import the target data into the Elasticsearch. The step of converting the first SQL statement and the second SQL statement into program execution logic through a parser includes: The first data source expression in the first SQL statement is matched using regular expressions, and the second data source expression in the second SQL statement is matched using regular expressions. Add a first identifier to the first data source name in the first data source expression, and add a second identifier to the second data source name in the second data source expression; The step of using Spark's runtime mechanism to execute the insertion command and import the target data into Elasticsearch includes: Create a custom ElasticSearch data source using Spark's runtime mechanism; Based on the preset view rules of the temporary view, the target data is selected from the data source of the Kafka cluster; The target data is loaded into Spark in real time using the Spark Streaming application; The insert command is executed to transform the target data in Spark and insert it into the Elasticsearch data source.
2. The data import method based on SQL parameters according to claim 1, characterized in that, Create a custom ElasticSearch data source using Spark's runtime mechanism, including: Locate the META-INF / service directory; Add the fully qualified path to the Es Source Provider class to the org.apache.spark.sql.Data Source Register file in the META-INF / service directory; Implement the Data Source Register class into the Es Source Provider class; In the Data Source Register class, set shortName to insert_es; The target data is written to the ElasticSearch data source using the es-hadoop library through the mechanism of the Spark streaming task.
3. The data import method based on SQL parameters according to claim 1, characterized in that, Before defining the first SQL statement and the second SQL statement according to the preset configuration information, the method further includes: Set the view configuration parameters of the temporary view, wherein the view configuration parameters include at least one of the connection information between the temporary view and the ElasticSearch, node information, and index name.
4. The data import method based on SQL parameters according to claim 1, characterized in that, Extract the statement parameters of the second SQL statement and use these parameters as input parameters for the stream task logical plan, including: Extract at least one of the Kafka cluster connection information, subscribed topics, and consumer offset from the second SQL statement as the input parameter; The input parameters are encapsulated into a KV structure; The encapsulated input parameters are written into the streaming task logic plan.
5. The data import method based on SQL parameters according to claim 1, characterized in that, The method further includes: When the streaming task logic plan is an operation to insert into a table, preset supplementary parameters are written to the ElasticSearch data source, wherein the preset supplementary parameters are used to indicate that the target data is written to the ElasticSearch data source.
6. A data import device based on SQL parameters, characterized in that, include: The statement definition unit is configured to define a first SQL statement and a second SQL statement according to preset configuration information. The first SQL statement is used to create a temporary view in the Spark Session computing engine session. The data source of the temporary view is ElasticSearch. The second SQL statement is used to load the target data of the Kafka cluster into Spark and import the target data in Spark into ElasticSearch through the temporary view. The conversion unit is configured to convert the first SQL statement and the second SQL statement into program execution logic through a parser; The extraction unit is configured to extract the statement parameters of the second SQL statement and use the statement parameters as input parameters of the stream task logic plan; The creation unit is configured to create insert commands and start Spark streaming computation tasks via the API interface; The import unit is configured to execute the insert command using the Spark runtime mechanism to import the target data into the Elasticsearch. The step of converting the first SQL statement and the second SQL statement into program execution logic through a parser includes: The first data source expression in the first SQL statement is matched using regular expressions, and the second data source expression in the second SQL statement is matched using regular expressions. Add a first identifier to the first data source name in the first data source expression, and add a second identifier to the second data source name in the second data source expression; The step of using Spark's runtime mechanism to execute the insertion command and import the target data into Elasticsearch includes: Create a custom ElasticSearch data source using Spark's runtime mechanism; Based on the preset view rules of the temporary view, the target data is selected from the data source of the Kafka cluster; The target data is loaded into Spark in real time using the Spark Streaming application; The insert command is executed to transform the target data in Spark and insert it into the Elasticsearch data source.
7. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the data import method based on SQL parameters as described in any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the data import method based on SQL parameters as described in any one of claims 1 to 5.