A data query method and apparatus based on Kafka

Abstract

This invention discloses a data query method and apparatus based on Kafka, comprising: determining a structured query language tree based on in-memory data in Kafka; wherein the in-memory data includes real-time streaming data corresponding to real-time streaming tasks; truncating the real-time streaming data based on the structured query language tree to generate a truncated table corresponding to the real-time streaming data; and querying target data from the truncated table using preset query conditions. This invention can improve data query efficiency.

Description

Technical Field

[0001] This invention relates to the field of big data analytics, and more specifically, to a data query method and apparatus based on Kafka. Background Technology

[0002] Currently, in the field of big data analytics, Kafka, as a message broker, is an indispensable component. Kafka is a high-throughput distributed publish-subscribe messaging system capable of processing all action stream data from consumers on a website.

[0003] In practice, it has been found that data queries are frequently required within messages stored in Kafka. Current methods for this involve writing structured query languages ​​(SQL) via the command line. However, this approach is cumbersome and inefficient.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a data query method and apparatus based on Kafka to at least improve data query efficiency.

[0006] According to one aspect of the present invention, a data query method based on Kafka is provided. The method includes: determining a structured query language tree based on in-memory data in Kafka; wherein the in-memory data includes real-time streaming data corresponding to real-time streaming tasks; truncating the real-time streaming data based on the structured query language tree to generate a truncated table corresponding to the real-time streaming data; and querying target data from the truncated table using preset query conditions.

[0007] As an optional implementation, the method further includes setting a limited data flag and an end offset for the real-time streaming task to allow the real-time streaming task to consume in-memory data in Kafka.

[0008] As an optional implementation, the memory data is stored in a circular data buffer in Kafka, and the step of querying target data from the truncated table using preset query conditions includes: determining the amount of query data matching the preset query conditions; determining the buffer size information matching the query data amount information; determining the target buffer matching the buffer size information from the circular data buffer; determining the corresponding target data from the truncated table using the preset query conditions, and reading the target data into the target buffer.

[0009] As an optional implementation, the method further includes: writing data into and consuming data through the circular data buffer.

[0010] As an optional implementation, determining the corresponding target data from the truncated table based on the preset query conditions and reading the target data into the target buffer includes: parsing the preset query conditions to obtain a Structured Query Language (SCL) table field; determining the target data corresponding to the SCL table field from the truncated table and reading the target data into the target buffer.

[0011] As an optional implementation, based on the structured query language tree, the real-time streaming data is truncated to generate a truncated table corresponding to the real-time streaming data, including: if the preset query conditions contain a specified query granularity, based on the structured query language tree, the real-time streaming data corresponding to the specified query granularity is truncated to generate a truncated table corresponding to the real-time streaming data.

[0012] As an optional implementation, the preset query conditions include time-based query conditions; and the method further includes: periodically consuming in-memory data in Kafka.

[0013] As an optional implementation, the method further includes: determining business analysis results based on the target data.

[0014] According to another aspect of the present invention, a Kafka-based data query apparatus is also provided, comprising: a data mapping unit, configured to determine a structured query language tree based on memory data in Kafka; wherein the memory data includes real-time streaming data corresponding to a real-time streaming task; a truncation table generation unit, configured to truncate the real-time streaming data based on the structured query language tree to generate a truncation table corresponding to the real-time streaming data; and a data query unit, configured to query target data from the truncation table according to preset query conditions.

[0015] As an optional implementation, the apparatus further includes a task setting unit for setting a limited data flag and an end offset position for the real-time streaming task to allow the real-time streaming task to consume in-memory data in Kafka.

[0016] As an optional implementation, the memory data is stored in a circular data buffer in Kafka, and the data query unit is specifically used to: determine the amount of query data matching the preset query conditions; determine the buffer size information matching the query data amount information; determine the target buffer matching the buffer size information from the circular data buffer; determine the corresponding target data from the truncated table according to the preset query conditions, and read the target data into the target buffer.

[0017] As an optional implementation, the apparatus further includes a data processing unit for writing data to and consuming data in Kafka through the circular data buffer.

[0018] As an optional implementation, the data query unit is specifically used to: parse the preset query conditions to obtain the structured query language table fields; determine the target data corresponding to the structured query language table fields from the truncated table, and read the target data into the target buffer.

[0019] As an optional implementation, the truncation table generation unit is specifically used to: if the preset query conditions include a specified query granularity, based on the structured query language tree, truncate the real-time streaming data corresponding to the specified query granularity, and generate a truncation table corresponding to the real-time streaming data.

[0020] As an optional implementation, the preset query conditions include time-based query conditions; and the data processing unit is further configured to: periodically consume in-memory data in Kafka.

[0021] As an optional implementation, the data processing unit is also used to: determine business analysis results based on the target data.

[0022] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the above-described Kafka-based data query method at runtime.

[0023] According to another aspect of the present invention, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described Kafka-based data query method through the computer program.

[0024] In this embodiment of the invention, by mapping the memory data in Kafka to a structured query language tree, directly truncating the real-time streaming data, generating a truncated table, and querying the required target data from the truncated table, complex, real-time streaming queries are realized, thereby improving data query efficiency. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0026] Figure 1 This is a flowchart of an optional Kafka-based data query method according to an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of the structure of an optional Kafka-based data query device according to an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] This invention provides an optional Kafka-based data query method, such as... Figure 1 As shown, this Kafka-based data query method includes:

[0032] S101, Based on the memory data in Kafka, determine the structured query language tree; wherein, the memory data includes the real-time streaming data corresponding to the real-time streaming task;

[0033] S102, based on the structured query language tree, the real-time stream data is truncated to generate a truncated table corresponding to the real-time stream data;

[0034] S103, Query the target data from the truncated table according to the preset query conditions.

[0035] In this embodiment, the executing entity can be an electronic device such as a terminal device or a server, or a cluster of electronic devices such as terminal devices or servers.

[0036] In this process, the executing entity stores the message data source in a Kafka cluster when processing real-time streaming tasks. When a consumer consumes data, it starts two consumption threads (low-level and high-level) and shares the results via RPC (Remote Procedure Call). After the consumed data is shared, it flows back to the SQL (Structured Query Language) engine. At this point, the executing entity can retrieve the in-memory data from Kafka and, based on the SQL engine, translate this data into a Structured Query Language tree (SQLTree). Specifically, the translation of in-memory data from Kafka into a Structured Query Language tree using the SQL engine can be implemented using the open-source Calcite project. For the corresponding Kafka cluster, JSON Models conforming to Calcite can be configured.

[0037] The execution entity can obtain the specified interface (Schema.SchemaFactory) provided by Calcite, and based on this specified interface, implement getTableMap (used to provide a table name and a mapping table) in the class. It can view the corresponding database object through the Map table in memory, and then use the truncated table as the table in the Schema. For the defined table type, the GetRowType method and the scan method need to be implemented. During initialization, the data types in the table need to be mapped and matched. The KafkaMemoryData type provides relevant custom type mappings, which can realize the implementation of data writing to Kafka as a data source loading and SQL Tree mapping.

[0038] Subsequently, the execution entity can truncate the real-time streaming data in the real-time streaming task based on the structured query language tree, generating a truncation table and storing it in memory. Specifically, the execution entity can retrieve all real-time streaming data in the current real-time streaming task based on the structured query language tree and summarize it to obtain the truncation table. Alternatively, the execution entity can retrieve a portion of the real-time streaming data in the current real-time streaming task based on the structured query language tree and summarize it to obtain the truncation table. When generating the truncation table, structured query language fields can be generated in the truncation table so that subsequent matching based on these fields can directly retrieve the target data from the truncation table.

[0039] Subsequently, the executing entity can directly query the target data from the truncated table using preset query conditions. Existing data query methods rely on cluster-started clients, command-line SQL writing for queries, and default to consuming the latest data. If a query from the beginning is required, additional parameters need to be set in the session, which is cumbersome, inefficient, and does not support complex, real-time streaming queries. However, the data query method in this application maps in-memory data to an SQL Tree, truncates it into tables, and queries on the truncated tables reduce the workload of manually writing SQL and configuring query parameters, thereby improving data query efficiency.

[0040] Real-time streaming data is an unrestricted sequence of structured data. The facts in real-time streaming data are immutable, meaning that new facts can be inserted into the stream, but existing facts will never be updated or deleted. Streams can be created from Kafka topics or derived from existing streams and tables.

[0041] A truncated table is a view of a stream or another table, representing a constantly changing collection of facts, similar to a traditional database table but enriched with streaming semantics. The facts in the table are mutable, meaning new facts can be inserted, and existing facts can be updated or deleted. Tables can be created from Kafka topics or derived from existing streams and tables.

[0042] As an optional implementation, the method further includes setting a limited data flag and an end offset for the real-time streaming task to allow the real-time streaming task to consume in-memory data in Kafka.

[0043] In this implementation, for real-time streaming tasks, the task judgment conditions need to be modified to ensure that, in the case of batch processing of real-time streaming tasks, Kafka data can be consumed, and that the real-time streaming task stops after consuming Kafka data to a specified position. The task judgment conditions can be the conditions for judging data consumption during the execution of the real-time streaming task. For example, the task judgment condition can be to determine whether the consumed data has reached a specified position; if so, the real-time streaming task stops. Alternatively, the task judgment condition can be to determine whether data can be consumed; if so, the data in memory in Kafka is retrieved.

[0044] Specifically, real-time streaming tasks in Kafka do not inherently support consuming Kafka (stream) type data; that is, they cannot consume in-memory data from Kafka. To address this, a boundedness flag and an end offset can be set for the real-time streaming task to allow it to consume in-memory data from Kafka. The boundedness flag can be set using `this.boundedness = Boundedness.BOUNDED`. This flag ensures that Kafka streaming data can be processed in batch tasks. The end offset specifies the data position where consumption ends.

[0045] As an optional implementation, the memory data is stored in a circular data buffer in Kafka, and the step of querying target data from the truncated table using preset query conditions includes: determining the amount of query data matching the preset query conditions; determining the buffer size information matching the query data amount information; determining the target buffer matching the buffer size information from the circular data buffer; determining the corresponding target data from the truncated table using the preset query conditions, and reading the target data into the target buffer.

[0046] In this embodiment, the executing entity can first determine the number of data items to be retrieved and the data size of each data item based on preset query conditions, and then integrate the number of data items and the data size to obtain the query data volume information. Specifically, the product of the number of data items and the data size can be calculated to obtain the query data volume information. Next, the buffer size information matching the query data volume information is further determined. The query data volume information indicates that the larger the data size, the larger the corresponding buffer. The executing entity can pre-establish a mapping relationship between the query data volume information and the buffer size information.

[0047] As an optional implementation, the method further includes: writing data into and consuming data through the circular data buffer.

[0048] In this implementation, a circular data buffer can be used at the underlying memory level. Through real-time write consumption combined with truncated queries, the circular data buffer cyclically utilizes current memory by consuming writes and truncating reads within the buffer. The buffer size is determined by the size of the queried data, effectively solving the problem of large query errors due to excessive data volume and thus avoiding memory overflow. Asynchronous query truncation does not affect normal data writing.

[0049] As an optional implementation, determining the corresponding target data from the truncated table based on the preset query conditions and reading the target data into the target buffer includes: parsing the preset query conditions to obtain a Structured Query Language (SCL) table field; determining the target data corresponding to the SCL table field from the truncated table and reading the target data into the target buffer.

[0050] In this embodiment, the preset query conditions may include SQL table fields. The executing entity can determine the target data corresponding to the SQL table fields from the truncated table and read the target data into the aforementioned target buffer. Specifically, during the generation phase, the corresponding SQL table fields can be generated in the truncated table.

[0051] As an optional implementation, based on the structured query language tree, the real-time streaming data is truncated to generate a truncated table corresponding to the real-time streaming data, including: if the preset query conditions contain a specified query granularity, based on the structured query language tree, the real-time streaming data corresponding to the specified query granularity is truncated to generate a truncated table corresponding to the real-time streaming data.

[0052] In this embodiment, the specified query granularity can be either partition-level or topic-level; this embodiment is not limited to either. If the specified query granularity is partition-level, the real-time streaming data corresponding to the specified partition can be truncated based on the structured query language tree to generate a truncated table corresponding to the real-time streaming data, thereby retrieving data that meets business requirements. If the specified query granularity is topic-level, the real-time streaming data corresponding to the specified topic can be truncated based on the structured query language tree to generate a truncated table corresponding to the real-time streaming data. The corresponding data and offset can be retrieved using the query conditions under that topic. It is recommended to retrieve the corresponding offset based on keywords. When querying based on partition-level granularity, the table is truncated according to the consumer's subscription strategy and query conditions to retrieve data under the specified partition of the topic that meets business requirements.

[0053] As an optional implementation, the preset query conditions include time-based query conditions; and the method further includes: periodically consuming in-memory data in Kafka.

[0054] In this implementation, the time query condition can be used to query before scheduled consumption. Preferably, it can be used to query data within a specific Topic, perform quality monitoring before consuming data after writing it to Kafka, and perform preprocessing in advance.

[0055] As an optional implementation, the method further includes: determining business analysis results based on the target data.

[0056] In this embodiment, after obtaining the target data, the corresponding business scenario can be analyzed to obtain business analysis results. These business analysis results can be information obtained after analyzing the target data, and their representation can be text, charts, etc., which are not limited in this embodiment.

[0057] In this embodiment of the invention, by mapping the memory data in Kafka to a structured query language tree, directly truncating the real-time streaming data, generating a truncated table, and querying the required target data from the truncated table, complex, real-time streaming queries are realized, thereby improving data query efficiency.

[0058] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0059] Furthermore, embodiments of the present invention provide an optional Kafka-based data query device, such as... Figure 2 As shown, the Kafka-based data query device includes:

[0060] The data mapping unit 201 is used to determine a structured query language tree based on the memory data in Kafka; wherein the memory data includes real-time streaming data corresponding to the real-time streaming task.

[0061] The truncation table generation unit 202 is used to truncate the real-time stream data based on the structured query language tree and generate a truncation table corresponding to the real-time stream data.

[0062] The data query unit 203 is used to query target data from the truncated table according to preset query conditions.

[0063] In processing real-time streaming tasks, the message data source is stored in a Kafka cluster. When a consumer consumes data, two consumption threads, one low-level and one high-level, are started, and the consumption results are shared via RPC (Remote Procedure Call). After the consumed data is shared, it flows back to the SQL (Structured Query Language) engine. At this point, the in-memory data in Kafka can be retrieved, and based on the SQL engine, it is translated into a Structured Query Language tree (SQL Tree). Specifically, the translation of in-memory data in Kafka into a Structured Query Language tree based on the SQL engine can be implemented using the open-source Calcite project. For the corresponding Kafka cluster, JSON Models conforming to Calcite can be configured.

[0064] This involves obtaining a specified interface (Schema.SchemaFactory) provided by Calcite and implementing getTableMap (a mapping table for providing a table name) in the class based on this interface. The corresponding database object is then viewed through the in-memory Map table, and the truncated table is used as the table in the Schema. For the defined table type, the GetRowType and scan methods need to be implemented. During initialization, the data types in the table need to be mapped and matched. The KafkaMemoryData type provides relevant custom type mappings, thus enabling the implementation of data writing to Kafka, data source loading, and SQL Tree mapping.

[0065] Next, based on the Structured Query Language (SCL) tree, the real-time streaming data in the real-time streaming task can be truncated to generate a truncated table, which is then stored in memory. Specifically, the truncated table can be generated by retrieving all real-time streaming data from the current real-time streaming task using the SCL tree, or by retrieving only a portion of the real-time streaming data from the current real-time streaming task using the SCL tree. When generating the truncated table, SCL fields can be added to it to allow subsequent matching based on these fields to directly retrieve target data from the truncated table.

[0066] Afterwards, target data can be directly retrieved from the truncated table using preset query conditions. Existing data query methods rely on cluster-started clients, command-line SQL writing for queries, and default to consuming the latest data. If a query from the beginning is required, additional parameters need to be set in the session, which is cumbersome, inefficient, and does not support complex, real-time streaming queries. However, the data query method in this application maps in-memory data to an SQL Tree, truncates it into tables, and queries on the truncated tables reduce the workload of manually writing SQL and configuring query parameters, thereby improving data query efficiency.

[0067] Real-time streaming data is an unrestricted sequence of structured data. The facts in real-time streaming data are immutable, meaning that new facts can be inserted into the stream, but existing facts will never be updated or deleted. Streams can be created from Kafka topics or derived from existing streams and tables.

[0068] A truncated table is a view of a stream or another table, representing a constantly changing collection of facts, similar to a traditional database table but enriched with streaming semantics. The facts in the table are mutable, meaning new facts can be inserted, and existing facts can be updated or deleted. Tables can be created from Kafka topics or derived from existing streams and tables.

[0069] As an optional implementation, the apparatus further includes a task setting unit for setting a limited data flag and an end offset position for the real-time streaming task to allow the real-time streaming task to consume in-memory data in Kafka.

[0070] In this implementation, for real-time streaming tasks, the task judgment conditions need to be modified to ensure that, in the case of batch processing of real-time streaming tasks, Kafka data can be consumed, and that the real-time streaming task stops after consuming Kafka data to a specified position. The task judgment conditions can be the conditions for judging data consumption during the execution of the real-time streaming task. For example, the task judgment condition can be to determine whether the consumed data has reached a specified position; if so, the real-time streaming task stops. Alternatively, the task judgment condition can be to determine whether data can be consumed; if so, the data in memory in Kafka is retrieved.

[0071] Specifically, real-time streaming tasks in Kafka do not inherently support consuming Kafka (stream) type data; that is, they cannot consume in-memory data from Kafka. To address this, a boundedness flag and an end offset can be set for the real-time streaming task to allow it to consume in-memory data from Kafka. The boundedness flag can be set using `this.boundedness = Boundedness.BOUNDED`. This flag ensures that Kafka streaming data can be processed in batch tasks. The end offset specifies the data position where consumption ends.

[0072] As an optional implementation, the memory data is stored in a circular data buffer in Kafka, and the data query unit 203 is specifically used to: determine the amount of query data matching the preset query conditions; determine the buffer size information matching the query data amount information; determine the target buffer matching the buffer size information from the circular data buffer; determine the corresponding target data from the truncated table according to the preset query conditions, and read the target data into the target buffer.

[0073] In this embodiment, the executing entity can first determine the number of data items to be retrieved and the data size of each data item based on preset query conditions, and then integrate the number of data items and the data size to obtain the query data volume information. Specifically, the product of the number of data items and the data size can be calculated to obtain the query data volume information. Next, the buffer size information matching the query data volume information is further determined. The query data volume information indicates that the larger the data size, the larger the corresponding buffer. The executing entity can pre-establish a mapping relationship between the query data volume information and the buffer size information.

[0074] As an optional implementation, the apparatus further includes a data processing unit for writing data to and consuming data in Kafka through the circular data buffer.

[0075] In this implementation, a circular data buffer can be used at the underlying memory level. Through real-time write consumption combined with truncated queries, the circular data buffer cyclically utilizes current memory by consuming writes and truncating reads within the buffer. The buffer size is determined by the size of the queried data, effectively solving the problem of large query errors due to excessive data volume and thus avoiding memory overflow. Asynchronous query truncation does not affect normal data writing.

[0076] As an optional implementation, the data query unit 203 is specifically used to: parse the preset query conditions to obtain the structured query language table fields; determine the target data corresponding to the structured query language table fields from the truncated table, and read the target data into the target buffer.

[0077] In this embodiment, the preset query conditions may include SQL table fields. The executing entity can determine the target data corresponding to the SQL table fields from the truncated table and read the target data into the aforementioned target buffer. Specifically, during the generation phase, the corresponding SQL table fields can be generated in the truncated table.

[0078] As an optional implementation, the truncation table generation unit 202 is specifically used to: if the preset query conditions include a specified query granularity, based on the structured query language tree, truncate the real-time streaming data corresponding to the specified query granularity, and generate a truncation table corresponding to the real-time streaming data.

[0079] In this embodiment, the specified query granularity can be either partition-level or topic-level; this embodiment is not limited to either. If the specified query granularity is partition-level, the real-time streaming data corresponding to the specified partition can be truncated based on the structured query language tree to generate a truncated table corresponding to the real-time streaming data, thereby retrieving data that meets business requirements. If the specified query granularity is topic-level, the real-time streaming data corresponding to the specified topic can be truncated based on the structured query language tree to generate a truncated table corresponding to the real-time streaming data. The corresponding data and offset can be retrieved using the query conditions under that topic. It is recommended to retrieve the corresponding offset based on keywords. When querying based on partition-level granularity, the table is truncated according to the consumer's subscription strategy and query conditions to retrieve data under the specified partition of the topic that meets business requirements.

[0080] As an optional implementation, the preset query conditions include time-based query conditions; and the data processing unit is further configured to: periodically consume in-memory data in Kafka.

[0081] In this implementation, the time query condition can be used to query before scheduled consumption. Preferably, it can be used to query data within a specific Topic, perform quality monitoring before consuming data after writing it to Kafka, and perform preprocessing in advance.

[0082] As an optional implementation, the data processing unit is also used to: determine business analysis results based on the target data.

[0083] In this embodiment, after obtaining the target data, the corresponding business scenario can be analyzed to obtain business analysis results. These business analysis results can be information obtained after analyzing the target data, and their representation can be text, charts, etc., which are not limited in this embodiment.

[0084] In this embodiment of the invention, by mapping the memory data in Kafka to a structured query language tree, directly truncating the real-time streaming data, generating a truncated table, and querying the required target data from the truncated table, complex, real-time streaming queries are realized, thereby improving data query efficiency.

[0085] Furthermore, according to another aspect of the present invention, an electronic device for implementing the above-described Kafka-based data query method is also provided, such as... Figure 3 As shown, the electronic device includes a memory 302 and a processor 304. The memory 302 stores a computer program, and the processor 304 is configured to execute the steps of any of the above method embodiments via the computer program.

[0086] Optionally, in this embodiment, the electronic device may be located in at least one of a plurality of network devices in a computer network.

[0087] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0088] S1, Based on the in-memory data in Kafka, determine the structured query language tree; wherein, the in-memory data includes the real-time streaming data corresponding to the real-time streaming task;

[0089] S2, based on the structured query language tree, the real-time stream data is truncated to generate a truncated table corresponding to the real-time stream data;

[0090] S3, query the target data from the truncated table according to the preset query conditions.

[0091] Alternatively, as those skilled in the art will understand, Figure 3 The structure shown is for illustrative purposes only. The electronic device can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, and other terminal devices. Figure 3 This does not limit the structure of the aforementioned electronic device. For example, the electronic device may also include components that are more... Figure 3 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 3The different configurations shown.

[0092] The memory 302 can be used to store software programs and modules, such as the program instructions / modules corresponding to the Kafka-based data query method in this embodiment of the invention. The processor 304 executes various functional applications and data processing by running the software programs and modules stored in the memory 302, thereby realizing the aforementioned Kafka-based data query method. The memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 302 may further include memory remotely located relative to the processor 304, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 302 may be used, but is not limited to, to store information such as operation instructions. As an example, such as... Figure 3 As shown, the memory 302 may include, but is not limited to, various modules from the aforementioned device.

[0093] Optionally, the transmission device 306 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 306 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 306 is a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0094] In addition, the aforementioned electronic device also includes a display 308 and a connection bus 310.

[0095] According to another aspect of the present invention, a storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0096] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0097] S1, Based on the in-memory data in Kafka, determine the structured query language tree; wherein, the in-memory data includes the real-time streaming data corresponding to the real-time streaming task;

[0098] S2, based on the structured query language tree, the real-time stream data is truncated to generate a truncated table corresponding to the real-time stream data;

[0099] S3, query the target data from the truncated table according to the preset query conditions.

[0100] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0101] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0102] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0103] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0107] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data query method based on Kafka, characterized in that, include: Set a limited data flag and an end offset for the real-time streaming task to allow the real-time streaming task to consume in-memory data in Kafka; wherein the in-memory data is stored in a circular data buffer in Kafka; A structured query language tree is determined based on in-memory data in Kafka; wherein, the in-memory data includes real-time streaming data corresponding to real-time streaming tasks. Based on the structured query language tree, the real-time stream data is truncated to generate a truncated table corresponding to the real-time stream data; The target data is retrieved from the truncated table based on preset query conditions.

2. The method according to claim 1, characterized in that, The step of retrieving target data from the truncated table using preset query conditions includes: Determine the amount of query data that matches the preset query conditions; Determine the buffer size information that matches the amount of query data; Determine a target buffer that matches the buffer size information from the circular data buffer; Based on the preset query conditions, the corresponding target data is determined from the truncated table and read into the target buffer.

3. The method according to claim 2, characterized in that, The method further includes: Data is written to and consumed from the circular data buffer.

4. The method according to claim 2, characterized in that, The process of determining the corresponding target data from the truncated table based on the preset query conditions and reading the target data into the target buffer includes: Parse the preset query conditions to obtain the fields of the Structured Query Language table; From the truncated table, determine the target data corresponding to the field of the Structured Query Language table, and read the target data into the target buffer.

5. The method according to claim 1, characterized in that, Based on the structured query language tree, the real-time stream data is truncated to generate a truncated table corresponding to the real-time stream data, including: If the preset query conditions include a specified query granularity, based on the structured query language tree, the real-time stream data corresponding to the specified query granularity is truncated to generate a truncated table corresponding to the real-time stream data.

6. The method according to claim 1, characterized in that, The preset query conditions include time-based query conditions; as well as The method further includes: Consume in-memory data in Kafka on a regular schedule.

7. The method according to claim 1, characterized in that, The method further includes: Based on the target data, the business analysis results are determined.

8. A data query device based on Kafka, characterized in that, include: The task setting unit sets a limited data flag and an end offset position for the real-time streaming task to allow the real-time streaming task to consume in-memory data in Kafka; wherein, the in-memory data is stored in a circular data buffer in Kafka; A data mapping unit is used to determine a structured query language tree based on in-memory data in Kafka; wherein the in-memory data includes real-time streaming data corresponding to real-time streaming tasks. The truncation table generation unit is used to truncate the real-time stream data based on the structured query language tree and generate a truncation table corresponding to the real-time stream data. The data query unit is used to query target data from the truncated table based on preset query conditions.

Images