A kafka data quasi-real-time query method and device, electronic equipment and medium

Abstract

The present application relates to big data technology, and disclose a kind of kafka data quasi real-time query method, comprising: adding preset message header field to kafka data, the message body of kafka data is as independent field, message header field and independent field are reorganized, and updated kafka data is obtained;According to the storage structure of hive table, updated kafka data is imported into distributed file system, and storage kafka data is obtained;Mapping kafka data is mapped to the hive external table constructed in advance, and mapping kafka data is obtained, the continuity of mapping kafka data is verified using the offest field in message header field, and continuous kafka data is obtained;Continuous kafka data is inquired in real time.The present application also proposes a kind of kafka data quasi real-time query device, electronic equipment and storage medium.The present application can reduce the resource consumption when kafka data is inquired.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a near real-time query method, apparatus, electronic device, and computer-readable storage medium for Kafka data. Background Technology

[0002] As Kafka, a high-throughput distributed message subscription system, is widely used in data and other fields, there is a frequent need to query Kafka data, such as checking if a specific piece of data exists in Kafka, or whether Kafka has received a particular message. However, as a message middleware, Kafka's inherent storage structure is not suitable for query analysis. To enable querying of data in Kafka, it is necessary to perform data transfer processing to allow for near real-time querying.

[0003] Existing Kafka data query technologies typically involve querying data through Kafka management tools, using a real-time data consumption and in-memory processing approach, which only supports small datasets. In real-world production environments, massive amounts of data exist. Focusing solely on querying small amounts results in low system flexibility, an inability to support complex queries, and consequently, high resource consumption when querying Kafka data. Summary of the Invention

[0004] To address the aforementioned issues, embodiments of the present invention provide a near real-time Kafka data query method, apparatus, electronic device, and computer-readable storage medium.

[0005] In a first aspect, embodiments of the present invention provide a near real-time query method for Kafka data, including:

[0006] Add a preset message header field to the Kafka data, treat the message body of the Kafka data as an independent field, and reorganize the message header field and the independent field to obtain the updated Kafka data;

[0007] The updated Kafka data is imported into a preset distributed file system according to the preset storage structure of the Hive table, thus obtaining the stored Kafka data;

[0008] The stored Kafka data is mapped to a pre-built Hive external table to obtain mapped Kafka data. The continuity of the mapped Kafka data is verified using the offset field in the message header to obtain continuous Kafka data.

[0009] Perform real-time queries on the continuous Kafka data.

[0010] According to an embodiment of the present invention, the step of recombining the message header fields and the independent fields to obtain updated Kafka data includes:

[0011] Use the aforementioned message header field as the message header for updating Kafka data;

[0012] Use the independent field as the message body for updating Kafka data;

[0013] The updated Kafka data is obtained by reconstructing the message format of the updated Kafka data based on the message header and the message body.

[0014] According to an embodiment of the present invention, the step of importing the updated Kafka data into a preset distributed file system according to a preset Hive table storage structure to obtain stored Kafka data includes:

[0015] By rewriting the fields of the KafkaSource class in Flume, we obtain the rewritten KafkaSource class.

[0016] According to the storage structure of the hive table, the HeaderAndBodyTextEventSerializer class in Flume is rewritten for event serialization to obtain the rewritten HeaderAndBodyTextEventSerializer class;

[0017] Configure Flume's source component according to the rewritten KafkaSource class and the updated Kafka data to obtain configuration source information; configure Flume's channel component according to the preset buffer attributes to obtain configuration channel information; configure Flume's sink component according to the distributed file system and the rewritten HeaderAndBodyTextEventSerializer class to obtain configuration sink information.

[0018] The configuration source information, configuration channel information, and configuration sink information are used to generate a transmission file. Flume is then run based on the transmission file to obtain the stored Kafka data.

[0019] According to an embodiment of the present invention, before mapping the stored Kafka data to a pre-built Hive external table to obtain the mapped Kafka data, the method further includes:

[0020] Create the storage path and delimiter field for the Kafka data;

[0021] Generate the field attributes of the Hive external table based on the field attributes of the stored Kafka data;

[0022] The Hive external table is created based on the storage path, the delimiter field, and the field attributes.

[0023] According to an embodiment of the present invention, the step of verifying the continuity of the mapped Kafka data using the `offest` field in the message header to obtain continuous Kafka data includes:

[0024] Perform continuity verification on the numeric records in the offset field corresponding to the mapped Kafka data to obtain the numeric records in the offset field;

[0025] When the digital records are continuous, the mapped Kafka data is used as the continuous Kafka data.

[0026] According to an embodiment of the present invention, the real-time querying of the continuous Kafka data includes:

[0027] Using preset connection parameters, a preset Hive visualization client tool is connected to the Hive data warehouse to obtain a visualization interface for the Hive database;

[0028] The continuous Kafka data can be queried in real time through the visualization interface according to the preset SQL statement.

[0029] According to an embodiment of the present invention, the real-time querying of the continuous Kafka data includes:

[0030] The continuous Kafka data is queried in real time using a preset Hive-compatible query engine.

[0031] Secondly, embodiments of the present invention provide a near real-time Kafka data query device, characterized in that it includes:

[0032] The field reorganization module is used to add preset message header fields to Kafka data, treat the message body of the Kafka data as an independent field, and reorganize the message header fields and the independent fields to obtain updated Kafka data.

[0033] The Kafka data import module is used to import the updated Kafka data into a preset distributed file system according to the preset storage structure of the Hive table, so as to obtain the stored Kafka data.

[0034] The data continuity verification module is used to map the stored Kafka data to a pre-built Hive external table to obtain mapped Kafka data, and to verify the continuity of the mapped Kafka data using the offset field in the message header field to obtain continuous Kafka data.

[0035] The Kafka data real-time query module is used to perform real-time queries on the continuous Kafka data.

[0036] Thirdly, embodiments of the present invention provide an electronic device, which includes:

[0037] processor;

[0038] Memory used to store the processor's executable instructions;

[0039] The processor is configured to execute the instructions to implement a near real-time Kafka data query method as described in the first aspect above.

[0040] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a near real-time Kafka data query method as described in the first aspect above.

[0041] Compared with the prior art, the above-mentioned technical solution of the present invention has the following beneficial effects:

[0042] This invention uses Flume to import Kafka data into HDFS. During the import process, the Kafka message body is stored as an independent field, and necessary Kafka message header fields are added for data verification and analysis. The structure of this data in HDFS must conform to the storage structure of Hive tables. A Hive external table is created, pointing to the data already imported into HDFS. Data integrity is verified using a method based on Kafka offset continuity. A Hive query client is used for flexible analysis and querying of the data. Historical data is periodically compressed and merged to reduce resource consumption. This achieves flexible, SQL-based querying and analysis of full information in Kafka messages. Data query latency is controlled at the minute level (3 minutes in actual production environment, theoretically even shorter). Storage space and query performance can be horizontally scaled on demand to improve performance, with virtually no limit on the amount of historical data that can be stored and queried. Therefore, the near real-time Kafka data query method, device, electronic device, and computer-readable storage medium proposed in this invention can solve the problem of high resource consumption when querying Kafka data. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 The flowchart of the near real-time Kafka data query method according to Embodiment 1 of the present invention is shown;

[0045] Figure 2 This illustrates a flowchart of importing Kafka data into a distributed file system according to Embodiment 1 of the present invention.

[0046] Figure 3 The flowchart of constructing a Hive external table according to Embodiment 1 of the present invention is shown;

[0047] Figure 4 The diagram shows the functional modules of the Kafka data near real-time query device according to Embodiment 3 of the present invention.

[0048] Figure 5 The diagram shows the structural composition of the electronic device that implements the near real-time Kafka data query method according to Embodiment 5 of the present invention. Detailed Implementation

[0049] The present disclosure will be further described below with reference to the embodiments shown in the accompanying drawings.

[0050] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0051] This invention proposes a near real-time query method for Kafka data based on big data technology. Using distributed storage such as HDFS and Hive as a foundation, and combining it with a Kafka OFfest continuity verification method, Kafka data is mapped to a Hive table, enabling near real-time Kafka data querying. Compared to traditional methods, this near real-time Kafka data query technology offers greater query flexibility and reduces system resource consumption, demonstrating significant potential and application prospects in scenarios involving querying and tracing Kafka data.

[0052] Example 1

[0053] like Figure 1 As shown, this invention proposes a near real-time query method for Kafka data, including the following steps:

[0054] S1. Add a preset message header field to the Kafka data, treat the message body of the Kafka data as an independent field, and reorganize the message header field and the independent field to obtain updated Kafka data;

[0055] In this embodiment of the invention, the message header fields include information such as message partition, offset, timestamp, topic, and key. Necessary Kafka message header fields are added for data verification and analysis. Specifically, a partition is a physical grouping of a topic; a topic can be divided into multiple partitions, and each partition is an ordered queue. Offset refers to the consumer's displacement; the displacement of each message within a partition is fixed, but the displacement of the consumer consuming that partition will continuously advance as consumption progresses, but it can never exceed the displacement of the latest message in that partition. Timestamp is used to record the message sending timestamp. Topic is a logical concept representing a type of message, typically used to distinguish actual business messages. Key is the message key used when partitioning a message, determining which partition under a given topic the message is stored in.

[0056] In detail, the message body corresponding to the Kafka data is stored as an independent field, and necessary Kafka message header fields are added for data verification and analysis. The message body is used to store the actual message data.

[0057] In this embodiment of the invention, updating the Kafka data involves treating the message body of the existing Kafka data as an independent field and adding necessary message header fields to form a complete message format for subsequent import of the Kafka data into HDFS for storage. HDFS is a distributed file system that provides high-throughput data access, making it ideal for applications with large-scale datasets. An HDFS cluster consists of a NameNode and several DataNodes. The NameNode acts as the master server, managing the file system's namespace and client access operations to files; the DataNodes in the cluster manage the stored data.

[0058] In this embodiment of the invention, the step of recombining the message header fields and the independent fields to obtain updated Kafka data includes:

[0059] Use the aforementioned message header field as the message header for updating Kafka data;

[0060] Use the independent field as the message body for updating Kafka data;

[0061] The updated Kafka data is obtained by reconstructing the message format of the updated Kafka data based on the message header and the message body.

[0062] Specifically, the partition, offset, timestamp, topic, and key information from the message header fields are used as the message header for updating Kafka data. The Kafka message body is treated as an independent field and used as the message body for updating Kafka data. The message header and message body are then reassembled into a single message format to form the updated Kafka data. The message header fields can record the message sending time, message partition, message topic, etc.

[0063] Furthermore, the updated Kafka data is imported into HDFS for storage. During the import process, the structure of the updated Kafka data in HDFS must conform to the storage structure of the Hive table. Therefore, the storage structure of the Hive table needs to be analyzed to ensure that the updated Kafka data is stored in HDFS.

[0064] S2. Import the updated Kafka data into the preset distributed file system according to the preset storage structure of the Hive table to obtain the stored Kafka data;

[0065] In this embodiment of the invention, the storage structure of the Hive table is based on the table's storage structure, that is, the row delimiter, column delimiter, and file format are set according to the Hive table's storage structure. Therefore, Kafka data can be imported into HDFS using Flume according to the Hive table's storage structure. Flume is a highly available, highly reliable, distributed system provided by Cloudera for massive log collection, aggregation, and transmission. Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. The Hive data warehouse tool can map structured data files to a database table and provide SQL query functionality.

[0066] In this embodiment of the invention, the reference Figure 2 As shown, the step of importing the updated Kafka data into a preset distributed file system according to the preset Hive table storage structure to obtain stored Kafka data includes:

[0067] S21. Override the fields of the KafkaSource class in Flume to obtain the overridden KafkaSource class;

[0068] S22. Rewrite the HeaderAndBodyTextEventSerializer class in Flume according to the storage structure of the hive table to obtain the rewritten HeaderAndBodyTextEventSerializer class;

[0069] S23. Configure Flume's source component according to the rewritten KafkaSource class and the updated Kafka data to obtain configuration source information; configure Flume's channel component according to the preset buffer attributes to obtain configuration channel information; configure Flume's sink component according to the distributed file system and the rewritten HeaderAndBodyTextEventSerializer class to obtain configuration sink information.

[0070] S24. Generate a transmission file from the configuration source information, the configuration channel information, and the configuration sink information. Run Flume according to the transmission file to obtain the stored Kafka data.

[0071] In detail, the Flume classes KafkaSource and HeaderAndBodyTextEventSerializer were rewritten to retrieve information such as partition, offset, Kafka timestamp, topic, and key, and to write data to HDFS according to the Hive table data structure (conforming to Hive's row-partitioning and column-partitioning methods). The KafkaSource class is used to read information from Kafka topics, and the HeaderAndBodyTextEventSerializer class is a serializer for class header and body text events, writing the event body to the output stream and appending a newline character after each event.

[0072] Specifically, the configuration source information configures the type, source, and topic of Kafka data; the configuration channel information configures the buffer type and buffer capacity; the configuration sink information is configured based on the HDFS sink, such as agent.sinks.hdfsSink.type = hdfs. To import Kafka data into HDFS, the configuration source information, configuration channel information, and configuration sink information are used to generate a transfer file. In other words, running Flume can import Kafka data into HDFS, resulting in stored Kafka data. The Source is the component responsible for receiving data to the Flume Agent. The Source component can handle various types and formats of log data; the Sink continuously polls events in the Channel and removes them in batches, writing these events in batches to the storage or indexing system or sending them to another Flume Agent; the Channel is a buffer located between the Source and Sink, allowing the Source and Sink to operate at different rates. Therefore, the process involves using a Source to receive Kafka data, then transferring the Kafka data to a Channel, and finally using a Sink to transfer the Kafka data to HDFS for storage. In other words, Flume is used to import Kafka data into HDFS. During the import process, the Kafka message body is stored as an independent field, and necessary Kafka message header fields are added for data validation and analysis. Furthermore, the structure of this data in HDFS must conform to the storage structure of Hive tables.

[0073] Furthermore, in order to query Kafka data, the data stored in HDFS needs to be transferred to Hive. Therefore, Kafka data needs to be mapped to an external table in Hive for data querying.

[0074] S3. Map the stored Kafka data to a pre-built Hive external table to obtain mapped Kafka data. Use the offset field in the message header to verify the continuity of the mapped Kafka data to obtain continuous Kafka data.

[0075] In this embodiment of the invention, the Hive external table points to data that already exists in HDFS. When loading data and creating the table, it is not moved to the data warehouse directory, but only a link is established with the external data. When an external table is deleted, only the link is deleted, and the data is not deleted.

[0076] In this embodiment of the invention, the reference Figure 3As shown, before mapping the stored Kafka data to a pre-built Hive external table to obtain the mapped Kafka data, the process further includes:

[0077] S31. Create the storage path and delimiter field for the Kafka data;

[0078] S32. Generate the field attributes of the Hive external table based on the field attributes of the stored Kafka data;

[0079] S33. Create the Hive external table based on the storage path, the separator field, and the field attributes.

[0080] In detail, the storage path is a file that can be specified in HDFS, represented by location, which specifies the storage location of the table on HDFS; the delimiter field is divided into row delimiter and column delimiter, and the delimiter is specified by fields terminated by.

[0081] Specifically, the Kafka data is stored in HDFS according to the storage structure of Hive tables, that is, the field attributes of the external table are generated according to the field attributes of the Kafka data in HDFS. The field attributes include the column names in the external table and their corresponding data types.

[0082] For example, to create a Hive external table, you need to use the `external` keyword. For instance, to create an external table named `etable` with fields including ID, name, and age, you would use the following syntax: `create external table etable(id int, name string, age int) row format delimited fields terminated by '\t' map keys terminated by ':'location' / home / external';` Here, `row format delimited fields terminated by '\t'` defines the row separator character, and the separator character is '\t'; `map keys terminated by ':'` uses ':' to define the separator between the key and value in the map; `location' / home / external'` defines the HDFS data storage path, and ` / home / external` is the directory created in HDFS.

[0083] Furthermore, Hive is a data warehouse tool based on Hadoop that can map structured data files to a table and provide SQL-like query functionality. Therefore, after the external table is created, it is necessary to map the data in HDFS to the Hive external table, that is, load the data into the Hive external table. If the data is saved to the external.txt file, the data stored in HDFS will be mapped to the Hive external table. To import data into the external table using the command line, the command line would be: `hive(default)>load data local inpath ' / opt / module / datas / external.txt' into tableetable;` This means transferring the data stored in HDFS to the Hive external table.

[0084] In this embodiment of the invention, the data stored in HDFS is transferred to an external table in Hive. Every day at midnight, the offset of the data received the previous day is checked for continuity. If the data is continuous, it means that the data transferred to Hive is complete. If the data is not continuous, it means that the data is missing. This ensures the integrity of the data in the query system.

[0085] In this embodiment of the invention, the step of using the `offest` field in the message header to verify the continuity of the mapped Kafka data to obtain continuous Kafka data includes:

[0086] Perform continuity verification on the numeric records in the offset field corresponding to the mapped Kafka data to obtain the numeric records in the offset field;

[0087] When the digital records are continuous, the mapped Kafka data is used as the continuous Kafka data.

[0088] In detail, Kafka uses sequential read-write. Each time a message is produced, it is appended to the corresponding partition's file. The message read status is maintained by the consumer, so the offset in each partition is generally continuously increasing. Read messages are not deleted, hence the append-only sequential read-write process. After a consumer consumes a message, it produces a message on the broker that maintains the offset of each consumer, recording the offset of the message it has already read plus 1 as the new offset.

[0089] Specifically, each time a consumer sends a message, Kafka records the offset for that consumer under that topic. If there are 50 messages in total, then for each message, a corresponding offset is recorded, i.e., consumer_offsets-1, consumer_offsets-2, consumer_offsets-3, up to consumer_offsets-50. The continuity of the data is checked based on the numerical records corresponding to the offset of each message. If the corresponding offset of the data is from 1 to 50 without any missing numerical records, it means that the data is continuous; if the corresponding offset of the data is from 1 to 50 without any missing numerical records, it means that the data is not continuous. Therefore, the continuity of the data is verified based on the offset to ensure the data integrity of the query system.

[0090] Furthermore, after determining the continuity of the Kafka data transferred to Hive, full information query and analysis of the Kafka messages can be performed on the continuous Kafka data. The data query duration can be controlled within minutes, which greatly improves the efficiency of querying and analyzing Kafka data.

[0091] S4. Perform real-time query on the continuous Kafka data.

[0092] In this embodiment of the invention, Hive serves as a data warehouse, possessing a series of functions such as databases, tables, and functions. In use, Hive can be operated using visual tools, which is more intuitive and convenient than command-line operations.

[0093] In this embodiment of the invention, the real-time querying of the continuous Kafka data includes:

[0094] Using preset connection parameters, a preset Hive visualization client tool is connected to the Hive data warehouse to obtain a visualization interface for the Hive database;

[0095] The continuous Kafka data can be queried in real time through the visualization interface according to the preset SQL statement.

[0096] In detail, the connection parameters include host IP, database, password, etc.; the Hive visual client tool includes, but is not limited to, dbbeaver, which can connect to both Hive and MySQL. First, a connection is established with Hive in the Hive visual client tool by selecting "Create New Database Connection," choosing Apache Hive, configuring the connection parameters, and adding your own driver. Once the connection is successful, data query operations can be performed on Hive through the visual interface.

[0097] Specifically, you can use SQL queries to query data in Hive as needed. For example, to query the names of all people in the `etable` table, you can use the SQL statement `select name from eatable`. After a successful query, you can view the results visually in the interface. Therefore, you can use the `select` statement in SQL to query data in Hive in real time according to different needs.

[0098] Furthermore, for historical Kafka data (such as data from six months ago) transferred to Hive, it is periodically compressed into Gzip format to reduce storage space usage. At the same time, small files are merged to solve the problems of resource consumption and reduce the number of small files, thereby improving HDFS performance.

[0099] Example 2

[0100] To better understand the present invention, a second embodiment is provided below to further explain how the present invention performs real-time queries on continuous Kafka data while improving query efficiency.

[0101] In this embodiment of the invention, in order to improve query efficiency, other query engines compatible with Hive, such as Presto and Impala, can be used to perform data queries to improve query efficiency.

[0102] In this embodiment of the invention, the real-time querying of the continuous Kafka data includes:

[0103] The continuous Kafka data is queried in real time using a preset Hive-compatible query engine.

[0104] In detail, the Hive-compatible query engines include, but are not limited to, Presto and Impala. Presto is a distributed SQL query engine that separates the computation and storage layers. It does not store data and accesses various data sources (Storage) through Connector SPI. It is designed for high-speed, real-time data analysis, including complex, aggregation, join, and window functions. Impala adds SQL support and high-performance multi-user support to the underlying Hadoop-like system. It is a new computation engine implemented in C++ and Java, supports multiple file formats, and embeds the computation process into the nodes of the Hadoop infrastructure to minimize network bandwidth during computation. Impala consists of two components: FrontEnd, which receives queries and generates a distributed execution plan; and BackEnd, which is responsible for the actual execution of the plan. It extensively utilizes codegen to accelerate some repetitive computation processes and improve execution efficiency. Impala and Hive are often used in conjunction: Hive is used first for data transformation and processing; then Impala is used to perform rapid data analysis (batch processing + real-time) on the result set.

[0105] Specifically, using a Hive-compatible query engine allows for rapid data analysis, improving query efficiency and enabling flexible data analysis and querying. This allows for flexible, SQL-based querying and analysis of all Kafka message information.

[0106] Example 3

[0107] like Figure 4 As shown in the figure, this embodiment also provides a functional module diagram of a Kafka data near real-time query device.

[0108] The Kafka data near real-time query device 100 described in this embodiment can be installed in an electronic device. Depending on the functions implemented, the Kafka data near real-time query device 100 may include a field reorganization module 101, a Kafka data import module 102, a data continuity verification module 103, and a Kafka data real-time query module 104. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0109] In this embodiment, the functions of each module / unit are as follows:

[0110] The field reorganization module 101 is used to add a preset message header field to the Kafka data, treat the message body of the Kafka data as an independent field, and reorganize the message header field and the independent field to obtain updated Kafka data.

[0111] The Kafka data import module 102 is used to import the updated Kafka data into a preset distributed file system according to the preset storage structure of the Hive table, so as to obtain stored Kafka data.

[0112] The data continuity verification module 103 is used to map the stored Kafka data to a pre-built Hive external table to obtain mapped Kafka data, and to verify the continuity of the mapped Kafka data using the offset field in the message header field to obtain continuous Kafka data.

[0113] The Kafka data real-time query module 104 is used to perform real-time queries on the continuous Kafka data.

[0114] In detail, each module in the Kafka data near real-time query device 100 described in this embodiment of the invention uses the same technical means as the Kafka data near real-time query method described in Embodiment 1 and Embodiment 2, and can produce the same technical effect, which will not be repeated here.

[0115] Example 4

[0116] like Figure 5 As shown, this embodiment also provides a computer electronic device, which may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a Kafka near real-time data query program.

[0117] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing a Kafka near real-time data query program) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0118] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a Kafka near real-time data query program, but also to temporarily store data that has been output or will be output.

[0119] The communication bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0120] The communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0121] The figure only shows an electronic device with components. Those skilled in the art will understand that the structure shown in the figure does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0122] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0123] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0124] The near real-time Kafka data query program stored in the memory 11 of the electronic device is a combination of multiple instructions. When run in the processor 10, it can achieve the following:

[0125] Add a preset message header field to the Kafka data, treat the message body of the Kafka data as an independent field, and reorganize the message header field and the independent field to obtain the updated Kafka data;

[0126] The updated Kafka data is imported into a preset distributed file system according to the preset storage structure of the Hive table, thus obtaining the stored Kafka data;

[0127] The stored Kafka data is mapped to a pre-built Hive external table to obtain mapped Kafka data. The continuity of the mapped Kafka data is verified using the offset field in the message header to obtain continuous Kafka data.

[0128] Perform real-time queries on the continuous Kafka data.

[0129] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.

[0130] Furthermore, if the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0131] Example 5

[0132] This embodiment provides a storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of the near real-time Kafka data query method described above.

[0133] This program code can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 Steps of a specified function in one or more processes.

[0134] Storage media include permanent and non-permanent, removable and non-removable media, and can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by computing devices.

[0135] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application. When the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0136] It should be understood that the terms used in this way can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in a sequence other than those illustrated or described herein.

[0137] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0138] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0140] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0141] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0142] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0143] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A near real-time query method for Kafka data, characterized in that, The method includes: Add a preset message header field to the Kafka data, treat the message body of the Kafka data as an independent field, and reorganize the message header field and the independent field to obtain the updated Kafka data; The updated Kafka data is imported into a preset distributed file system according to the preset storage structure of the Hive table, thus obtaining the stored Kafka data; The stored Kafka data is mapped to a pre-built Hive external table to obtain mapped Kafka data. The continuity of the mapped Kafka data is verified using the offset field in the message header to obtain continuous Kafka data. Perform real-time queries on the continuous Kafka data; The step of recombining the message header fields and the independent fields to obtain updated Kafka data includes: Use the aforementioned message header field as the message header for updating Kafka data; Use the independent field as the message body for updating Kafka data; The updated Kafka data is obtained by reconstructing the message format of the updated Kafka data based on the message header and the message body; The step of importing the updated Kafka data into a preset distributed file system according to the preset Hive table storage structure to obtain stored Kafka data includes: By rewriting the fields of the KafkaSource class in Flume, we obtain the rewritten KafkaSource class. According to the storage structure of the hive table, the HeaderAndBodyTextEventSerializer class in Flume is rewritten for event serialization to obtain the rewritten HeaderAndBodyTextEventSerializer class; Configure Flume's source component according to the rewritten KafkaSource class and the updated Kafka data to obtain configuration source information; configure Flume's channel component according to the preset buffer attributes to obtain configuration channel information; configure Flume's sink component according to the distributed file system and the rewritten HeaderAndBodyTextEventSerializer class to obtain configuration sink information. The configuration source information, configuration channel information, and configuration sink information are used to generate a transmission file. Flume is then run based on the transmission file to obtain the stored Kafka data.

2. The near real-time Kafka data query method as described in claim 1, characterized in that, Before mapping the stored Kafka data to a pre-built Hive external table to obtain the mapped Kafka data, the process also includes: Create the storage path and delimiter field for the Kafka data; Generate the field attributes of the Hive external table based on the field attributes of the stored Kafka data; The Hive external table is created based on the storage path, the delimiter field, and the field attributes.

3. The near real-time Kafka data query method as described in claim 1, characterized in that, The step of using the `offest` field in the message header to verify the continuity of the mapped Kafka data to obtain continuous Kafka data includes: Perform continuity verification on the numeric records in the offset field corresponding to the mapped Kafka data to obtain the numeric records in the offset field; When the digital records are continuous, the mapped Kafka data is used as the continuous Kafka data.

4. The near real-time Kafka data query method as described in claim 1, characterized in that, The real-time querying of the continuous Kafka data includes: Using preset connection parameters, a preset Hive visualization client tool is connected to the Hive data warehouse to obtain a visualization interface for the Hive database; The continuous Kafka data can be queried in real time through the visualization interface according to the preset SQL statement.

5. The near real-time Kafka data query method as described in claim 1, characterized in that, The real-time querying of the continuous Kafka data includes: The continuous Kafka data is queried in real time using a preset Hive-compatible query engine.

6. A near real-time Kafka data query device, characterized in that, The device includes: The field reorganization module is used to add preset message header fields to Kafka data, treat the message body of the Kafka data as an independent field, and reorganize the message header fields and the independent fields to obtain updated Kafka data. The Kafka data import module is used to import the updated Kafka data into a preset distributed file system according to the preset storage structure of the Hive table, so as to obtain the stored Kafka data. The data continuity verification module is used to map the stored Kafka data to a pre-built Hive external table to obtain mapped Kafka data, and to verify the continuity of the mapped Kafka data using the offset field in the message header field to obtain continuous Kafka data. The Kafka data real-time query module is used for real-time querying of the continuous Kafka data; The Kafka data near real-time query device is specifically used for: Use the aforementioned message header field as the message header for updating Kafka data; Use the independent field as the message body for updating Kafka data; The updated Kafka data is obtained by reconstructing the message format of the updated Kafka data based on the message header and the message body; The Kafka data near real-time query device is specifically used for: By rewriting the fields of the KafkaSource class in Flume, we obtain the rewritten KafkaSource class. According to the storage structure of the hive table, the HeaderAndBodyTextEventSerializer class in Flume is rewritten for event serialization to obtain the rewritten HeaderAndBodyTextEventSerializer class; Configure Flume's source component according to the rewritten KafkaSource class and the updated Kafka data to obtain configuration source information; configure Flume's channel component according to the preset buffer attributes to obtain configuration channel information; configure Flume's sink component according to the distributed file system and the rewritten HeaderAndBodyTextEventSerializer class to obtain configuration sink information. The configuration source information, configuration channel information, and configuration sink information are used to generate a transmission file. Flume is then run based on the transmission file to obtain the stored Kafka data.

7. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the near real-time Kafka data query method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the near real-time Kafka data query method as described in any one of claims 1 to 5.

Images