Message processing method and system based on Kafka, client and network monitoring device

Abstract

The invention discloses a Kafka-based message processing method and system, a client and a network monitoring device, and relates to the technical field of message processing, and the disclosed Kafka-based message processing method comprises the steps of obtaining a to-be-transmitted message if a Kafka network bandwidth is abnormal; based on the feature type of the message, adopting a corresponding compression mode to perform compression to obtain a target message record, the compression modes corresponding to the messages of different feature types being different; the target message record is sent to a Kafka cluster; the technical problem that the message transmission efficiency is low under the condition that the Kafka network bandwidth is abnormal is solved, and the message transmission efficiency under the condition that the Kafka network bandwidth is abnormal is improved.

Description

Technical Field

[0001] The present application relates to the field of message processing technology, and in particular to a Kafka-based message processing method, system, client, and network monitoring device. Background Art

[0002] Currently, Kafka primarily consists of Kafka producer clients, Kafka consumer clients, and a Kafka cluster. Generally, a Kafka cluster consists of multiple proxy servers. When a Kafka producer client generates a new message, it is divided and classified into a topic. Each topic is divided into multiple partitions, which are deployed across multiple proxy servers. The Kafka producer client continuously sends messages to a specific topic. The proxy servers temporarily store messages from different topics and then forward them to the Kafka consumer client. Finally, the Kafka consumer client processes the messages generated by the Kafka producer client. However, during message transmission, if the Kafka network bandwidth is abnormal, messages cannot be transmitted in a timely manner, affecting message transmission efficiency. Summary of the Invention

[0003] The main purpose of this application is to provide a message processing method, system, client and network monitoring device based on Kafka, aiming to solve the technical problem of slow message transmission efficiency when the Kafka network bandwidth is abnormal.

[0004] To achieve the above objectives, this application proposes a Kafka-based message processing method applied to a Kafka producer client, comprising:

[0005] If the Kafka network bandwidth is abnormal, obtain the message to be transmitted;

[0006] Based on the characteristic type of the message, a corresponding compression method is used to compress the message to obtain a target message record, wherein different characteristic types of messages correspond to different compression methods;

[0007] Send the target message record to the Kafka cluster.

[0008] In one embodiment, the compressing the target message record by using a corresponding compression method based on the characteristic type of the message includes:

[0009] Performing feature recognition on the message to obtain a feature type of the message;

[0010] If the feature type is a structured feature, compress the message using a first compression method to obtain a target message record;

[0011] If the feature type is an unstructured feature, the message is compressed using a second compression method to obtain a target message record.

[0012] In one embodiment, compressing the message using a first compression method to obtain a target message record includes:

[0013] Extracting variable fields from the message and performing differential encoding on the variable fields; compressing a dictionary obtained by the differential encoding to obtain the target message record;

[0014] Alternatively, a dictionary pre-match is performed on the message to obtain a pre-match result; and the pre-match result is compressed to obtain the target message record.

[0015] In one embodiment, compressing the message using the second compression method to obtain the target message record includes:

[0016] performing incremental encoding on the message based on a sliding window method to obtain an incremental encoding result;

[0017] Extracting byte streams having differences in the incremental encoding result;

[0018] The byte streams with differences are compressed to obtain the target message record.

[0019] In one embodiment, the compressing the target message record by using a corresponding compression method based on the characteristic type of the message includes:

[0020] Compressing the message using the compression method to obtain a message body;

[0021] Determine a message header of the message body according to the compression method, wherein the message header is used to instruct the Kafka consumer client on how to decompress the message body;

[0022] The target message record is generated according to the message body and the message header.

[0023] In addition, to achieve the above objectives, this application also proposes a Kafka-based message processing method applied to a Kafka consumer client, including:

[0024] If the Kafka network bandwidth is abnormal, the content of the target topic subscribed by the Kafka consumer client is read from the Kafka cluster to obtain the target message record;

[0025] Parsing the target message record to obtain a message body and a message header in the target message record;

[0026] Determining a decompression method for the message body according to the message header;

[0027] The message body is decompressed using the decompression method to obtain the message sent by the Kafka producer client.

[0028] In addition, to achieve the above objectives, the present application also proposes a Kafka-based message processing method for a network monitoring device, comprising:

[0029] Get the write message information and read message information corresponding to each topic in the Kafka cluster;

[0030] Determine the Kafka network bandwidth based on the write and read message information corresponding to each topic;

[0031] If the Kafka network bandwidth is abnormal, a signal indicating the abnormal Kafka network bandwidth is broadcast to the Kafka producer client.

[0032] In one embodiment, the write message information includes the message sending time of the Kafka producer client and the first reception time of the feedback information after the Kafka cluster receives the message. The read message information includes the request sending time of the Kafka consumer client's read message request and the second reception time of the message feedback by the Kafka cluster. The determining of the Kafka network bandwidth according to the write information and read information corresponding to each topic includes:

[0033] Determining a first round-trip duration based on the message sending time and the first receiving time;

[0034] Determining a second round-trip duration based on the request sending time and the second receiving time;

[0035] The Kafka network bandwidth condition is determined according to the first round-trip time and / or the second round-trip time.

[0036] In one embodiment, determining the Kafka network bandwidth according to the first round-trip time and / or the second round-trip time includes:

[0037] If the first round-trip time is greater than or equal to a preset time and / or the second round-trip time is greater than or equal to the preset time, it is determined that the Kafka network bandwidth is abnormal;

[0038] If the first round-trip time is less than the preset time and the second round-trip time is less than the preset time, it is determined that the Kafka network bandwidth is normal.

[0039] In addition, to achieve the above objectives, this application also proposes a message processing system based on Kafka, including:

[0040] A network monitoring device is used to obtain the write message information and read message information corresponding to each topic in the Kafka cluster; determine the Kafka network bandwidth situation based on the write message information and read message information corresponding to each topic; if the Kafka network bandwidth is abnormal, broadcast a signal of the Kafka network bandwidth abnormality to the Kafka producer client;

[0041] The Kafka producer client is used to obtain messages to be transmitted if the Kafka network bandwidth is abnormal; based on the characteristic type of the message, it uses the corresponding compression method to compress the target message record, where different characteristic types of messages correspond to different compression methods; and sends the target message record to the Kafka cluster;

[0042] The Kafka consumer client is configured to read the content of the target topic subscribed by the Kafka consumer client from the Kafka cluster to obtain a target message record if the Kafka network bandwidth is abnormal; parse the target message record to obtain a message body and a message header in the target message record; determine a decompression method for the message body based on the message header; and decompress the message body using the decompression method to obtain a message sent by the Kafka producer client;

[0043] A Kafka cluster is used to store the target message records.

[0044] In addition, to achieve the above-mentioned purpose, the present application also proposes a Kafka producer client, comprising: a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein the computer program is configured to implement the steps of the Kafka-based message processing method as described above.

[0045] In addition, to achieve the above-mentioned purpose, the present application also proposes a Kafka consumer client, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the Kafka-based message processing method as described above.

[0046] In addition, to achieve the above-mentioned purpose, the present application also proposes a network monitoring device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the Kafka-based message processing method as described above.

[0047] In the event of abnormal Kafka network bandwidth, this application compresses the message using the corresponding compression method based on the message's characteristic type to obtain the target message record; the target message record is then sent to the target topic corresponding to the Kafka cluster. Because the corresponding compression method can be dynamically adjusted for different message characteristic types, the compression requirements of different types of messages can be met; after the message is compressed, the message data size becomes smaller, and the network bandwidth occupied during the message transmission process will also be reduced, thus improving the transmission efficiency of various types of messages in the event of abnormal Kafka network bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0050] Figure 1 This is a flow chart of an embodiment of a message processing method based on Kafka provided by this application;

[0051] Figure 2 A flowchart of another embodiment of the message processing method based on Kafka provided by this application;

[0052] Figure 3 A flowchart of another embodiment of the message processing method based on Kafka provided by this application;

[0053] Figure 4 This is a schematic diagram of the architecture of the message processing system based on Kafka in an embodiment of the present application;

[0054] Figure 5 This is a structural diagram of the Kafka producer client in an embodiment of the present application.

[0055] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0057] Currently, Kafka primarily consists of Kafka producer clients, Kafka consumer clients, and a Kafka cluster. Generally, a Kafka cluster consists of multiple proxy servers. When a Kafka producer client generates a new message, it is divided and classified into a topic. Each topic is divided into multiple partitions, which are deployed across multiple proxy servers. The Kafka producer client continuously sends messages to a specific topic. The proxy servers temporarily store messages from different topics and then forward them to the Kafka consumer client. Finally, the Kafka consumer client processes the messages generated by the Kafka producer client. However, during message transmission, if the Kafka network bandwidth is abnormal, messages cannot be transmitted in a timely manner, affecting message transmission efficiency.

[0058] In response to the above problems, this application proposes a message processing method based on Kafka. The main technical solutions include: if the Kafka network bandwidth is abnormal, obtain the message to be transmitted; based on the characteristic type of the message, use the corresponding compression method to compress it to obtain the target message record, wherein the compression method corresponding to messages of different characteristic types is different; send the target message record to the Kafka cluster. Since it is possible to dynamically adjust the corresponding compression method for compression according to the characteristic type of different messages, the compression requirements of different types of messages are met, the compression method is enriched, and the compression method is no longer fixed, and can also be dynamically adjusted according to the characteristic type of the message; after the message is compressed, the amount of message data becomes smaller, and the network bandwidth occupied during the message transmission process will become smaller, thereby improving the transmission efficiency of various types of messages in the case of abnormal Kafka network bandwidth.

[0059] It should be noted that the Kafka-based message processing method proposed in this application can be applied to any scenario that requires the use of Kafka distributed messages. This application does not specifically limit the use scenario of the Kafka-based message processing method.

[0060] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0061] Based on this, the embodiment of the present application provides a message processing method based on Kafka, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of a Kafka-based message processing method applied to a Kafka producer client. In this embodiment, the Kafka-based message processing method includes steps S10 to S30:

[0062] Step S10: If the Kafka network bandwidth is abnormal, obtain the message to be transmitted.

[0063] Kafka network bandwidth anomalies can be caused by external attacks, such as virus attacks, or by a sudden increase in message volume, which increases the demand for network bandwidth. Other causes can also occur.

[0064] The Kafka producer client can obtain the write message information corresponding to each topic in the Kafka cluster; based on the write message information corresponding to each topic, the Kafka network bandwidth situation can be determined to determine whether the Kafka network bandwidth is abnormal. Alternatively, the network monitoring device can obtain the write message information and read message information corresponding to each topic in the Kafka cluster; based on the write message information and read message information corresponding to each topic, the Kafka network bandwidth situation can be determined; if the Kafka network bandwidth is abnormal, a signal indicating the Kafka network bandwidth abnormality is broadcast to the Kafka producer client, allowing the Kafka producer client to determine whether the Kafka network bandwidth is abnormal.

[0065] In a feasible implementation, the write message information includes the message sending time of the Kafka producer client and the first receiving time of the feedback information after the Kafka cluster receives the message, and the read message information includes the request sending time of the Kafka consumer client's read message request and the second receiving time when the Kafka consumer client receives the message feedback from the Kafka cluster based on the read message request. According to the write information and read information corresponding to each topic, the Kafka network bandwidth situation is determined, including: determining the first round-trip time according to the message sending time and the first receiving time; determining the second round-trip time according to the request sending time and the second receiving time; determining the Kafka network bandwidth situation according to the first round-trip time and / or the second round-trip time. Among them, the first round-trip time can be determined according to the difference between the message sending time and the first receiving time; and the second round-trip time can be determined according to the difference between the request sending time and the second receiving time. Determining the Kafka network bandwidth based on the first round-trip time and / or the second round-trip time includes: determining that the Kafka network bandwidth is abnormal if the first round-trip time is greater than or equal to a preset time and / or the second round-trip time is greater than or equal to a preset time; and determining that the Kafka network bandwidth is normal if the first round-trip time is less than the preset time and the second round-trip time is less than the preset time. Performing Kafka network bandwidth anomaly detection through a network monitoring device and broadcasting a Kafka network bandwidth anomaly signal to the Kafka producer client not only enables Kafka network bandwidth anomaly detection, but also reduces the data processing pressure of the Kafka producer client and improves the data transmission efficiency of the Kafka producer client.

[0066] In another feasible implementation, the written message information includes the packet loss rate of the message sent by the Kafka producer client, and the read message information includes the packet loss rate of the message received by the Kafka consumer client; the Kafka network bandwidth situation is determined based on the packet loss rate of the message sent by the Kafka producer client and / or the packet loss rate of the message received by the Kafka consumer client. Specifically, the Kafka network bandwidth situation is determined based on the packet loss rate of the message sent by the Kafka producer client and / or the packet loss rate of the message received by the Kafka consumer client: if the packet loss rate of the message sent by the Kafka producer client is greater than or equal to the preset packet loss rate and the packet loss rate of the message received by the Kafka consumer client is greater than or equal to the preset packet loss rate, it is determined that the Kafka network bandwidth is abnormal; if the packet loss rate of the message sent by the Kafka producer client is less than the preset packet loss rate or the packet loss rate of the message received by the Kafka consumer client is greater than or equal to the preset packet loss rate, it is determined that the Kafka network bandwidth is normal.

[0067] It should be noted that when the Kafka network bandwidth is normal, messages are transmitted directly without compression, reducing compression performance losses. When the Kafka network bandwidth is abnormal, messages need to be compressed to reduce the message data size, thereby reducing the network bandwidth occupied and improving the transmission efficiency of various types of messages in the event of abnormal Kafka network bandwidth.

[0068] Retrieving messages to be transmitted refers to obtaining messages to be transmitted to the corresponding topic in the Kafka cluster. These messages can be generated by Kafka producer clients.

[0069] Step S20: Based on the characteristic type of the message, a corresponding compression method is used to compress the message to obtain a target message record, wherein different characteristic types of messages have different corresponding compression methods.

[0070] The feature types of the message include structured features and unstructured features, etc., and can also be divided into other feature types.

[0071] Compression methods can be categorized into different types based on message characteristics. For example, GZIP, Snappy, and LZ4 compression methods can be used, each employing a different compression algorithm. Messages with different characteristics correspond to different compression methods. Dynamically selecting the appropriate compression method based on the message's characteristics can meet compression requirements for different messages, reduce the data size of messages with different characteristics, and improve transmission efficiency.

[0072] Among them, messages are stored in the Kafka cluster in the form of records. Each target message record contains a message body and a message header. After the producer encapsulates the message into a target message record, it sends it to the Kafka cluster for storage through the specified topic.

[0073] In one feasible implementation, a mapping relationship between characteristic types of various messages and corresponding compression methods can be pre-established, and the corresponding compression method can be obtained based on the characteristic type of the message and the mapping relationship. After obtaining the corresponding compression method, the message is compressed using the corresponding compression method to obtain the target message record.

[0074] Step S30: Send the target message record to the Kafka cluster.

[0075] Among them, the Kafka cluster consists of multiple proxy servers. When the Kafka producer client generates a new message, it will be divided and classified into a topic. Each topic is divided into multiple partitions, which are deployed on multiple proxy servers. The Kafka producer client continuously sends messages to a certain type of topic. The proxy server temporarily stores messages of different topics and then forwards them to the Kafka consumer client. The Kafka consumer client needs to subscribe to the topic in advance and obtain messages of the corresponding topic. Finally, the Kafka consumer client obtains messages of the subscribed topic and processes the obtained messages.

[0076] In a feasible implementation, before sending the target message record to the Kafka cluster, it is necessary to establish a communication connection relationship between the Kafka producer client and the Kafka cluster. Based on the communication connection relationship, the target message record is sent to the Kafka cluster.

[0077] In practice, Kafka producer clients communicate with proxy servers in a Kafka cluster via TCP to send messages. These clients send messages to a proxy server in the Kafka cluster, typically using either asynchronous or synchronous transmission. When sending messages, the producer selects a specific partition, typically partitioned by message key or automatically assigned by Kafka. If a proxy server fails, other proxies elect to take over data replication from the failed proxy server. Because message transmission utilizes the reliable transmission mechanism of TCP, data integrity is ensured.

[0078] In this embodiment, when Kafka network bandwidth is abnormal, the target message record is compressed using the corresponding compression method based on the message's characteristic type. The target message record is then sent to the target topic corresponding to the Kafka cluster. Because the corresponding compression method can be dynamically adjusted according to the characteristic type of the message, the compression requirements of different types of messages can be met. After the message is compressed, the message data size is reduced, and the network bandwidth occupied during the message transmission process is also reduced. This improves the transmission efficiency of various types of messages in the event of Kafka network bandwidth abnormalities.

[0079] In a feasible implementation, step S20 may include steps S21 to S23:

[0080] Step S21, performing feature recognition on the message to obtain a feature type of the message;

[0081] In one implementation, a message can be sequentially subjected to magic number checking, entropy analysis, and JSON structured checking to determine its characteristic type. Magic number checking works by identifying specific byte sequences in the message header, typically 2-8 bytes, to determine the binary protocol type. Specifically, a magic number dictionary for common binary protocols can be maintained. The first 8 bytes of the message are scanned and compared against the dictionary key. A match is found, and the message is directly classified as the corresponding binary type. These binary messages are subsequently prioritized for transmission and processing. Entropy analysis distinguishes encrypted / compressed data from structured / text data. The message's entropy value is calculated. If it exceeds a preset value, the message is classified as a compressed / encrypted byte stream and prioritized for transmission and processing. If the entropy value is less than or equal to the preset value, JSON structured checking is performed. JSON structured checking can include checking whether the first and last characters are { / } or [ / ]. Quote pairing ("key":"value"), comma delimiters, and nested structure consistency can be verified. Frequently used JSON field names (such as "timestamp" and "event_id") can be preloaded to reduce deep parsing overhead.

[0082] For example, if a message is in JSON format, its feature type is structured; if a message is in text format, its feature type is structured; and if a message is in byte stream format, its feature type is unstructured. Messages can be pre-classified to determine their feature types.

[0083] Step S22: If the feature type is a structured feature, compress the message using a first compression method to obtain a target message record;

[0084] In one feasible method, if the feature type is a structured feature, the variable fields in the message are extracted, and the variable fields are differentially encoded; the dictionary obtained by differential encoding is compressed to obtain the target message record. Specifically, the fields in the message can be located using a standard path expression, and the variable fields in the message can be obtained by comparing historical json format message samples. The variable fields are differentially encoded to obtain a differentially encoded dictionary, and the differentially encoded dictionary is compressed to obtain the target message record. For example, assuming that the variable field sequence is [100, 101, 105, 110], the differentially encoded dictionary is [100, 1, 4, 5]. In this way, the message in json format can be compressed to obtain the target message record.

[0085] In another feasible manner, if the feature type is a structured feature, the message can be pre-matched with a dictionary to obtain a pre-match result; the pre-match result can be compressed to obtain the target message record. Specifically, a hash value can be calculated for the input message, and the hash table can be queried. If the same hash value exists, the historical position is located and compared byte by byte; if the match is successful, the offset and the match length are output to obtain the pre-match result. The offset refers to the offset of the matching position from the current window, and the match length indicates the length of the repeated string. For example, assuming the message is "abcd_efghabcd_gh", the pre-match finds that "abcd" is repeated at offset 9, then the offset is determined to be 9 and the match length is 4. In this way, messages in byte stream format can be compressed to obtain the target message record.

[0086] Step S23: If the feature type is an unstructured feature, compress the message using a second compression method to obtain a target message record.

[0087] In one feasible method, if the feature type is an unstructured feature, the message is incrementally encoded based on a sliding window to obtain an incremental encoding result; the byte streams with differences in the incremental encoding result are extracted; and the byte streams with differences are compressed. A fixed-size historical data window is maintained, and old data is overwritten when new messages enter, forming a dynamically updated reference library. A hash value is calculated for each 4-byte block of the new message, and a match is searched for in the hash table of the window: if a match is found, the offset and match length are recorded. If there is no match, the current byte is regarded as a new literal. In this way, messages in text format can be compressed to obtain target message records.

[0088] The above compression method essentially applies the Per-Title compression principle. When network bandwidth is abnormal, this application compresses or simplifies the key information in the message to extract key information or reduce the size of the data set, thereby reducing the amount of data recorded in the target message and improving the transmission efficiency of the message.

[0089] In this embodiment, appropriate compression methods are selected for the characteristic types of different messages to obtain target message records, which meet the compression requirements of different messages and can maximize the reduction of the data volume of different messages, thereby reducing the network bandwidth occupied by the message and improving the transmission efficiency of the message.

[0090] In a feasible implementation, step S20 may include steps A21 to A23:

[0091] Step A21, compressing the message using the compression method to obtain a message body;

[0092] Step A22: determining a message header of the message body according to the compression method, wherein the message header is used to instruct the Kafka consumer client on a decompression method of the message body;

[0093] Among them, the message header can be obtained by encoding the name of the compression method. According to the message header, the compression method used by the message body can be determined. The subsequent Kafka consumer client can obtain the decompression method corresponding to the compression method based on the message header and decompress the message body, thereby achieving smooth decompression of the message body.

[0094] Step A23: Generate the target message record according to the message body and the message header.

[0095] In one practicable manner, the message header and the message body may be combined and concatenated according to a set arrangement order to obtain a target message record.

[0096] In this embodiment, a corresponding message header is set for the message body to instruct the Kafka consumer client on how to decompress the message body, facilitating the smooth decompression of subsequent messages. Furthermore, the corresponding decompression method can be set for different message bodies to meet the decompression requirements of messages with different characteristics, thereby improving message decompression efficiency.

[0097] Based on the same inventive concept, the embodiment of the present application provides a message processing method based on Kafka, referring to Figure 2 , Figure 2 This is a flow chart of an embodiment of a Kafka-based message processing method applied to a Kafka consumer client. In this embodiment, the Kafka-based message processing method includes steps S110 to S140:

[0098] Step S110: If the Kafka network bandwidth is abnormal, read the content of the target topic subscribed by the Kafka consumer client from the Kafka cluster to obtain the target message record;

[0099] Kafka network bandwidth anomalies can be caused by external attacks, such as virus attacks, or by a sudden increase in message volume, which increases the demand for network bandwidth. Other causes can also occur.

[0100] The Kafka consumer client can obtain the read message information corresponding to each topic in the Kafka cluster; based on the read message information corresponding to each topic, the Kafka network bandwidth situation can be determined to determine whether the Kafka network bandwidth is abnormal. Alternatively, the network monitoring device can obtain the write message information and read message information corresponding to each topic in the Kafka cluster; based on the write message information and read message information corresponding to each topic, the Kafka network bandwidth situation can be determined; if the Kafka network bandwidth is abnormal, a signal indicating the Kafka network bandwidth abnormality is broadcast to the Kafka consumer client, allowing the Kafka consumer client to determine whether the Kafka network bandwidth is abnormal.

[0101] In a feasible implementation, the write message information includes the message sending time of the Kafka producer client and the first receiving time of the feedback information after the Kafka cluster receives the message. The read message information includes the request sending time of the Kafka consumer client's read message request and the second receiving time of the message fed back by the Kafka cluster. According to the write information and read information corresponding to each topic, the determination of the Kafka network bandwidth includes: determining the first round-trip time according to the message sending time and the first receiving time; determining the second round-trip time according to the request sending time and the second receiving time; and determining the Kafka network bandwidth according to the first round-trip time and / or the second round-trip time. Among them, the first round-trip time can be determined according to the difference between the message sending time and the first receiving time; and the second round-trip time can be determined according to the difference between the request sending time and the second receiving time. Determining the Kafka network bandwidth based on the first round-trip time and / or the second round-trip time includes: determining that the Kafka network bandwidth is abnormal if the first round-trip time is greater than or equal to a preset time and / or the second round-trip time is greater than or equal to a preset time; and determining that the Kafka network bandwidth is normal if the first round-trip time is less than the preset time and the second round-trip time is less than the preset time. Detecting Kafka network bandwidth anomalies through a network monitoring device and broadcasting a signal indicating the Kafka network bandwidth anomaly to a Kafka consumer client not only enables detection of Kafka network bandwidth anomalies but also reduces data processing pressure on the Kafka consumer client and improves data transmission efficiency of the Kafka consumer client.

[0102] In another feasible implementation, the written message information includes the packet loss rate of the message sent by the Kafka producer client, and the read message information includes the packet loss rate of the message received by the Kafka consumer client; the Kafka network bandwidth situation is determined based on the packet loss rate of the message sent by the Kafka producer client and / or the packet loss rate of the message received by the Kafka consumer client. Specifically, the Kafka network bandwidth situation is determined based on the packet loss rate of the message sent by the Kafka producer client and / or the packet loss rate of the message received by the Kafka consumer client: if the packet loss rate of the message sent by the Kafka producer client is greater than or equal to the preset packet loss rate and the packet loss rate of the message received by the Kafka consumer client is greater than or equal to the preset packet loss rate, it is determined that the Kafka network bandwidth is abnormal; if the packet loss rate of the message sent by the Kafka producer client is less than the preset packet loss rate or the packet loss rate of the message received by the Kafka consumer client is greater than or equal to the preset packet loss rate, it is determined that the Kafka network bandwidth is normal.

[0103] It should be noted that when the Kafka network bandwidth is normal, the message is processed directly without decompression, reducing the performance loss of decompression. When the Kafka network bandwidth is abnormal, it is necessary to select an appropriate decompression method based on the characteristics of the message type.

[0104] Step S120, parsing the target message record to obtain the message body and message header in the target message record;

[0105] Step S130, determining a decompression method for the message body according to the message header;

[0106] In one feasible implementation, the message header can be obtained by encoding the name of the compression method. The compression method used by the message body can be determined based on the message header. The Kafka consumer client can obtain the decompression method corresponding to the compression method based on the message header and decompress the message body, thereby achieving smooth decompression of the message body. For example, if the compression method is determined to be GZIP based on the message header, the decompression method is GZIP decompression; if the compression method is determined to be Snappy based on the message header, the decompression method is Snappy decompression; if the compression method is determined to be LZ4 based on the message header, the decompression method is LZ4 decompression. The compression method and decompression method are corresponding.

[0107] Step S140: decompress the message body using the decompression method to obtain the message sent by the Kafka producer client.

[0108] The above-mentioned GZIP decompression method, Snappy decompression method, and LZ4 decompression method are conventional technologies in this field and will not be described in detail here. The innovation of this application is that during the Kafka message transmission process, when facing abnormal Kafka network bandwidth, corresponding compression and decompression methods are added for messages of different feature types to improve the transmission efficiency of messages.

[0109] In this embodiment, due to the abnormality of the Kafka network bandwidth, different message bodies are decompressed using corresponding decompression methods to meet the decompression requirements of different types of messages and improve the decompression efficiency of the messages.

[0110] Based on the same inventive concept, the embodiment of the present application provides a message processing method based on Kafka, referring to Figure 3 , Figure 3 This is a flow chart of an embodiment of a Kafka-based message processing method for a network monitoring device of the present application. In this embodiment, the Kafka-based message processing method includes steps S210 to S230:

[0111] Step S210: Obtain the write message information and read message information corresponding to each topic in the Kafka cluster;

[0112] The write message information corresponding to each topic refers to the message information written by the Kafka producer client to the corresponding topic in the corresponding Kafka cluster, which can be the round-trip time of the message, packet loss rate, etc.

[0113] The read message information corresponding to each topic refers to the information of the message read by the Kafka consumer client from the corresponding topic in the corresponding Kafka cluster, which can be the round-trip time of the message, packet loss rate, etc.

[0114] Step S220: Determine the Kafka network bandwidth based on the write message information and read message information corresponding to each topic;

[0115] Kafka network bandwidth conditions include abnormal and normal Kafka network bandwidth. Abnormal Kafka network bandwidth can be caused by external attacks, such as virus attacks, or by a sudden increase in message volume, which increases bandwidth demand. Other factors can also cause abnormal Kafka network bandwidth.

[0116] Step S230: If the Kafka network bandwidth is abnormal, a signal indicating the abnormal Kafka network bandwidth is broadcast to the Kafka producer client.

[0117] In other embodiments, the network monitoring device may also broadcast a signal indicating abnormal Kafka network bandwidth to the Kafka consumer client.

[0118] In this embodiment, the network bandwidth is monitored for abnormalities through a network monitoring device, and a signal indicating the abnormal Kafka network bandwidth is broadcast to the Kafka producer client through broadcasting, so that the Kafka producer client can know the current network bandwidth situation and dynamically select a suitable compression algorithm for compression to avoid large-scale network storms.

[0119] In a feasible implementation, a mapping relationship between the write message information, read message information and Kafka network bandwidth situation corresponding to each topic can be pre-established, and the Kafka network bandwidth situation can be determined based on the mapping relationship.

[0120] In a feasible implementation, the write message information includes the message sending time of the Kafka producer client and the first receiving time of the feedback information after the Kafka cluster receives the message. The read message information includes the request sending time of the Kafka consumer client's read message request and the second receiving time of the message fed back by the Kafka cluster. According to the write information and read information corresponding to each topic, the determination of the Kafka network bandwidth includes: determining the first round-trip time according to the message sending time and the first receiving time; determining the second round-trip time according to the request sending time and the second receiving time; and determining the Kafka network bandwidth according to the first round-trip time and / or the second round-trip time. Among them, the first round-trip time can be determined according to the difference between the message sending time and the first receiving time; and the second round-trip time can be determined according to the difference between the request sending time and the second receiving time. Determining the Kafka network bandwidth based on the first round-trip time and / or the second round-trip time includes: determining that the Kafka network bandwidth is abnormal if the first round-trip time is greater than or equal to a preset time and / or the second round-trip time is greater than or equal to a preset time; and determining that the Kafka network bandwidth is normal if the first round-trip time is less than the preset time and the second round-trip time is less than the preset time. Performing Kafka network bandwidth anomaly detection through a network monitoring device and broadcasting a Kafka network bandwidth anomaly signal to a Kafka consumer client and a Kafka producer client not only enables Kafka network bandwidth anomaly detection, but also reduces the data processing pressure of the Kafka consumer client and the Kafka producer client, thereby improving the data transmission efficiency of the Kafka consumer client and the Kafka producer client. The preset time can be set according to actual conditions. When the first round-trip time is greater than or equal to the preset time and / or the second round-trip time is greater than or equal to the preset time, it indicates that the Kafka network bandwidth is abnormal. When the first round-trip time is less than the preset time and the second round-trip time is less than the preset time, it indicates that the Kafka network bandwidth is normal.

[0121] In another feasible implementation, the written message information includes the packet loss rate of the message sent by the Kafka producer client, and the read message information includes the packet loss rate of the message received by the Kafka consumer client; the Kafka network bandwidth situation is determined based on the packet loss rate of the message sent by the Kafka producer client and / or the packet loss rate of the message received by the Kafka consumer client. Specifically, the Kafka network bandwidth situation is determined based on the packet loss rate of the message sent by the Kafka producer client and / or the packet loss rate of the message received by the Kafka consumer client: if the packet loss rate of the message sent by the Kafka producer client is greater than or equal to the preset packet loss rate and the packet loss rate of the message received by the Kafka consumer client is greater than or equal to the preset packet loss rate, it is determined that the Kafka network bandwidth is abnormal; if the packet loss rate of the message sent by the Kafka producer client is less than the preset packet loss rate or the packet loss rate of the message received by the Kafka consumer client is greater than or equal to the preset packet loss rate, it is determined that the Kafka network bandwidth is normal.

[0122] It should be noted that when the Kafka network bandwidth is normal, messages are processed directly without compression or decompression, reducing the performance loss of compression and decompression. When the Kafka network bandwidth is abnormal, it is necessary to dynamically select the appropriate compression method for compression and decompression based on the characteristics of the message type.

[0123] It should be noted that the above examples are only used to understand this application and do not constitute a limitation on the message processing method based on Kafka in this application. More simple transformations based on this technical concept are all within the scope of protection of this application.

[0124] Based on the same inventive concept, this application also provides a message processing system based on Kafka, please refer to Figure 4 , the Kafka-based message processing system includes:

[0125] A network monitoring device is used to obtain the write message information and read message information corresponding to each topic in the Kafka cluster; determine the Kafka network bandwidth situation based on the write message information and read message information corresponding to each topic; if the Kafka network bandwidth is abnormal, broadcast a signal of the Kafka network bandwidth abnormality to the Kafka producer client;

[0126] Optionally, the write message information includes the message sending time of the Kafka producer client and the first receiving time of the feedback information after the Kafka cluster receives the message, and the read message information includes the request sending time of the Kafka consumer client's read message request and the second receiving time of the message feedback by the Kafka cluster. The network monitoring device is also used to: determine the first round-trip time based on the message sending time and the first receiving time; determine the second round-trip time based on the request sending time and the second receiving time; determine the Kafka network bandwidth situation based on the first round-trip time and / or the second round-trip time.

[0127] Optionally, the network monitoring device is also used to: determine that the Kafka network bandwidth is abnormal if the first round-trip time is greater than or equal to the preset time and / or the second round-trip time is greater than or equal to the preset time; if the first round-trip time is less than the preset time and the second round-trip time is less than the preset time, determine that the Kafka network bandwidth is normal.

[0128] The Kafka producer client is used to obtain messages to be transmitted if the Kafka network bandwidth is abnormal; based on the characteristic type of the message, it uses the corresponding compression method to compress the target message record, where different characteristic types of messages correspond to different compression methods; and sends the target message record to the Kafka cluster;

[0129] Optionally, the Kafka producer client is also used to: perform feature identification on the message to obtain the feature type of the message; if the feature type is a structured feature, use a first compression method to compress the message to obtain a target message record; if the feature type is an unstructured feature, use a second compression method to compress the message to obtain a target message record.

[0130] Optionally, the Kafka producer client is also used to: extract variable fields in the message and perform differential encoding on the variable fields; compress the dictionary obtained by differential encoding to obtain the target message record; or perform dictionary pre-matching on the message to obtain a pre-matching result; compress the pre-matching result to obtain the target message record.

[0131] Optionally, the Kafka producer client is also used to: incrementally encode the message based on a sliding window method to obtain an incremental encoding result; extract the byte stream with differences in the incremental encoding result; and compress the byte stream with differences to obtain the target message record.

[0132] Optionally, the Kafka producer client is also used to: compress the message using the compression method to obtain a message body; determine the message header of the message body based on the compression method, wherein the message header is used to instruct the Kafka consumer client on how to decompress the message body; and generate the target message record based on the message body and the message header.

[0133] The Kafka consumer client is configured to read the content of the target topic subscribed by the Kafka consumer client from the Kafka cluster to obtain a target message record if the Kafka network bandwidth is abnormal; parse the target message record to obtain a message body and a message header in the target message record; determine a decompression method for the message body based on the message header; and decompress the message body using the decompression method to obtain a message sent by the Kafka producer client;

[0134] A Kafka cluster is used to store the target message records.

[0135] It should be noted that there can be multiple Kafka producer clients and Kafka consumer clients in this application, for example Figure 4 For example, there are two Kafka producer clients and two Kafka consumer clients. Each Kafka producer client and Kafka consumer client can transmit messages of the same feature type, or messages of different feature types. For example, Kafka producer client 1 and Kafka consumer client 1 can transmit JSON format messages, and Kafka producer client 2 and Kafka consumer client 2 can transmit text format messages.

[0136] A Kafka cluster can include one or more broker servers.

[0137] The network monitoring device can send a signal of Kafka network bandwidth anomaly to the Kafka producer client in a broadcast manner, so that the Kafka producer client can dynamically select a suitable compression method for messages of different feature types. The subsequent Kafka consumer client can also decompress based on the decompression method indicated by the message header in the target message record.

[0138] The Kafka-based message processing system provided by this application adopts the Kafka-based message processing method of the above-mentioned embodiment, which can solve the technical problem of slow message transmission efficiency when the Kafka network bandwidth is abnormal. Compared with the prior art, the beneficial effects of the Kafka-based message processing system provided by this application are the same as the beneficial effects of the Kafka-based message processing method provided by the above-mentioned embodiment, and the other technical features of the Kafka-based message processing system are the same as the features disclosed in the above-mentioned embodiment method, which are not repeated here.

[0139] Based on the same inventive concept, the present application provides a Kafka producer client, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the Kafka-based message processing method in the above embodiment.

[0140] Reference below Figure 5 , which shows a schematic diagram of the structure of a Kafka producer client suitable for implementing an embodiment of the present application. The Kafka producer client in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Application Descriptions, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 5 What is shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0141] like Figure 5As shown, the Kafka producer client may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM) 1004. Various programs and data required for the operation of the Kafka producer client are also stored in the random access memory 1004. The processing device 1001, the read-only memory 1002 and the random access memory 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the Kafka producer client to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a Kafka producer client with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0142] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0143] The Kafka producer client provided in this application adopts the Kafka-based message processing method in the above-mentioned embodiment, which can solve the technical problem of slow message transmission efficiency when the Kafka network bandwidth is abnormal. Compared with the prior art, the beneficial effects of the Kafka producer client provided in this application are the same as the beneficial effects of the Kafka-based message processing method provided in the above-mentioned embodiment, and the other technical features of the Kafka producer client are the same as those disclosed in the previous embodiment method, which are not repeated here.

[0144] Based on the same inventive concept, the present application provides a Kafka consumer client, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the Kafka-based message processing method in the above embodiment.

[0145] The Kafka consumer client in the embodiments of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Application Description, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., as well as fixed terminals such as digital TVs, desktop computers, etc.

[0146] The Kafka consumer client may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for the operation of the Kafka consumer client are also stored in the random access memory 1004. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, hard disk, etc.; and communication devices 1009. The communication devices 1009 can allow the Kafka consumer client to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a Kafka consumer client with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0147] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0148] The Kafka consumer client provided in this application utilizes the Kafka-based message processing method of the aforementioned embodiment, which can resolve the technical issue of slow message transmission efficiency when Kafka network bandwidth is abnormal. Compared with the prior art, the beneficial effects of the Kafka consumer client provided in this application are the same as those of the Kafka-based message processing method provided in the aforementioned embodiment, and the other technical features of the Kafka consumer client are the same as those disclosed in the aforementioned embodiment, which are not further described here.

[0149] Based on the same inventive concept, the present application provides a network monitoring device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the Kafka-based message processing method in the above embodiment.

[0150] The network monitoring device in the embodiments of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Application Description, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., as well as fixed terminals such as digital TVs, desktop computers, etc.

[0151] The network monitoring device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the network monitoring device. The processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007, such as a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008, such as a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003, such as a magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the network monitoring device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a network monitoring device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0152] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0153] The network monitoring device provided by this application utilizes the Kafka-based message processing method of the aforementioned embodiment, which can resolve the technical issue of slow message transmission efficiency when Kafka network bandwidth is abnormal. Compared with the prior art, the beneficial effects of the network monitoring device provided by this application are the same as those of the Kafka-based message processing method of the aforementioned embodiment, and the other technical features of this network monitoring device are the same as those disclosed in the aforementioned embodiment, and are not further described here.

[0154] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A message processing method based on Kafka, characterized in that: Applied to the Kafka producer client, the Kafka-based message processing includes: If the Kafka network bandwidth is abnormal, obtain the message to be transmitted; Based on the characteristic type of the message, a corresponding compression method is used to compress the message to obtain a target message record, wherein different characteristic types of messages correspond to different compression methods; Send the target message record to the Kafka cluster.

2. The message processing method based on Kafka according to claim 1, characterized in that: The target message record is obtained by compressing the message using a corresponding compression method based on the characteristic type of the message. Performing feature recognition on the message to obtain a feature type of the message; If the feature type is a structured feature, compress the message using a first compression method to obtain a target message record; If the feature type is an unstructured feature, the message is compressed using a second compression method to obtain a target message record.

3. The message processing method based on Kafka according to claim 2, characterized in that: The message is compressed using a first compression method to obtain a target message record including: Extracting variable fields from the message and performing differential encoding on the variable fields; compressing a dictionary obtained by the differential encoding to obtain the target message record; Alternatively, a dictionary pre-match is performed on the message to obtain a pre-match result; and the pre-match result is compressed to obtain the target message record.

4. The message processing method based on Kafka according to claim 2, characterized in that The compressing the message using the second compression method to obtain the target message record includes: performing incremental encoding on the message based on a sliding window method to obtain an incremental encoding result; Extracting byte streams having differences in the incremental encoding result; The byte streams with differences are compressed to obtain the target message record.

5. The message processing method based on Kafka according to claim 2, characterized in that: The target message record is obtained by compressing the message using a corresponding compression method based on the characteristic type of the message. Compressing the message using the compression method to obtain a message body; Determine a message header of the message body according to the compression method, wherein the message header is used to instruct the Kafka consumer client on how to decompress the message body; The target message record is generated according to the message body and the message header.

6. A message processing method based on Kafka, characterized in that: Applied to the Kafka consumer client, the Kafka-based message processing includes: If the Kafka network bandwidth is abnormal, the content of the target topic subscribed by the Kafka consumer client is read from the Kafka cluster to obtain the target message record; Parsing the target message record to obtain a message body and a message header in the target message record; Determining a decompression method for the message body according to the message header; The message body is decompressed using the decompression method to obtain the message sent by the Kafka producer client.

7. A message processing method based on Kafka, characterized in that: Applied to a network monitoring device, the Kafka-based message processing method includes: Get the write message information and read message information corresponding to each topic in the Kafka cluster; Determine the Kafka network bandwidth based on the write and read message information corresponding to each topic; If the Kafka network bandwidth is abnormal, a signal indicating the abnormal Kafka network bandwidth is broadcast to the Kafka producer client.

8. The message processing method based on Kafka according to claim 7, characterized in that: The write message information includes the message sending time of the Kafka producer client and the first receiving time of the feedback information after the Kafka cluster receives the message. The read message information includes the request sending time of the Kafka consumer client's read message request and the second receiving time of the message feedback by the Kafka cluster. The determination of the Kafka network bandwidth according to the write information and read information corresponding to each topic includes: Determining a first round-trip duration based on the message sending time and the first receiving time; Determining a second round-trip duration based on the request sending time and the second receiving time; The Kafka network bandwidth is determined according to the first round-trip time and / or the second round-trip time.

9. The message processing method based on Kafka according to claim 8, characterized in that: Determining the Kafka network bandwidth according to the first round-trip time and / or the second round-trip time includes: If the first round-trip time is greater than or equal to a preset time and / or the second round-trip time is greater than or equal to the preset time, it is determined that the Kafka network bandwidth is abnormal; If the first round-trip time is less than the preset time and the second round-trip time is less than the preset time, it is determined that the Kafka network bandwidth is normal.

10. A message processing system based on Kafka, characterized in that: The Kafka-based message processing system includes: A network monitoring device is used to obtain the write message information and read message information corresponding to each topic in the Kafka cluster; determine the Kafka network bandwidth situation based on the write message information and read message information corresponding to each topic; if the Kafka network bandwidth is abnormal, broadcast a signal of the Kafka network bandwidth abnormality to the Kafka producer client; The Kafka producer client is used to obtain messages to be transmitted if the Kafka network bandwidth is abnormal; based on the characteristic type of the message, it uses the corresponding compression method to compress the target message record, where different characteristic types of messages correspond to different compression methods; and sends the target message record to the Kafka cluster; The Kafka consumer client is configured to read the content of the target topic subscribed by the Kafka consumer client from the Kafka cluster to obtain a target message record if the Kafka network bandwidth is abnormal; parse the target message record to obtain a message body and a message header in the target message record; determine a decompression method for the message body based on the message header; and decompress the message body using the decompression method to obtain a message sent by the Kafka producer client; A Kafka cluster is used to store the target message records.

11. A Kafka producer client, characterized in that The Kafka producer client includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the Kafka-based message processing method according to any one of claims 1 to 5.

12. A Kafka consumer client, characterized in that: The Kafka consumer client includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the Kafka-based message processing method according to claim 6.

13. A network monitoring device, characterized in that: The network monitoring device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the Kafka-based message processing method according to any one of claims 7 to 9.

Images