Log data sending method and device

Abstract

The invention provides a log data sending method and device, the method is applied to network equipment, a message middleware kafka service is configured in the network equipment, and a plurality of message topics corresponding to the kafka service are configured in a log server; the method comprises the following steps: for log data generated this time, acquiring the current priority of each topic; obtaining a packet loss level of packet loss generated in a log data sending process; if the packet loss level is not in a permissible packet loss level range, preferentially sending the log data generated at this time to the top which is not lower than a set priority according to the current priority of each top; and if the packet loss level is within the allowable packet loss level range, sending the log data generated this time to each topic at the same time. Therefore, when the log data is sent based on kafka service, packet loss of the log data in the sending process caused by large data volume is reduced.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a method and device for sending log data. Background Art

[0002] The logs generated in the current network equipment can be directly sent to the log server through system logs and fast logs, or sent to the log server through the message middleware Kafka service. However, since the system log is implemented by sending it through the centralized main control board, there are performance issues when sending log data in a multi-board distributed environment. Similarly, the system log sent through the Kafka service also has the same performance issues.

[0003] The method of sending log data to the log server based on the Kafka service is as follows: the network device as a producer creates a Kafka service locally, under which one or more message topics can be configured; when multiple topics are configured, the amount of data that is the same as the number of topics will be sent. For example, if 10 topics are currently configured and the network device generates 10 log data, then the number of logs sent to the 10 topics simultaneously through the Kafka service is 100, which is equivalent to sending the same data to each topic. When the amount of log data generated is large, the amount of data sent to each topic will also be large, and there will be a problem of log data loss due to the large amount of data. Summary of the invention

[0004] In view of this, the present application provides a log data sending method and device, which are used to reduce the packet loss of log data during the sending process due to the large amount of data when sending log data based on the Kafka service.

[0005] Specifically, the present application is implemented through the following technical solutions:

[0006] According to a first aspect of the present application, a method for sending log data is provided, which is applied to a network device, wherein the network device is configured with a message middleware Kafka service, and a log server is configured with multiple message topics corresponding to the Kafka service; the method comprises:

[0007] For the log data generated this time, obtain the current priority of each message topic;

[0008] Get the packet loss level of the packet loss generated during the log data sending process;

[0009] If the packet loss level is not within the permitted packet loss level range, the log data generated this time is sent to the message topic with a priority not lower than the set priority according to the current priority of each message topic;

[0010] If the packet loss level is within the permitted packet loss level range, the log data generated this time is sent to each message topic at the same time.

[0011] According to a second aspect of the present application, a log data sending device is provided, which is applied to a network device, wherein the network device is configured with a message middleware Kafka service, and a log server is configured with multiple message topics corresponding to the Kafka service; the device comprises:

[0012] The acquisition unit is used to obtain the current priority of each message topic for the log data generated this time;

[0013] An obtaining unit, used to obtain a packet loss level generated during the log data sending process;

[0014] A sending unit, configured to send the log data generated this time to a message topic with a priority not lower than a set priority according to the current priority of each message topic if the packet loss level is not within the permitted packet loss level range;

[0015] The sending unit is further configured to send the log data generated this time to each message topic at the same time if the packet loss level is within the permitted packet loss level range.

[0016] According to the third aspect of the present application, an electronic device is provided, including a processor and a machine-readable storage medium, the machine-readable storage medium storing a computer program that can be executed by the processor, and the processor is prompted by the computer program to execute the method provided in the first aspect of the embodiment of the present application.

[0017] According to the fourth aspect of the present application, a machine-readable storage medium is provided, which stores a computer program. When called and executed by a processor, the computer program prompts the processor to execute the method provided in the first aspect of the embodiment of the present application.

[0018] Beneficial effects of the embodiments of the present application:

[0019] In the log data sending method and device provided by the embodiment of the present application, for the log data generated this time, the current priority of each message topic is obtained; the packet loss level of the packet loss generated during the log data sending process is obtained; if the packet loss level is not within the permitted packet loss level range, then according to the current priority of each message topic, the log data generated this time is sent to the message topic topic with a priority not lower than the set priority; if the packet loss level is within the permitted packet loss level range, the log data generated this time is sent to each message topic topic at the same time. In this method, before sending the log data this time, the current sending environment is evaluated first, that is, the packet loss level during the log data sending process is obtained, and then the log data is sent according to the packet loss level and the priority of each message topic. When the packet loss level is not within the permitted packet loss level range, the log data is sent to the message topic with a high priority first, instead of sending the log data to all message topics at the same time. In this way, the amount of log data sent at the same time is reduced, and the packet loss problem caused by the large amount of data is solved. In addition, when the packet loss level is within the permitted packet loss level range, it indicates that the current sending environment is good, and even if log data is sent to all message topics at the same time, no significant packet loss will occur. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of a method for sending log data provided by an embodiment of the present application;

[0021] Figure 2a This is a schematic diagram of an application scenario of the log data sending method provided in an embodiment of the present application;

[0022] Figure 2b This is another application scenario schematic diagram of the log data sending method provided in an embodiment of the present application;

[0023] Figure 3 It is a structural schematic diagram of a log data sending device provided in an embodiment of the present application;

[0024] Figure 4 It is a hardware structure diagram of an electronic device for implementing a message processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application.

[0026] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more corresponding listed items.

[0027] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] Before introducing the log data sending method provided by this application, the technical terms involved in this application are first introduced:

[0029] 1. The production and consumption of messages in the Kafka service, production, consumption and other message management operations are performed around the message topic.

[0030] 2. The producer is the party that publishes a message to the message topic. The ultimate purpose of publishing a message is to pass the message content to other systems / modules so that the other party can process the message according to the agreement. In the log data sending method provided by the present application, the network device acts as the above-mentioned producer and publishes a message to the message status in the log server.

[0031] 3. Message topic is a publish and subscribe message model of message queue. Producers publish messages to message topics, and multiple consumers subscribe to messages of the message topics. There is no direct relationship between producers and consumers. In this application, the message topic is set in the log server, and the network device, as a producer, publishes log messages containing log data to the message topic in the log server.

[0032] 4. Consumers are the party that subscribes to messages from message topics. The ultimate purpose of subscribing to messages is to process the message content. For example, when the log data sending method provided in this application is applied in a log integration scenario, the monitoring and alarm platform as a consumer subscribes to log messages from message topics, identifies alarm logs and sends alarm messages / emails.

[0033] The log data sending method provided by this application is described in detail below.

[0034] See also Figure 1, Figure 1 This is a flow chart of a log data sending method provided by the present application. The method can be applied to a network device, wherein the network device is configured with a message middleware Kafka service, and the log server is configured with multiple message topics corresponding to the Kafka service; when the above network device implements the method, it may include the following steps:

[0035] Step 101: For the log data generated this time, obtain the current priority of each message topic.

[0036] Specifically, the log server is used to manage and maintain the log data generated by each network device during the business service process. Therefore, the network device can send the log data to the log server. In order to increase the sending speed, the Kafka mechanism is adopted, that is, the Kafka service is set in the network device and the log server, and multiple message topics corresponding to the Kafka service are set in the log server. The network device sends the log data to the multiple message topics in the log server based on the Kafka service.

[0037] In order to solve the problem of packet loss in the process of sending log data based on the kafka service, this embodiment proposes to set a priority for each message topic. For example, in the initial state, the same default priority can be set for each message topic. On this basis, the priority of each message topic is dynamically changed according to the consumption of each message topic. Specifically, the priority of each message topic is related to the consumption frequency of the message topic. For example, the higher the consumption frequency of a message topic, the higher the frequency of the message topic used by the consumer. If the log data is not sent to the message topic in time, it will have a greater impact on the consumption of the log data. Based on this, a higher priority can be set for the message topic; on the contrary, the lower the consumption frequency, the lower the frequency of the message topic used by the consumer. Even if the log data is not sent to the message topic in time, it will not have a greater impact. Based on this, a lower priority can be set for the message topic. Based on this, the dynamic update of the priority of each message topic can be achieved.

[0038] On this basis, when the network device detects that new log data is generated, it can first obtain the priority of each message topic, and then execute the subsequent log data sending process based on the priority.

[0039] It should be noted that the network device, as a producer, can obtain the priority of each message topic from the log server. Specifically, the log server regularly counts the consumption frequency of each message topic, and then determines the priority of each consumption topic. On this basis, the log server can regularly feedback the priority of each message topic to the network device, or, after receiving the priority acquisition request sent by the network device, the log server will feedback the priority of each message topic to the network device.

[0040] Step 102: Obtain the packet loss level of packets lost during the log data sending process.

[0041] In this step, in practical applications, it is inevitable that there will be packet loss, which may be due to a large amount of data or an unstable link. Therefore, in order to reduce packet loss during the log transmission process, the present embodiment can also obtain the packet loss level that generates packet loss during the log data transmission process. The log data is transmitted based on the packet loss level.

[0042] Specifically, the packet loss level obtained above can be the packet loss level obtained most recently. Specifically, the packet loss situations generated in the process of sending logs at adjacent times are similar. Therefore, the packet loss level obtained most recently can characterize the current communication link situation to a certain extent, and further can characterize the possible packet loss in sending the log data based on the current state to a certain extent. In order to reduce or solve the packet loss in the process of sending the log data this time, this embodiment proposes to use the packet loss level as a reference factor for sending the log data this time.

[0043] Optionally, step 102 may be performed according to the following embodiment: collecting statistics on a packet loss rate of packets lost during the log data sending process; and obtaining the packet loss level according to the packet loss rate.

[0044] Specifically, the corresponding relationship between the packet loss level and the packet loss rate range can be preset, as shown in Table 1. In this way, after the packet loss rate is calculated, the packet loss level can be determined based on the above corresponding relationship. The above packet loss rate can be calculated by counting the number of packet losses per unit time, and determining the ratio between the number of packet losses and the total number of log data uploaded per unit time as the packet loss rate, and the unit time can be 1 second, etc.

[0045] Table 1

[0046]

[0047]

[0048] Alternatively, in another embodiment, step 102 may be performed according to the following embodiment: counting the number of packet losses generated per unit time during the log data sending process, and determining the packet loss level based on the number of packet losses.

[0049] Specifically, the corresponding relationship between the number range of packet loss per unit time and the packet loss level can be preset, as shown in Table 2. In this way, after the number of packet loss is obtained through statistics, the target number range to which the number of packet loss belongs can be determined based on the corresponding relationship, and based on this, the packet loss level corresponding to the target number range is recorded as the packet loss level to be obtained.

[0050] Table 2

[0051] Quantity range Packet loss level 0-99 Level 0 100-999 Level 1 1000- Level 2 …… ……

[0052] It is worth noting that the above Table 1 is only an example and does not constitute a specific limitation on the correspondence between the range of packet loss numbers and packet loss levels.

[0053] Step 103: If the packet loss level is not within the permitted packet loss level range, the log data generated this time is sent preferentially to the message topics with a priority not lower than the set priority according to the current priority of each message topic.

[0054] In this step, the above-mentioned permitted packet loss level range may only include at least 1 packet loss level, etc. For example, as shown in reference Table 2, the above-mentioned permitted packet loss level range is set to [0,1). On this basis, when the packet loss level obtained this time is not within the above-mentioned permitted packet loss level range, it indicates that the packet loss situation in the current transmission environment is slightly serious. At this time, in order to solve this packet loss problem, a set priority will be set based on the priority of each message topic, and then a target message topic with a priority not lower than the above-mentioned set priority will be determined, and then the log data generated this time will be sent to the target message topic at the same time, and the log data generated this time will not be sent to other message topics except the target message topic temporarily. That is to say, this application no longer sends the log data generated this time to all message topics at the same time, thereby reducing the amount of log data sent at the same time, thereby solving the packet loss problem caused by the large amount of log data sent.

[0055] Optionally, the specific value of the above-mentioned priority setting can be configured according to actual conditions, and this embodiment does not limit this. For example, when the priorities of each message topic are sorted from high to low, the set priority can be the priority corresponding to TOP3. It should be noted that the priorities of each message topic can be the same or different, and can be specifically determined according to the consumption frequency of each message topic.

[0056] Step 104: If the packet loss level is within the permitted packet loss level range, the log data generated this time is sent to each message topic at the same time.

[0057] In this step, when the packet loss level obtained this time is within the above-mentioned permitted packet loss level range, it can indicate that the current packet loss is very small, and can even be understood as no packet loss, which further indicates that the current log data uploading environment is good. Based on this, this step proposes that the log data generated this time can be sent to each message topic at the same time, and even if the log data generated this time is sent to each message topic at the same time, there will be no problem of packet loss due to the large amount of data.

[0058] It should be noted that the present application does not impose too many restrictions on the execution order of step 101 and step 102. Figure 1 The process shown in the figure may be performed by first performing step 102 and then performing step 101, and the specific method may be determined according to the actual situation.

[0059] In the implementation of the log data sending method provided by the present application, for the log data generated this time, the current priority of each message topic is obtained; the packet loss level of the packet loss generated during the log data sending process is obtained; if the packet loss level is not within the permitted packet loss level range, then according to the current priority of each message topic, the log data generated this time is preferentially sent to the message topic with a priority not lower than the set priority; if the packet loss level is within the permitted packet loss level range, the log data generated this time is sent to each message topic at the same time. In this method, before sending the log data this time, the current sending environment is evaluated first, that is, the packet loss level during the log data sending process is obtained, and then the log data is sent according to the packet loss level and the priority of each message topic. When the packet loss level is not within the permitted packet loss level range, the log data is preferentially sent to the message topic with a high priority, instead of sending the log data to all message topics at the same time. In this way, the amount of log data sent at the same time is reduced, and the packet loss problem caused by the large amount of data is solved. In addition, when the packet loss level is within the permitted packet loss level range, it indicates that the current sending environment is good, and even if log data is sent to all message topics at the same time, no significant packet loss will occur.

[0060] Optionally, based on any of the above embodiments, in this embodiment, after executing step 103, the following process may also be included: obtaining the packet loss level generated during the log data sending process again; when the packet loss level obtained again is within the permitted packet loss level range, sending the log data generated this time to the message topic with a priority lower than the set priority; when the packet loss level obtained again is not within the permitted packet loss level range, sending the log data generated this time to the message topic with a priority lower than the set priority according to the priority of the message topic lower than the set priority.

[0061] Specifically, since the log data generated this time is to be sent to each message topic, when the log data is sent according to the priority, after the log data is sent to the message topics with a priority not lower than the set priority, since the above sending process has reduced packet loss to a certain extent, and since the log data has not yet been sent to the message topics with a priority lower than the set priority, before sending the log data to the message status lower than the set priority, the packet loss level can be obtained again according to the above method; when the packet loss level is within the permitted packet loss level range, the log data generated this time can be sent to each message topic with a priority lower than the set priority at the same time; otherwise, for the remaining message topics, continue to send the log data generated this time to the high-priority message topics in descending order of the priorities of the remaining message topics, until the log data is sent to all message topics.

[0062] Optionally, based on any of the above embodiments, in this embodiment, step 103 can be executed according to the following process: if the packet loss level is not within the permitted packet loss level range, and the packet loss level exceeds the set packet loss level, at least one TCP connection is established with the log server; the log data generated this time to be sent to each message topic is sent to the corresponding message topic through each corresponding TCP connection.

[0063] Specifically, in actual applications, multiple packet loss levels may be divided. Therefore, when it is determined that the currently obtained packet loss level is not within the permitted packet loss level range, if the scheme of sending log data to high-priority message topics is implemented in order of priority from large to small, when the packet loss situation is serious, there may be packet loss problems when sending according to this scheme. In view of this, the present embodiment can also set a set packet loss level, which is greater than the packet loss level corresponding to the right end point of the permitted packet loss level range; on this basis, when the packet loss level obtained in step 102 is greater than the aforementioned set packet loss level, it indicates that the current sending environment is relatively poor, even if log data is only sent to high-priority message topics, there may be packet loss problems during the log data sending process. In view of this, in order to solve this problem, the present embodiment proposes to create at least one TCP connection with the log server, and then allocate corresponding TCP connections to the message topics respectively, and then send log data to the message topics through the TCP connections corresponding to the message topics. In this way, the packet loss problem during the log data sending process is solved.

[0064] Specifically, under a Kafka service, a TCP connection is established between the network device and the log server, and the network device sends log data to each message topic in the log server through the TCP connection at the same time; accordingly, in the implementation Figure 1 In the process shown, the network device also executes step 103 or step 104 based on the TCP connection.

[0065] Specifically, when the network device establishes at least one TCP connection with the log server, taking the establishment of one connection as an example for explanation, one source port can be configured, the source port can be activated, and then one TCP connection can be established with the log server through the source port.

[0066] In the process of executing step 103, when the packet loss level is greater than the set level, the network device will propose to establish at least one TCP connection with the log server in addition to the original 1 TCP connection. Then, based on the original 1 TCP connection and the newly established at least one TCP connection, the log data sent to each message topic is load-shared. In this way, the packet loss problem that may exist when continuing to send log data under the current poor sending environment is solved. Moreover, by establishing a TCP connection, multiple TCP connections are used to load-share the log data sent to each message topic, thereby further reducing the packet loss. In addition, since multiple TCP connections are involved in the sending of log data, when sending according to priority, the purpose of sending log data to low-priority message topics can also be achieved.

[0067] When sending log data based on multiple TCP connections, the log data can be sent in a polling manner according to the priority of each message topic. For example, there are currently 3 TCP connections, recorded as tcp1 to tcp3, and 6 message topics, recorded as topic1 to topic6. The priorities of these 6 message topics decrease in sequence, that is, topic1 has the highest priority and topic6 has the lowest priority. When sending log data, log data is sent to topic1 and topic4 through tcp1, but log data can be sent to topic1 first; similarly, log data is sent to topic2 and topic5 through tcp2, but log data can be sent to topic2 first, and log data is sent to topic3 ​​and topic6 through tcp3, but log data can be sent to topic3 ​​first.

[0068] Of course, based on multiple TCP connections, there can be other sending methods. For example, when the number of TCP connections is lower than the number of message topics, the target number of target message topics that are consistent with the number of selected TCP connections are selected from each message topic in the order of the priority of each message topic from high to low. The target message topic corresponds to the TCP connection one by one. Based on this, log data can be sent to the corresponding target message topic based on each TCP connection. In order to better understand this embodiment, it can also be explained by taking the current 3 connections of tcp1, tcp2 and tcp3, and the current 6 message topics of topic1 to topic6 as an example. The priorities of these 6 message topics decrease in turn, that is, topic1 has the highest priority and topic6 has the lowest priority. On this basis, the target message topics selected according to the above method are topic1 to topic3. At this time, the log data generated this time can be sent to topic1 through tcp1, the log data generated this time can be sent to topic2 through tcp2, and then the log data generated this time can be sent through topic3. We do not send the log data generated this time to other message topics. This ensures that high-priority message topics can obtain the latest log data in a timely manner. At the same time, since the present application sends log data to corresponding message topics based on multiple TCP connections, it also solves the problem of packet loss caused by the large amount of log data and / or the poor current sending environment.

[0069] On this basis, after sending log data to topic1~topic3 ​​based on these three TCP connections respectively, log data can be sent to topic4~topic6 based on these three TCP connections again. In this way, on the basis of solving the packet loss problem, it is guaranteed that log data is successfully sent to each message topic.

[0070] It should be noted that when sending log data to the corresponding message topic based on each TCP connection, the topic name of the message topic can be carried when sending the log data. Different message topics have different corresponding topic names. In this way, after the log server receives the log data, it can know which message topic the current log data should be written to based on the received topic name.

[0071] Furthermore, the log data sending method provided in this embodiment may also include the following process: after sending the log data generated this time to the corresponding message topic based on the respective corresponding TCP connections, deleting multiple TCP connections to retain one TCP connection.

[0072] Specifically, after the log data of this time has been sent to each message topic based on multiple TCP connections, it indicates that the current multiple TCP connections are in an idle state, and since the log data has been sent, it indicates that the current sending environment is relatively good, and the probability of packet loss is small. Therefore, in order to avoid the waste of link resources, this embodiment proposes to retain only one TCP connection among all TCP connections, and then delete the remaining TCP connections. For example, delete the multiple TCP connections created last time, and only retain the original TCP connection.

[0073] It is worth noting that before deleting the TCP connection, the following process can also be performed: obtain the packet loss level generated by the log sending process again; when the packet loss level obtained again is within the permitted packet loss level range, or the packet loss level is not higher than the set packet loss level, execute the step of deleting multiple TCP connections to retain one TCP connection. In this way, after the log data generated this time is sent to each message topic, the current packet loss level is dynamically monitored, and the TCP connection is deleted only when the current packet loss level is within the permitted packet loss level range, or the packet loss level is not higher than the set packet loss level. If any one of them is not met, it means that several TCP connections can be retained and the subsequent log data generated can be sent, thereby reducing packet loss to a certain extent.

[0074] It is worth noting that the above log data can be log data of a quick log, or system data that supports the quick log function. Before the log data sending method provided by this embodiment is used, it can be confirmed whether the quick log function is enabled. After the network device obtains the log data, if it finds that the quick log function is currently enabled, the log data can be sent using the log data sending method provided by any embodiment of the present application.

[0075] In order to better understand this embodiment, Figure 2a The application scenario of the log data sending method shown in the figure is used as an example to illustrate. Figure 2a The network device includes a fast log process and a business module, and the fast log process has a kafka thread that supports the kafka service. Each business module generates log data when performing business processing, and the log data needs to be sent to the log server. In this embodiment, the business module determines whether it supports the fast log function. When the fast log function is supported, the log data generated this time can be sent to the fast log process, and the kafka thread under the fast log process is called by the kafka process, and the kafka thread implements the upload of the log data.

[0076] Specifically, the log server is provided with a kafka agent supporting the kafka service, which is recorded as kafka broker. The kafka agent is used to manage the message topics in the log server. On this basis, the kafka process sends the log data based on the TCP connection established between the network device and the kafka agent in the log server.

[0077] Specifically, for the log data generated this time, the Kafka thread can first obtain the priority of each message topic from the Kafka agent in the log server; in addition, the Kafka thread can also obtain the packet loss level of the packet loss that exists in the process of sending log data; when the packet loss level is within the permitted packet loss level range, the log data generated this time is sent to each message topic at the same time through the current TCP connection between the Kafka agent and the Kafka agent; in this way, after receiving the log data, the Kafka agent can write the log data to the corresponding message topic according to the topic name of the message topic carried by the log data.

[0078] When the packet loss level is not within the permitted packet loss level range, but is lower than the set packet loss level, the priority of each message topic is followed. For example, log data is sent to the top three message topics in priority. That is, the Kafka thread sends log data carrying the topic names of the top three message topics to the Kafka proxy through the TCP connection. In this way, after receiving the log data, the Kafka proxy can obtain the topic name and then write the corresponding log data to the corresponding message topic.

[0079] When the packet loss level is not within the permitted packet loss level range, but the packet loss level is greater than the set packet loss level, the network device can create at least one source port, and then establish a corresponding number of TCP connections with the Kafka agent based on the at least one source port. In this way, the Kafka thread can send log data to the Kafka agent based on the original TCP connection and the newly created TCP connection. For which TCP connection is used to send log data to which message topic, please refer to the description in the aforementioned related embodiments, and the steps are described in detail one by one here. In this way, the problem of packet loss is solved in the process of sending log data to the log server based on the Kafka service. Moreover, the present embodiment also has the function of expanding the sending channel, and log data is sent on this basis, which can further reduce the packet loss rate under the condition of log data congestion.

[0080] in addition, Figure 2b A schematic diagram of another application scenario of the log data sending method is given. In this application scenario, the log server can exist in the form of a log server cluster, that is, the log server is one of the cluster. In actual applications, a kafka service will be created in each log server in the log server cluster. However, in general, the log server cluster includes a primary log server and a backup log server. The primary log server interacts with each network device, and then the primary log server distributes the log data obtained from the network device to other backup log servers as needed; when the primary log server fails, a backup log server can be switched to the primary log server, and then a TCP connection can be established with the network device to receive the log data uploaded by the network device through the kafka service. The upload process can refer to the log data sending method provided by any of the embodiments provided by the aforementioned network device, and will not be described in detail here. The way each primary log server processes the received log data can refer to Figure 2a The execution process of the log server in the scenario shown is not repeated here.

[0081] So far, in the scenario of sending log data in multiple message topics, by adopting the above log data sending method, the packet loss during the log data sending process is reduced to the greatest extent, the performance bottleneck is broken, and the integrity of the log data when it is sent out is improved.

[0082] Based on the same inventive concept, the present application also provides a log data sending device corresponding to the above log data sending method. The implementation of the log data sending device can refer to the above description of the log data sending method, which will not be discussed here one by one.

[0083] See also Figure 3 , Figure 3A log data sending device provided by an exemplary embodiment of the present application is applied to a network device, wherein the network device is configured with a message middleware kafka service, and a log server is configured with multiple message topics corresponding to the kafka service; the device comprises:

[0084] The acquisition unit 301 is used to acquire the current priority of each message topic for the log data generated this time;

[0085] An obtaining unit 302 is used to obtain a packet loss level generated during the log data transmission process;

[0086] The sending unit 303 is configured to send the log data generated this time to the message topics with a priority not lower than the set priority according to the current priority of each message topic if the packet loss level is not within the permitted packet loss level range;

[0087] The sending unit 303 is further configured to send the log data generated this time to each message topic at the same time if the packet loss level is within the permitted packet loss level range.

[0088] Therefore, before sending the log data this time, the current sending environment is evaluated first, that is, the packet loss level during the log data sending process is obtained, and then the log data is sent according to the packet loss level and the priority of each message topic. When the packet loss level is not within the above-mentioned permitted packet loss level range, the log data is sent to the message topic with a high priority first, instead of sending the log data to all message topics at the same time. In this way, the amount of log data sent at the same time is reduced, and the packet loss problem caused by the large amount of data is solved. In addition, when the packet loss level is within the permitted packet loss level range, it indicates that the current sending environment is good, and even if the log data is sent to all message topics at the same time, it will not cause a large packet loss.

[0089] Optionally, in this embodiment, the above-mentioned sending unit 303 is specifically used to establish at least one TCP connection with the log server if the packet loss level is not within the permitted packet loss level range and the packet loss level exceeds the set packet loss level; and send the log data generated this time to be sent to each message topic through each corresponding TCP connection to the corresponding message topic.

[0090] Furthermore, the log data sending device provided in this embodiment may also include:

[0091] A deleting unit (not shown in the figure) is used to delete multiple TCP connections to retain one TCP connection after sending the log data generated this time to the corresponding message topic topic based on the respective corresponding TCP connections.

[0092] Optionally, in this embodiment, the obtaining unit 302 is further configured to obtain again the packet loss level of packet loss generated during the log data sending process after the sending unit 303 executes the following: if the packet loss level is not within the permitted packet loss level range, the log data generated this time is preferentially sent to the message topic with a priority not lower than the set priority according to the current priority of each message topic;

[0093] The sending unit 303 is further configured to send the log data generated this time to a message topic with a priority lower than a set priority when the packet loss level again obtained by the obtaining unit is within the permitted packet loss level range.

[0094] Optionally, in this embodiment, the obtaining unit 302 is specifically used to count the packet loss rate of packets lost during the log data sending process; and obtain the packet loss level according to the packet loss rate.

[0095] Based on the same inventive concept, an embodiment of the present application provides an electronic device, which may be the above-mentioned network device. Figure 4 As shown, the electronic device may include a processor 401 and a machine-readable storage medium 402, the machine-readable storage medium 402 stores a computer program that can be executed by the processor 401, and the processor 401 is prompted by the computer program to execute the log data sending method provided by any embodiment of the present application. In addition, the electronic device also includes a communication interface 403 and a communication bus 404, wherein the processor 401, the communication interface 403, and the machine-readable storage medium 402 communicate with each other through the communication bus 404.

[0096] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0097] The communication interface is used for communication between the above electronic device and other devices.

[0098] The machine-readable storage medium 402 may be a memory, which may include a random access memory (RAM), a DDR SRAM (Double Data Rate Synchronous Dynamic Random Access Memory), or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the processor.

[0099] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0100] In addition, this embodiment also provides a machine-readable storage medium, which stores a computer program. When called and executed by a processor, the computer program prompts the processor to execute the method provided by any embodiment of the present application.

[0101] As for the electronic device and machine-readable storage medium embodiments, since the method contents involved are basically similar to the aforementioned method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0102] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0103] The implementation process of the functions and effects of each unit / module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0104] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units / modules described as separate components may or may not be physically separated, and the components displayed as units / modules may or may not be physical units / modules, that is, they may be located in one place, or they may be distributed on multiple network units / modules. Some or all of the units / modules may be selected according to actual needs to achieve the purpose of the present application. A person of ordinary skill in the art can understand and implement it without creative work.

[0105] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for sending log data, characterized in that: Applied in a network device, the network device is configured with a message middleware Kafka service, and the log server is configured with multiple message topics corresponding to the Kafka service; The method comprises: For the log data generated this time, obtain the current priority of each message topic; Get the packet loss level of the packet loss generated during the log data sending process; If the packet loss level is not within the permitted packet loss level range, the log data generated this time is sent to the message topic with a priority not lower than the set priority according to the current priority of each message topic; If the packet loss level is within the permitted packet loss level range, the log data generated this time is sent to each message topic at the same time.

2. The method according to claim 1, characterized in that If the packet loss level is not within the permitted packet loss level range, the log data generated this time is sent to the message topic with a priority not lower than the set priority according to the current priority of each message topic, including: If the packet loss level is not within the permitted packet loss level range, and the packet loss level exceeds the set packet loss level, at least one TCP connection is established with the log server; The log data generated this time to be sent to each message topic is sent to the corresponding message topic through the corresponding TCP connection.

3. The method according to claim 2, characterized in that Also includes: After sending the log data generated this time to the corresponding message topic topic based on the respective corresponding TCP connections, multiple TCP connections are deleted to retain one TCP connection.

4. The method according to claim 1, characterized in that: After executing if the packet loss level is not within the permitted packet loss level range, sending the log data generated this time to the message topic with a priority not lower than the set priority according to the current priority of each message topic, the method further includes: The packet loss level of the packet loss generated during the log data sending process is obtained again; When the packet loss level obtained again is within the permitted packet loss level range, the log data generated this time is sent to a message topic with a priority lower than the set priority.

5. The method according to claim 1, characterized in that Get the packet loss level of the packets lost during the log data sending process, including: Collect the packet loss rate of packets lost during the log data sending process; The packet loss level is obtained according to the packet loss rate.

6. A log data sending device, characterized in that: Applied in a network device, the network device is configured with a message middleware Kafka service, and the log server is configured with multiple message topics corresponding to the Kafka service; The device comprises: The acquisition unit is used to obtain the current priority of each message topic for the log data generated this time; An obtaining unit, used to obtain a packet loss level generated during the log data sending process; A sending unit, configured to send the log data generated this time to a message topic with a priority not lower than a set priority according to the current priority of each message topic if the packet loss level is not within the permitted packet loss level range; The sending unit is further configured to send the log data generated this time to each message topic at the same time if the packet loss level is within the permitted packet loss level range.

7. The device according to claim 6, characterized in that The sending unit is specifically used to establish at least one TCP connection with the log server if the packet loss level is not within the permitted packet loss level range and the packet loss level exceeds the set packet loss level; and send the log data generated this time to each message topic to the corresponding message topic through each corresponding TCP connection.

8. The device according to claim 7, characterized in that Also includes: The deleting unit is used to delete multiple TCP connections to reserve one TCP connection after sending the log data generated this time to the corresponding message topic topic based on the respective corresponding TCP connections.

9. The device according to claim 6, characterized in that The obtaining unit is further configured to obtain again the packet loss level of packet loss generated during the log data sending process after the sending unit executes the sending of the log data to the message topic with a priority not lower than the set priority according to the current priority of each message topic if the packet loss level is not within the permitted packet loss level range; The sending unit is further configured to send the log data generated this time to a message topic with a priority lower than a set priority when the packet loss level again obtained by the obtaining unit is within the permitted packet loss level range.

10. The device according to claim 6, characterized in that The obtaining unit is specifically used to count the packet loss rate of packets lost during the log data sending process; and obtain the packet loss level according to the packet loss rate.

Images