A log data sending method and device

Abstract

The application provides a log data sending method and device. The method is applied to a network device. The network device is configured with a message middleware kafka service. A log server is configured with a plurality of message topics topic corresponding to the kafka service. In the method, for log data generated this time, the priority of each topic at present is obtained. A packet loss level of packet loss generated in a log data sending process is obtained. If the packet loss level is not within a permitted packet loss level range, the log data generated this time is preferentially sent to a topic not lower than a set priority according to the priority of each topic at present. If the packet loss level is within the permitted packet loss level range, the log data generated this time is simultaneously sent to each topic. Thus, packet loss of log data in a sending process caused by large data volume when log data is sent based on the kafka service is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication technology, in particular to a log data sending method and device. BACKGROUND

[0002] The log generated in the current network device can be directly sent to the log server through system log and fast log, or sent to the log server through the message middleware kafka service. However, due to the centralized master control board sending mode of the system log, there is a performance problem in sending log data in a multi-board distributed environment. Similarly, the system log sent through the kafka service also has the same performance problem.

[0003] The sending method of sending log data to the log server based on the kafka service is roughly as follows: the network device as a producer creates a kafka service locally, and one or more message topics topic can be configured under the service; when multiple topics are configured, the same multiple data amount as the number of topics will be sent, for example, 10 topics are currently configured, and the network device generates 10 log data, so that 100 log entries are sent to 10 topics through the kafka service, which is equivalent to sending the same data to each topic. When the amount of generated log data is large, the amount of data sent to each topic will also be large, which will cause the problem of log data loss due to large data amount. SUMMARY

[0004] Therefore, the present application provides a log data sending method and device to reduce the packet loss of log data in the sending process due to large data amount when sending log data based on the kafka service.

[0005] Specifically, the present application is realized by the following technical solutions:

[0006] According to the first aspect of the present application, a log data sending method is provided, which is applied to a network device, the network device is configured with a message middleware kafka service, and a log server is configured with a plurality of message topics topic corresponding to the kafka service; the method comprises:

[0007] For the log data generated this time, the current priority of each message topic topic is obtained;

[0008] The packet loss level of the packet loss in the log data sending process is obtained;

[0009] If the packet loss level is not within the range of the permitted packet loss levels, the generated log data is sent to the message topics with a priority higher than the set priority according to the current priorities of the message topics.

[0010] If the packet loss level is within the range of the permitted packet loss levels, the generated log data is sent to all the message topics.

[0011] According to a second aspect of the present application, a log data sending apparatus is provided, which is applied to a network device, the network device is configured with a message middleware kafka service, and the log server is configured with a plurality of message topics topic corresponding to the kafka service; the apparatus comprises:

[0012] a obtaining unit configured to obtain the current priorities of the message topics topic for the generated log data;

[0013] a obtaining unit configured to obtain the packet loss level of the packet loss generated in the log data sending process;

[0014] a sending unit configured to, if the packet loss level is not within the range of the permitted packet loss levels, send the generated log data to the message topics with a priority higher than the set priority according to the current priorities of the message topics topic;

[0015] the sending unit is further configured to, if the packet loss level is within the range of the permitted packet loss levels, send the generated log data to all the message topics.

[0016] According to a third aspect of the present application, an electronic device is provided, which comprises a processor and a machine readable storage medium, the machine readable storage medium stores a computer program capable of being executed by the processor, and the processor is prompted by the computer program to execute the method provided in the first aspect of the present application.

[0017] According to a fourth aspect of the present application, a machine readable storage medium is provided, which stores a computer program, and when the computer program is called and executed by the processor, the computer program prompts the processor to execute the method provided in the first aspect of the present application.

[0018] The beneficial effects of the embodiments of the present application are as follows:

[0019] The log data sending method and device provided in the embodiments of the present application comprise the following steps: obtaining the current priority of each message topic topic for the log data generated this time; obtaining the packet loss level generated in the log data sending process; if the packet loss level is not within the permitted packet loss level range, sending the log data generated this time to the message topics topic with a priority no lower than the set priority according to the current priority of each message topic topic; and if the packet loss level is within the permitted packet loss level range, sending the log data generated this time to each message topic topic at the same time. In the method, the current sending environment is evaluated before the log data generated this time is sent, that is, the packet loss level in the log data sending process is obtained, and then the log data is sent according to the packet loss level and the priority of each message topic. When the packet loss level is not within the permitted packet loss level range, the log data is sent to the message topics with a high priority, instead of being sent to all message topics at the same time, thereby reducing the data volume of the log data sent at the same time, and further solving the packet loss problem caused by a large data volume. In addition, when the packet loss level is within the permitted packet loss level range, it indicates that the current sending environment is good, and even if the log data is sent to all message topics at the same time, no large packet loss will occur. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a flowchart of a log data sending method provided in the embodiments of the present application;

[0021] Figure 2a is a schematic diagram of an application scenario of the log data sending method provided in the embodiments of the present application;

[0022] Figure 2b is another schematic diagram of an application scenario of the log data sending method provided in the embodiments of the present application;

[0023] Figure 3 is a structural schematic diagram of a log data sending device provided in the embodiments of the present application;

[0024] Figure 4 is a hardware structural schematic diagram of an electronic device for implementing a packet processing method provided in the embodiments of the present application. DETAILED DESCRIPTION

[0025] The exemplary embodiments will be described in detail herein with reference to the drawings. Unless otherwise indicated, the same numbers on the drawings indicate the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application.

[0026] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in this application, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0027] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another. For example, a first information can be termed a second information, and, similarly, a second information can be termed a first information, without departing from the scope of the present application. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining" or "in response to a determination."

[0028] Before introducing the log data sending method provided by the present application, the technical terms involved in the present application are introduced:

[0029] 1. The production and consumption of messages in the kafka service are carried out around the message topic (topic) for production, consumption and other message management operations.

[0030] 2. The producer is a party that publishes messages to a message topic. The ultimate purpose of publishing messages is to deliver the message content to other systems / modules so that the other party can process the message according to the agreement. In the log data sending method provided by the present application, the network device acts as the above-mentioned producer and publishes messages to the message state in the log server.

[0031] 3. The message topic is a publish / subscribe message model of the message queue. The producer publishes messages to the message topic, and multiple consumers subscribe to the messages of the message topic. The producer and the consumer have no direct relationship with each other. In the present application, the message topic is set in the log server, and the network device acts as the producer to publish the log message containing the log data to the message topic in the log server.

[0032] 4. The consumer (consumer) is a party that subscribes to messages from a message topic. The ultimate purpose of subscribing to messages is to process the message content. For example, when the log data sending method provided by the present application is applied in a log integration scenario, the monitoring and alarm platform as the consumer subscribes to log messages from the message topic, identifies alarm logs and sends alarm messages / mails.

[0033] The log data sending method provided by the present application is described in detail below.

[0034] Reference is made to Figure 1, Figure 1 is a flowchart of a log data sending method provided by the present application, which can be applied to a network device, wherein a message middleware kafka service is configured in the network device, and a plurality of message topics corresponding to the kafka service are configured in a log server; when implementing the method, the network device can include the following steps:

[0035] Step 101: For the log data generated this time, the current priority of each message topic is obtained.

[0036] Specifically, the log server is used to manage and maintain the log data generated by each network device in the service process, therefore, the network device can upload the log data to the log server, and in order to improve the uploading speed, the kafka mechanism is adopted, that is, the kafka service is set in the network device and the log server, a plurality of message topics corresponding to the kafka service are set in the log server, and the network device uploads the log data to the plurality of message topics in the log server based on the kafka service.

[0037] In order to solve the problem of packet loss of log data in the log uploading process based on the kafka service, the present embodiment proposes to set a priority for each message topic, for example, in the initial state, the same default priority can be set for each message topic. On this basis, the priority of each message topic is dynamically changed according to the consumption of each message topic. Specifically, the priority of each message topic is related to the consumption frequency of the message topic, for example, the higher the consumption frequency of a message topic, the higher the frequency of using the message topic by the consumer, and then if the log data is not uploaded to the message topic in time, it will have a greater impact on log data consumption, based on which, a higher priority can be set for the message topic; on the contrary, the lower the consumption frequency, the lower the frequency of using the message topic by the consumer, that is, even if the log data is not uploaded to the message topic in time, it will not have a greater impact, based on which, a lower priority can be set for the message topic. Based on this, the dynamic updating of the priority of each message topic can be realized.

[0038] On this basis, when the network device monitors that new log data is generated this time, the priority of each message topic can be obtained first, and then the subsequent log data sending process is executed based on the priority.

[0039] It should be noted that the network device as a producer, when obtaining the priority of each message topic, can be obtained from the log server. Specifically, the log server periodically counts the consumption frequency of each message topic, and then determines the priority of each consumption topic. On this basis, the log server can periodically feed back the priority of each message topic to the network device, or the log server feeds back the priority of each message topic to the network device after receiving the priority acquisition request sent by the network device.

[0040] Step 102, obtaining the packet loss level of the packet loss generated in the log data sending process.

[0041] In this step, in actual application, packet loss will inevitably exist, which may be due to large data volume or unstable link. Therefore, in order to reduce packet loss in the log sending process, the embodiment can also obtain the packet loss level of the packet loss generated in the log data sending process. The log data is sent based on the packet loss level.

[0042] Specifically, the above obtained packet loss level can be the packet loss level obtained last time. Specifically, the packet loss generated in the log sending process of adjacent time is similar, so the above packet loss level obtained last time can represent the current communication link to a certain extent, and can also represent the possible packet loss of sending this time's log data based on the current state. In order to reduce or solve the packet loss in this log data sending process, the embodiment proposes to take the packet loss level as a reference factor for this time's log data sending.

[0043] Optionally, step 102 can be performed according to the following embodiment: counting the packet loss rate of the packet loss generated in the log data sending process; obtaining the packet loss level according to the packet loss rate.

[0044] Specifically, the corresponding relationship between the packet loss level and the packet loss rate range can be set in advance, as shown in Table 1. In this way, after calculating the packet loss rate, the packet loss level can be determined based on the above corresponding relationship. The calculation method of the above packet loss rate can be: counting the number of packet loss in unit time, and determining the ratio between the number of packet loss and the total number of log data uploaded in unit time as the packet loss rate. The unit time can be 1 second or the like.

[0045] Table 1

[0046]

[0047]

[0048] Alternatively, in another embodiment, step 102 can be performed as follows: the number of lost packets generated in a unit of time during the sending of the log data is counted, and the lost packet level is determined based on the number of lost packets.

[0049] Specifically, a correspondence between the number range of the number of lost packets in a unit of time and the lost packet level can be set in advance, as shown in Table 2. In this way, when the number of lost packets is obtained this time, the target number range to which the number of lost packets belongs can be determined based on the correspondence, and the lost packet level corresponding to the target number range is recorded as the required lost packet level.

[0050] Table 2

[0051] Number range Packet loss level 0-99 Level 0 100-999 Level 1 1000- Level 2 …… ……

[0052] It should be noted that Table 1 described above is only an example and does not constitute a specific limitation on the correspondence between the number range of the number of lost packets and the lost packet level.

[0053] Step 103: If the lost packet level is not within the permitted lost packet level range, the log data generated this time is preferentially sent to the message topics whose current priorities are not lower than a set priority according to the priorities of the message topics topic.

[0054] In this step, the permitted lost packet level range can include at least one lost packet level, etc. For example, referring to Table 2, the permitted lost packet level range is set to [0, 1), and based on this, when the lost packet level obtained this time is not within the permitted lost packet level range, it indicates that the packet loss is slightly more serious under the current transmission environment. In order to solve this packet loss problem, a set priority is set based on the priorities of the message topics, and the target message topics whose priorities are not lower than the set priority are determined, and the log data generated this time is sent to the target message topics at the same time, and the log data generated this time is temporarily not sent to the message topics other than the target message topics. That is, the application no longer sends the log data generated this time to all message topics at the same time. In this way, the amount of data sent at the same time is reduced, thereby solving the problem of packet loss caused by the large amount of data sent for sending log data.

[0055] Alternatively, the set priority can be configured according to actual conditions, and the present embodiment does not limit this. For example, when the priorities of the message topics are sorted from high to low, the set priority can be the priority corresponding to TOP3. It should be noted that the priorities of the message topics can be the same or different, and can be determined according to the consumption frequency of the message topics.

[0056] Step 104: if the packet loss level is within the permitted packet loss level range, then the current generated log data is sent to each message topic topic at the same time.

[0057] In this step, when the current obtained packet loss level is within the permitted packet loss level range, it indicates that the current packet loss is very small, or even no packet loss, and thus indicates that the current log data uploading environment is good. Based on this, the current generated log data can be sent to each message topic at the same time, and even if the current generated log data is sent to each message topic at the same time, there will be no packet loss problem caused by large data volume.

[0058] It should be noted that the execution order of steps 101 and 102 is not limited too much in this application. It can be executed according to the flow shown in Figure 1 It can be executed first, and then step 101 can be executed. The specific execution can be determined according to the actual situation.

[0059] In the log data sending method provided in this application, for the current generated log data, the current priority of each message topic topic is obtained, the packet loss level generated in the log data sending process is obtained, if the packet loss level is not within the permitted packet loss level range, then the current generated log data is preferentially sent to the message topic topic with a priority not lower than the set priority according to the current priority of each message topic topic; if the packet loss level is within the permitted packet loss level range, then the current generated log data is sent to each message topic topic at the same time. In this method, before the current log data sending is performed, the current sending environment is evaluated, that is, the packet loss level in the log data sending process is obtained, and then the log data is sent according to the packet loss level and the priority of each message topic. When the packet loss level is not within the permitted packet loss level range, the log data is preferentially sent to the message topic with a high priority, instead of being sent to all message topics at the same time. In this way, the data volume of the log data sent at the same time is reduced, and thus the packet loss problem caused by large data volume is solved. In addition, when the packet loss level is within the permitted packet loss level range, it indicates that the current sending environment is good, and even if the log data is sent to all message topics at the same time, no large packet loss will occur.

[0060] Optionally, based on any of the above embodiments, in this embodiment, after performing step 103, the following process can also be included: obtaining again the packet loss level generated in the process of sending log data; when the obtained packet loss level is within the range of the permitted packet loss level, sending the generated log data to the message topics with priority lower than the set priority; when the obtained packet loss level is not within the range of the permitted packet loss level, sending the generated log data to the message topics with priority lower than the set priority according to the priority of the message topics.

[0061] Specifically, since the generated log data is to be sent to each message topic, when sending the log data according to the priority, after sending the log data to the message topics with priority no lower than the set priority, since the above sending process has reduced the packet loss to a certain extent, and since the log data has not been sent to the message topics with priority lower than the set priority, the packet loss level can be obtained again according to the foregoing method before sending the log data to the message topics with priority lower than the set priority; when the packet loss level is within the range of the permitted packet loss level, the generated log data can be sent to each message topic with priority lower than the set priority at the same time; otherwise, for the remaining message topics, the generated log data is continued to be sent to the message topics with high priority according to the order from high to low based on the priority of the remaining message topics, until the log data is sent to all message topics.

[0062] Optionally, based on any of the above embodiments, in this embodiment, step 103 can be performed according to the following process: if the packet loss level is not within the range of the permitted packet loss level, and the packet loss level exceeds the set packet loss level, at least one tcp connection is established with the log server; the generated log data to be sent to each message topic is sent to the corresponding message topic through the corresponding tcp connection.

[0063] Specifically, in actual application, multiple packet loss levels can be divided, and therefore, when it is determined that the current obtained packet loss level is not within the range of the permitted packet loss levels, if the scheme of preferentially sending log data to the message topics with high priority in the order of priority from high to low is implemented, when the packet loss is serious, packet loss can also occur when the log data is sent according to the scheme. Therefore, the embodiment can further set a set packet loss level, which is greater than the packet loss level corresponding to the right endpoint of the range of the permitted packet loss levels. On this basis, when the packet loss level obtained in step 102 is greater than the foregoing set packet loss level, it indicates that the current sending environment is poor, and even if the log data is only sent to the message topics with high priority, packet loss can also occur in the process of sending the log data. Therefore, in order to solve this problem, the embodiment proposes to create at least one tcp connection with the log server, then allocate corresponding tcp connections to the message topics respectively, and then send the log data to the message topics through the tcp connections corresponding to the message topics, thereby solving the problem of packet loss in the process of sending the log data.

[0064] Specifically, in general, under a kafka service, a tcp connection is established between the network device and the log server, and the network device simultaneously sends log data to each message topic topic in the log server through the tcp connection. Correspondingly, when the flow shown in the embodiment is implemented, the network device also performs step 103 or step 104 based on the tcp connection. Figure 1 As shown in the flow, the network device also performs step 103 or step 104 based on the tcp connection.

[0065] Specifically, when the network device establishes at least one tcp connection with the log server, taking the establishment of one connection as an example, one source port can be configured, the source port is activated, and then one tcp connection is established with the log server through the source port.

[0066] In the process of performing step 103, when it is identified that the packet loss level is greater than the set level, the network device proposes to establish at least one tcp connection with the log server in addition to the original one tcp connection. Then, the log data sent to each message topic is processed by load sharing based on the original one tcp connection and the newly established at least one tcp connection, thereby solving the problem of possible packet loss when the log data is continuously sent in the case that the current sending environment is poor. In addition, by establishing multiple tcp connections to perform load sharing on the log data sent to each message topic, the packet loss is further reduced, and in addition, since multiple tcp connections are involved in the sending of the log data, when the log data is sent according to the priority, the purpose of sending the log data to the message topics with low priority can also be achieved.

[0067] When sending log data based on multiple TCP connections, the log data can be sent in a polling manner according to the priority of each message topic. For example, there are currently 3 TCP connections, denoted as tcp1-tcp3, and 6 message topics, denoted as topic1-topic6, and the priority of the 6 message topics decreases in turn, that is, the priority of topic1 is the highest, and the priority of topic6 is the lowest. When sending log data, log data is sent to topic1 and topic4 through tcp1, but log data can be sent to topic1 first. Similarly, log data is sent to topic2 and topic5 through tcp2, but log data can be sent to topic2 first, and log data is sent to topic3 and topic6 through tcp3, but log data can be sent to topic3 first.

[0068] Of course, based on multiple TCP connections, there can be other sending methods, for example, when the number of TCP connections is less than the number of message topics, then according to the priority of each message topic from high to low, from the target number of target message topics corresponding to the number of selected TCP connections, the target message topic corresponds to the TCP connection one by one; Based on this, log data can be sent to the corresponding target message topic based on each TCP connection. In order to better understand this embodiment, it can also be explained by taking the current 3 connections tcp1, tcp2 and tcp3, and the current 6 message topics topic1-topic6 as an example, the priority of the 6 message topics decreases in turn, that is, the priority of topic1 is the highest, and the priority of topic6 is the lowest, and based on this, the target message topics selected according to the above method are topic1-topic3, then at this time, the log data generated this time can be sent to topic1 through tcp1, the log data generated this time can be sent to topic2 through tcp2, and then the log data generated this time can be sent to topic3, and the log data generated this time can not be sent to other message topics. Thus, it is ensured that the high-priority message topic can obtain the latest log data in time, and at the same time, since the application sends log data to corresponding message topics based on multiple TCP connections, the problem of large data volume of log data and / or packet loss caused by poor current sending environment is solved.

[0069] On this basis, after sending log data to topic1-topic3 based on the 3 TCP connections respectively, log data can be sent to topic4-topic6 based on the 3 TCP connections respectively, thereby ensuring that log data is successfully sent to each message topic on the basis of solving the problem of packet loss.

[0070] It should be noted that when sending the log data to the corresponding message topic based on each tcp connection, the topic name of the message topic can be carried while sending the log data. Different message topics correspond to different topic names. In this way, after the log server receives the log data, it can know which message topic the current log data is to be written into based on the received topic name.

[0071] Further, the log data sending method provided by the embodiment can further include the following process: after sending the log data generated this time to the corresponding message topic topic based on the respective corresponding tcp connection, deleting the plurality of tcp connections to retain one tcp connection.

[0072] Specifically, after the log data this time has been sent to each message topic based on the plurality of tcp connections, it indicates that the plurality of tcp connections are in an idle state, and since the log data has been sent, it indicates that the current sending environment is relatively good, and the probability of packet loss is small. Therefore, in order to avoid wasting link resources, the embodiment proposes to retain only one tcp connection in all tcp connections, and then delete the remaining tcp connections. For example, delete the plurality of tcp connections created last time, and retain only the original tcp connection.

[0073] It should be noted that before deleting the tcp connection, the following process can also be performed: obtaining the packet loss level generated by the log sending process again; when the obtained packet loss level again is within the permitted packet loss level range, or the packet loss level is not higher than the set packet loss level, performing the step of deleting the plurality of tcp connections to retain one tcp connection. In this way, after the log data generated this time is sent to each message topic, the current packet loss level is dynamically monitored, and when the current packet loss level is within the permitted packet loss level range or the packet loss level is not higher than the set packet loss level, the deletion of the tcp connection is performed. If any of them is not satisfied, it indicates that a plurality of tcp connections can be retained, and the sending of the generated log data is performed subsequently, thereby also reducing packet loss to a certain extent.

[0074] It should be noted that the log data can be log data of fast log or system data supporting fast log function. Before the log data sending method provided by the embodiment is used, it can be determined whether the fast log function is enabled. After the network device obtains the log data, it is found that the fast log function is enabled, and the log data sending method provided by any embodiment of the present application is used to send the log data.

[0075] In order to better understand the embodiment, the application scenario of the log data sending method shown in Figure 2a will be described.Figure 2a The network device comprises a fast log process and a service module, the fast log process comprising a kafka thread supporting a kafka service. Each service module generates log data when processing a service, and the log data needs to be uploaded to a log server. The embodiment proposes that the service module determines whether it supports a fast log function, and when the fast log function is supported, the service module sends the generated log data to the fast log process, the fast log process calls the kafka thread, and the kafka thread uploads the log data.

[0076] Specifically, the log server is provided with a kafka broker supporting a kafka service, denoted as kafka broker, and the kafka broker is used to manage message topics in the log server. On this basis, the kafka process uploads log data based on a tcp connection established between the network device and the kafka broker in the log server.

[0077] Specifically, the kafka thread can first obtain the priority of each message topic from the kafka broker in the log server for the generated log data. In addition, the kafka thread can also obtain the packet loss level of the packet loss existing in the log data sending process. When the packet loss level is within the permitted packet loss level range, the kafka thread sends the generated log data to each message topic topic through the current tcp connection with the kafka broker. In this way, after the kafka broker receives the log data, the kafka broker can write the log data into the corresponding message topic according to the topic name of the message topic carried by the log data.

[0078] When the packet loss level is not within the permitted packet loss level range, but the packet loss level is lower than the set packet loss level, the kafka thread sends the log data to the message topics with the top 3 priorities, i.e., the kafka thread sends the log data carrying the topic names of the top 3 message topics to the kafka broker through the tcp connection. In this way, after the kafka broker receives the log data, the kafka broker can obtain the topic name and then write the corresponding log data into the corresponding message topic.

[0079] When the packet loss level is not within the range of the permitted packet loss level, but is greater than the set packet loss level, the network device can create at least one source port, and then establish a corresponding number of tcp connections with the kafka broker based on the at least one source port. In this way, the kafka thread can send log data to the kafka broker based on the original tcp connection and the newly created tcp connection. Specifically, which tcp connection is used to send log data to which message topic can refer to the description in the foregoing embodiments, and will not be described in detail here. In this way, the problem of packet loss in the process of sending log data to the log server based on the kafka service is solved. Moreover, the embodiment also has the function of expanding the sending channel, and sending log data on this basis can further reduce the packet loss rate under the condition of log data congestion.

[0080] In addition, Figure 2b Another application scenario of the log data sending method is shown in the figure. In this application scenario, the log server can exist in the form of a log server cluster, that is, the log server is one of the cluster. In actual application, a kafka service can be created in each log server in the log server cluster, but generally, the log server cluster includes one primary log server and one standby log server. The primary log server interacts with each network device, and then distributes the log data obtained from the network device to other standby log servers as needed. When the primary log server fails, a standby log server can be switched to the primary log server, and then establishes a tcp connection with the network device to receive the log data uploaded by the network device through the kafka service. The uploading process can refer to the log data sending method provided by any of the foregoing embodiments of the network device, and will not be described in detail here. The processing manner of each primary log server for the received log data can refer to the execution process of the log server in the scenario shown in Figure 2a The execution process of the log server in the scenario shown in

[0081] In the scenario of sending log data to multiple message topics, the log data sending method described above can maximize the reduction of packet loss in the log data sending process, break the performance bottleneck, and improve the integrity of the data when the log data is sent out.

[0082] Based on the same inventive concept, the present application also provides a log data sending device corresponding to the log data sending method described above. The implementation of the log data sending device can refer to the description of the log data sending method described above, and will not be described here.

[0083] Referring to Figure 3 , Figure 3The log data sending device provided by an example embodiment of the present application is applied to a network device, wherein a message middleware kafka service is configured in the network device, and a plurality of message topics corresponding to the kafka service are configured in a log server; the device comprises:

[0084] The obtaining unit 301 is configured to obtain the current priority of each message topic for the log data generated this time.

[0085] The obtaining unit 302 is configured to obtain a packet loss level of packet loss generated in the log data sending process.

[0086] The sending unit 303 is configured to, if the packet loss level is not within the range of the permitted packet loss level, send the log data generated this time to the message topics with a priority not lower than a set priority according to the current priority of each message topic.

[0087] The sending unit 303 is further configured to, if the packet loss level is within the range of the permitted packet loss level, send the log data generated this time to each message topic at the same time.

[0088] Therefore, before sending the log data this time, the current sending environment is evaluated, that is, the packet loss level in the log data sending process is obtained, and then the log data is sent according to the packet loss level and the priority of each message topic. When the packet loss level is not within the range of the permitted packet loss level, the log data is preferentially sent to the message topics with a high priority, instead of being sent to all message topics at the same time. In this way, the amount of data sent at the same time is reduced, and the problem of packet loss caused by a large amount of data is solved. In addition, when the packet loss level is within the range of the permitted packet loss level, it indicates that the current sending environment is good, and even if the log data is sent to all message topics at the same time, a large packet loss will not occur.

[0089] Optionally, in the present embodiment, the sending unit 303 is specifically configured to, if the packet loss level is not within the range of the permitted packet loss level and the packet loss level exceeds a set packet loss level, establish at least one tcp connection with the log server; and send the log data generated this time to each message topic through a corresponding tcp connection.

[0090] Further, the log data sending device provided by the present embodiment can further comprise:

[0091] A deletion unit (not shown in the figure) is configured to delete the plurality of TCP connections after sending the generated log data to the corresponding message topics topic based on the respective corresponding TCP connections, so as to reserve one TCP connection.

[0092] Optionally, in the embodiment, the obtaining unit 302 is further configured to, after the sending unit 303 performs the operation of sending the generated log data to the message topics topic with a priority no lower than the set priority according to the current priority of the message topics topic if the packet loss level is not within the range of the permitted packet loss levels, obtain again the packet loss level of the packet loss in the log data sending process.

[0093] The sending unit 303 is further configured to, when the packet loss level obtained again by the obtaining unit is within the range of the permitted packet loss levels, send the generated log data to the message topics topic with a priority lower than the set priority.

[0094] Optionally, in the embodiment, the obtaining unit 302 is specifically configured to: count a packet loss rate of the packet loss in the log data sending process; and obtain the packet loss level according to the packet loss rate.

[0095] Based on the same inventive concept, the embodiments of the present application provide an electronic device, which can be the network device described above. As shown in Figure 4 The electronic device can include a processor 401 and a machine readable storage medium 402, the machine readable storage medium 402 stores a computer program capable of being executed by the processor 401, and the processor 401 is prompted by the computer program to execute the log data sending method provided by any of the embodiments of the present application. In addition, the electronic device further includes a communication interface 403 and a communication bus 404, wherein the processor 401, the communication interface 403, and the machine readable storage medium 402 complete the communication among each other through the communication bus 404.

[0096] The communication bus mentioned above can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or only one type of bus.

[0097] The communication interface is used for the communication between the electronic device and other devices.

[0098] The machine readable storage medium 402 described above can be a memory, which can include a random access memory (RAM), a DDR SRAM (Double Data Rate Synchronous Dynamic Random Access Memory), and can also include a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0099] The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc., and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0100] In addition, the embodiment also provides a machine readable storage medium, which stores a computer program. When the computer program is called and executed by a processor, the computer program causes the processor to execute the method provided by any of the embodiments of the present application.

[0101] For the electronic device and the machine readable storage medium embodiments, since the method content involved is basically similar to the foregoing method embodiments, the description is relatively simple, and the related parts are described in the method embodiment part.

[0102] It should be noted that, in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0103] The implementation process of the functions and roles of each unit / module in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0104] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be seen from the part of the method embodiment. The device embodiments described above are only schematic, and the units / modules shown as separate components can or can not be physically separate, and the components shown as units / modules can or can not be physical units / modules, that is, they can be located in one place, or distributed on multiple network units / modules. Part or all of the units / modules can be selected to achieve the purpose of the scheme of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0105] The above only describes the preferred embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for sending log data, characterized in that, This is applied in network devices, where the network devices are configured with the message middleware Kafka service, and the log server is configured with multiple message topics corresponding to the Kafka service; The method includes: Based on the log data generated this time, obtain the current priority of each message topic; Obtain the packet loss level that occurred during the log data transmission process; If the packet loss level is not within the permissible packet loss level range, then according to the current priority of each message topic, the log data generated this time will be sent to message topics with a priority not lower than the set priority. If the packet loss level is within the permissible packet loss level range, then the generated log data will be sent to each message topic simultaneously.

2. The method according to claim 1, characterized in that, If the packet loss level is not within the permissible packet loss level range, then according to the current priority of each message topic, the log data generated this time will be sent first to message topics with a priority not lower than the set priority, including: If the packet loss level is not within the permitted packet loss level range, and the packet loss level exceeds the set packet loss level, then at least one TCP connection is established with the log server; The log data generated this time, which is to be sent to each message topic, will be sent to the corresponding message topic through their respective TCP connections.

3. The method according to claim 2, characterized in that, Also includes: After sending the generated log data to the corresponding message topic using their respective TCP connections, multiple TCP connections are deleted to retain only one TCP connection.

4. The method according to claim 1, characterized in that, After executing the instruction that if the packet loss level is not within the permissible packet loss level range, the log data generated this time should be sent to message topics with a priority not lower than the set priority, according to the current priority of each message topic, the instruction further includes: Obtain the packet loss level that occurred during the log data transmission process again; If the packet loss level is obtained again within the allowed packet loss level range, the generated log data is sent to a message topic with a lower priority than the set priority.

5. The method according to claim 1, characterized in that, Obtain the packet loss level that occurred during log data transmission, including: The packet loss rate during the transmission of statistical log data; The packet loss level is obtained based on the packet loss rate.

6. A log data sending device, characterized in that, This is applied in network devices, where the network devices are configured with the message middleware Kafka service, and the log server is configured with multiple message topics corresponding to the Kafka service; The device includes: The acquisition unit is used to obtain the current priority of each message topic based on the log data generated this time; The acquisition unit is used to obtain the packet loss level that occurs during the transmission of log data; The sending unit is configured to, if the packet loss level is not within the permissible packet loss level range, send the log data generated this time to message topics with a priority not lower than a set priority, according to the current priority of each message topic. The sending unit is further configured to simultaneously send the generated log data to each message topic if the packet loss level is within the permissible packet loss level range.

7. The apparatus according to claim 6, characterized in that, The sending unit is specifically configured to establish at least one TCP connection with the log server if the packet loss level is not within the permitted packet loss level range and the packet loss level exceeds the set packet loss level; and to send the log data generated this time to each message topic through their respective TCP connections.

8. The apparatus according to claim 7, characterized in that, Also includes: The deletion unit is used to delete multiple TCP connections after sending the generated log data to the corresponding message topic based on their respective TCP connections, so as to retain only one TCP connection.

9. The apparatus according to claim 6, characterized in that, The obtaining unit is further configured to, after the sending unit executes the following steps: if the packet loss level is not within the permissible packet loss level range, send the generated log data to message topics with a priority not lower than the set priority according to the current priority of each message topic, and then obtain the packet loss level of the packet loss that occurred during the log data sending process again; The sending unit is further configured to send the generated log data to a message topic with a lower priority if the packet loss level obtained by the obtaining unit is within the permitted packet loss level range.

10. The apparatus according to claim 6, characterized in that, The obtaining unit is specifically used to calculate the packet loss rate during the transmission of log data; and to obtain the packet loss level based on the packet loss rate.

Images