Log message deduplication based on adaptive time windows

Abstract

Example techniques for adaptive time window based log message deduplication are described herein. In an example, a message value is obtained from a received log message. Further, a number of log messages received in a time window having the message value is counted. Upon expiration of the time window, the log message from which the message value was obtained and the counted number are transmitted. A length of the time window in which a subsequent count of the log message is to be performed is determined based on various parameters.

Description

[0001] This application is a divisional application of the application with the application number 202110436249.7 and titled “System, method and computer readable medium for log message deduplication based on adaptive time window” and filed on April 22, 2021. BACKGROUND

[0002] Log messages can be generated by a computing device to provide information about the operation of the computing device, such as a failure in a component of the computing device or a result of a scan operation in the computing device. The log messages can be transmitted to a backend computing device (also referred to as a backend device) for processing and to take remedial action. BRIEF DESCRIPTION OF DRAWINGS

[0003] The following detailed description references the drawings, wherein:

[0004] Figure 1 A system for performing log message deduplication based on adaptive time window is shown in accordance with example implementations of the present subject matter.

[0005] Figure 2 A network environment having a system for performing log message deduplication based on adaptive time window is shown in accordance with example implementations of the present subject matter.

[0006] FIG. 3(a) illustrates a dictionary in accordance with example implementations of the present subject matter.

[0007] FIG. 3(b) illustrates a dictionary in accordance with example implementations of the present subject matter.

[0008] FIG. 3(c) illustrates a dictionary in accordance with example implementations of the present subject matter.

[0009] Figure 4 A method performed during a time window and at expiry of the time window is shown in accordance with example implementations of the present subject matter.

[0010] Figure 5 A method for determining a length of a second time window is shown in accordance with example implementations of the present subject matter.

[0011] Figure 6 A method for determining a length of a second time window is shown in accordance with example implementations of the present subject matter.

[0012] Figure 7 A method for transmitting a tagged message is shown in accordance with example implementations of the present subject matter.

[0013] Figure 8 A method for processing log messages of different priorities is shown in accordance with example implementations of the present subject matter.

[0014] Figure 9A method for performing adaptive time window based log message deduplication is shown, in accordance with example implementations of the present subject matter.

[0015] Figure 10 A method for counting and transmitting tagged messages corresponding to second priority (SP) log messages is shown, in accordance with example implementations of the present subject matter.

[0016] Figure 11 A computing environment implementing a non-transitory computer readable medium for performing adaptive time window based log message deduplication is shown, in accordance with example implementations of the present subject matter. DETAILED DESCRIPTION

[0017] Log messages generated at a computing device can be sent to a backend device for further processing. For example, multiple log messages can be redundant or repetitive over time when the same operational state of the computing device occurs at different times. Processing of redundant log messages wastes computing resources of the backend device and can also delay processing of unique log messages. Further, at times, log messages can be generated at a high rate in the computing device, for example, due to a malfunction in a component or due to loss of network connectivity. Arrival of log messages at the backend device at a high rate can overwhelm the backend device, causing further delay in processing of log messages.

[0018] Further, in some cases, log messages will be transformed by the computing device so that they can be processed by the backend device. Therefore, if log messages are generated at a high rate, several rounds of transformation will be performed in a short period of time, causing a large consumption of resources at the computing device. In the case of redundant log messages, this consumption of resources is wasteful.

[0019] To prevent transmission of redundant log messages to the backend device, a deduplication technique is typically utilized that involves discarding redundant messages received within a particular duration. However, this technique can be ineffective in a flood scenario of random duration, where log messages are generated very frequently, or in an idle scenario.

[0020] The present subject matter relates to adaptive time window based log message deduplication. The present subject matter prevents transmission of redundant messages to the backend device even in a flood scenario.

[0021] According to an example implementation, a message value is obtained from an incoming log message. The message value can be all or a portion of the log message, or can be derived from all or a portion of the log message. In an example, the message values obtained from redundant log messages can be the same. Further, the number of incoming log messages having a particular message value can be counted. To perform the counting, a dictionary storing a plurality of message values and a count associated with each message value can be utilized. Subsequently, the log messages from which the message values can be obtained and the counts associated with the message values can be transmitted to a backend device.

[0022] Since a single log message corresponding to a plurality of redundant log messages is transmitted, the number of redundant messages received by the backend device is reduced. Further, each log message can not be converted for processing by the backend device. Thus, the resources spent in the conversion of log messages can be reduced. The transmission of a single log message corresponding to a plurality of redundant log messages results in deduplication.

[0023] The counting can be performed in a first time window. Further, the transmission of the log messages and the counts can be performed upon expiration of the first time window. The counting can then be restarted in a second time window. In an example, in addition to the log messages and the counts, a metric indicative of the temporal distribution of the incoming log messages can also be transmitted to the backend device. The metric can include, for example, the length (i.e., duration) of the first time window, the last timestamp within the first time window at which a log message corresponding to the first message value was received, the average inter-arrival time of the log messages corresponding to the first message value, and a histogram indicative of the temporal distribution. The metric can aid the backend device in inferring the root cause of the generation of the log messages and taking corrective actions. Thus, the transmission of the metric prevents loss of information due to deduplication.

[0024] In an example implementation, the length (i.e., duration) of the second time window can be determined based on the number of log messages received in the first time window and the number of different message values obtained in the first time window. For example, if the ratio of the number of log messages to the number of different message values is less than a threshold value, the length of the second time window can be set to a length greater than the length of the first time window. This is because a small ratio value indicates that more log messages are transmitted to the backend device, and an increase in the length of the time window can reduce the number of log messages transmitted. Thus, the length of the second time window is computed in this way controls the number of log messages transmitted to the backend device.

[0025] In an example, the length of the second time window may be calculated based on a first rate at which log messages are received in a first time window and a second rate at which log messages are received in a zeroth time window preceding the first time window. The first rate and the second rate may indicate the presence of a flooding scenario, and thus, setting the length of the second time window based on the rate prevents the transmission of a large number of redundant messages.

[0026] The following description refers to the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and the following description to represent the same or similar parts. Although multiple examples are described in this description, modifications, variations and other implementations are possible and are intended to be covered herein.

[0027] Figure 1 A system 100 for performing log message deduplication based on an adaptive time window according to an example implementation of the present subject matter is shown. The system 100 can be implemented as a computing device, such as a desktop computer, a laptop computer, a server, etc. The system 100 includes a processor 102 and a memory 104 coupled to the processor 102. The processor 102 can also be referred to as a hardware processor 102.

[0028] Processor 102 may be implemented as a microprocessor, a microcomputer, a microcontroller, a digital signal processor, a central processing unit, a state machine, a logic circuit system, and / or any device that can manipulate signals based on operational instructions. Among other capabilities, processor 102 can fetch and execute computer-readable instructions contained in memory 104. The computer-readable instructions include instructions 106, instructions 108, instructions 110, instructions 112, and instructions 114. The functionality of processor 102 may be provided using dedicated hardware as well as hardware capable of executing machine-readable instructions.

[0029] The memory 104 may include any non-transitory computer-readable medium, including volatile memory (e.g., RAM) and / or non-volatile memory (e.g., EPROM, flash memory, memristor, etc.). The memory 104 may also be an external memory unit, such as a flash drive, an optical drive, an external hard drive, etc.

[0030] In addition to the processor 102 and the memory 104, the system 100 may also include (a plurality of) interfaces and system data ( Figure 1 The interface(s) may include various instruction-based and hardware interfaces that allow interaction with a user and with other communication and computing devices, such as network entities, web servers, external repositories, and peripheral devices. System data may serve as a repository for storing data that may be retrieved, processed, received, or created by the instructions.

[0031] In operation, the system 100 can transmit log messages and counts to a backend device (not shown) in the middle by executing the instructions 106-114. The processor 102 can receive log messages in a first time window. The log messages can be generated by a source device, which can be the system 100 or a device connected to the system 100. The log messages can include messages generated by the source device and can provide information about the working of the source device. For example, the log messages can indicate that a new component is installed in the source device or that the working temperature of a component of the source device is high. The reception of the log messages can be performed by executing the instruction 106. Figure 1 The processor 102 can obtain a message value from each of the log messages. The message value, for example, can be the entire log message, the entire log message except for a particular portion of the log message, the particular portion of the log message, a value derived from all or a portion of the log message, or any combination thereof. In an example, similar log messages can have the same message value. The message value can be obtained by executing the instruction 108.

[0032] The processor 102 can count the number of log messages received in the first time window from which the first message value is obtained. The log messages from which the first message value is obtained can be referred to as log messages corresponding to the first message value. On expiry of the first time window, the log message corresponding to the first message value is transmitted to the backend device. Further, the count of the log messages can also be transmitted. Thus, a single log message corresponding to a plurality of log messages corresponding to the first message value can be transmitted. In an example, more than one log message corresponding to the plurality of log messages can be transmitted on expiry of the first time window. However, the number of log messages transmitted can be less than the number of log messages received. For the purpose of discussion, in the explanation provided below, a single log message is explained to be transmitted on expiry of the first time window.

[0033] The transmission of the log message corresponding to the plurality of log messages results in deduplication. The counting can be performed by the processor 102 executing the instruction 110. Further, the transmission to the backend device can be performed by executing the instruction 112.

[0034] In an example, in addition to the log message and the count, a metric indicating the temporal distribution of the incoming log messages can also be transmitted to the backend device. The metric, for example, can include the length of the first time window, the last timestamp within the first time window at which the log message corresponding to the first message value was received, the average inter-arrival time of the log messages corresponding to the first message value, and a histogram indicating the temporal distribution. The log message, the count, and the metric can be collectively referred to as a tagged message.

[0035] In an example, in addition to the log message and the count, a metric indicating the temporal distribution of the incoming log messages can also be transmitted to the backend device. The metric, for example, can include the length of the first time window, the last timestamp within the first time window at which the log message corresponding to the first message value was received, the average inter-arrival time of the log messages corresponding to the first message value, and a histogram indicating the temporal distribution. The log message, the count, and the metric can be collectively referred to as a tagged message.

[0036] At the expiry of the first time window, the count of log messages can be restarted and performed in the second time window. The processor 102 can determine the length of the second time window based on the number of log messages received in the first time window and the number of different message values obtained in the first time window. The number of different message values obtained in the first time window can be less than the number of log messages received in the first time window, as a particular message value can be obtained from multiple log messages. For example, consider that 20 log messages are received during the first time window, and a first message value is obtained from ten of these messages, while a second message value is obtained from the remaining ten messages. Thus, the number of different message values obtained in the first time window is two.

[0037] In an example, in addition to the number of log messages and the number of different message values, other parameters can be used to determine the length of the second time window. As the length of the time window is adapted based on various parameters, the time window can be referred to as an adaptive time window. Further, the deduplication performed can be referred to as adaptive time window based deduplication. Various example aspects of adaptive time window based deduplication will be explained in greater detail below.

[0038] Figure 2 A network environment with a system 100 for performing adaptive time window based log message deduplication is shown, in accordance with an example implementation of the present subject matter. The system 100 can be connected to a source device 202 and a backend device 204. The source device 202 can generate log messages during its operation. In an example, the source device 202 can be a hyper-converged infrastructure system that integrates compute, storage, and networking. The log messages generated by the source device 202 can include alerts generated by the file system 206, kernel, and / or user space (not shown in FIG. 2) of the source device 202. The log messages can be transmitted by a proxy 208 running on the source device 202 for processing by the backend device 204. The proxy 208, for example, can be a remote device access (RDA) proxy. Figure 2

[0039] In an example, the backend device 204 can be part of a cloud network that will monitor the operation of the source device 202 and take corrective action. For example, the source device 202 can be installed on a remote site, such as an offshore drilling rig, and the backend device 204 can be part of a cloud network that will manage the performance of the devices installed on the remote site.

[0040] ​The system 100 can receive log messages from the source device 202, de-duplicate the log messages, and transmit a subset of the received log messages to the backend device 204. In an example, the log messages can be sent as part of a marker message that also includes a count and a metric indicating a temporal distribution of the log messages. In an example, the system 100 can be an intermediary server between the source device and the backend device 204. In an example, the intermediary server can act as a master controller of connections to the backend device 204.

[0041] Although the system 100 is shown as distinct from the source device 202, in an example, the system 100 can be the same as the source device 202. In such an example, both the generation and transmission of the log messages can be performed by the system 100. To transmit the log messages, a de-duplication component (not shown) of the system 100 can intercept log messages generated by a log generation component (such as a file system 206, a kernel, or a user space of the system 100) and perform de-duplication. Further, a marker message can be generated using the log messages, the count, and the metric indicating the temporal distribution of the log messages. In an example, the marker message can be generated in a format readable by the backend device 204, such as a JavaScript Object Notation (JSON) format. The marker message can then be transmitted to the proxy 208 for transmission to the backend device 204. If the marker message is transmitted by the source device 202, the number of messages transmitted to the intermediary server can be reduced, resulting in further reduction of network resource consumption. For the purposes of discussion, the source device 202 is considered distinct from the system 100. Figure 2

[0042] The system 100 can communicate with the source device 202 and the backend device 204 through a communication network 210. The communication network 210 can be a wireless or wired network or a combination thereof. The communication network 210 can be a collection of separate networks that are interconnected to each other and function as a single large network (e.g., the Internet or an intranet). Examples of such separate networks include a Global System for Mobile Communications (GSM) network, a Universal Mobile Telecommunications System (UMTS) network, a Personal Communications Service (PCS) network, a Time Division Multiple Access (TDMA) network, a Code Division Multiple Access (CDMA) network, a Next Generation Network (NGN), a Public Switched Telephone Network (PSTN), and an Integrated Services Digital Network (ISDN). Depending on the technology, the communication network 210 can include various network entities such as transceivers, gateways, and routers.

[0043] In some cases, the source device 202 can generate a large number of log messages. Further, multiple log messages can be repetitive and, thus, redundant. To prevent the backend device 204 from being overwhelmed by all the log messages generated by the source device 202, the system 100 can receive the log messages and identify the redundant messages.

[0044] ​In an example, upon receiving a log message, the system 100 can obtain a message value from the log message. The message value obtained from the log message can be referred to as a message value corresponding to the log message or a message value from the log message. Further, the log message from which the message value is obtained can be referred to as a log message having the message value. The message value may, for example, be an identifier of the log message, the entire log message, the entire log message except for a timestamp in the log message which can indicate a time of generation of the log message, a hash of the log message or a portion thereof, a portion of the log message other than a unique identifier in the log message, or any combination thereof. The hash of the log message can be obtained by subjecting the log message to a hash function. The unique identifier may, for example, be any globally unique identifier (GUID) or universally unique identifier (UUID) in the log message. By not considering the unique identifier for obtaining the message value, log messages which are substantially similar to each other but differ in details which are not useful in identifying potential errors are identified as similar to each other. For example, the message value to be used can be selected based on resources spent in generating the message value and comparing the message values. In an example, the message value can be compared with message values in the dictionary 212 which can be stored on the system 100.

[0045] FIG. 3(a) illustrates the dictionary 212, in accordance with example implementations of the subject innovation. The dictionary 212 can include a plurality of message values, such as a first message value and a second message value. The dictionary 212 can be populated by message values obtained from incoming log messages. For example, the dictionary 212 can be populated by hash values of incoming log messages. Further, the dictionary 212 can have a single copy of a particular message value. Thus, each message value in the dictionary 212 can be different from other message values in the dictionary. In an example, the dictionary 212 can be populated by message values obtained from log messages if the message values are not present in the dictionary 212. For example, if a message value obtained from a log message is the second message value, and if it is determined that the second message value is not present in the dictionary 212 based on a comparison of the second message value with message values in the dictionary 212, the second message value can be added to the dictionary 212. The population of the dictionary 212 in this manner ensures that the dictionary 212 can maintain a record of message values received in incoming log messages. In an example, if the message value is a log message other than a unique identifier in the log message, the message value stored in the dictionary 212 can be a regular expression (regex) version of the log message which can mask the unique identifier.

[0046] Dictionary 212 can also store counts of the number of log messages from which a particular message value is obtained. For example, dictionary 212 can store a count of the number of log messages from which a first message value is obtained, a count of the number of messages from which a second message value is obtained, and so on. The number of messages from which a given message value is obtained can be referred to as a count associated with the message value. For example, if five log messages from which a first message value is obtained are received, the count associated with the first message value can be 5. In an example, dictionary 212 can include a table in which counts associated with message values are indicated for the message values.

[0047] For each incoming log message from which a message value is obtained, the count associated with the message value can be incremented by 1. For example, upon receiving a log message, a message value is obtained and compared to the message values stored in the dictionary. If the obtained message value matches a particular message value, the count associated with the message value is incremented by 1. If the obtained message value does not match any message value in dictionary 212, the obtained message value is added to dictionary 212 and the count associated with the message value is set to 1. The comparison of message values and the incrementing of counts associated with message values in dictionary 212 can be referred to as counting the number of log messages corresponding to the message values.

[0048] In an example, counting the number of log messages corresponding to a message value can be performed in a particular time window. Upon expiration of the time window, the count associated with the message value and one of the log messages having the message value and received during the particular time window can be transmitted to backend device 204. To facilitate the transmission of log messages upon expiration of the time window, in an example, log messages can also be stored in dictionary 212. Further, the count associated with each message value can be reset in dictionary 212. Subsequently, counting the number of log messages corresponding to a message value can be restarted in a subsequent time window. In the case where message values are retained but counts are reset upon expiration of the time window, such a dictionary 212 can be referred to as a static dictionary. If dictionary 212 is a static dictionary, dictionary 212 can be preloaded with a list of message values typically observed during startup or initialization of system 100.

[0049] In an example, instead of resetting the count associated with a message value, the message value and count can be evicted from dictionary 212, thereby purging dictionary 212. Thereafter, in a subsequent time window, dictionary 212 can be repopulated with message values and counts can be performed in the manner explained above. Purging dictionary 212 upon expiration of the time window helps remove message values obtained from a previous window and reduce the size of dictionary 212. In this case, dictionary 212 can be referred to as a dynamic dictionary.

[0050] FIG. 3(b) illustrates a dictionary 212, in accordance with example implementations of the subject matter. The dictionary 212 can be a nested dictionary, where message values are arranged in a hierarchical manner. Thus, more than one message value can be nested in another message value. For example, as shown, a third message value and a fourth message value can be nested in a first message value, and a fifth message value and a sixth message value can be nested in a second message value.

[0051] The nested message values in the dictionary 212 can be part of or can be derived from a portion of an incoming log message, and the nested message values can be another portion of or can be derived from another portion of the incoming log message. Further, a nested message value can have multiple corresponding nested message values. For example, a log message having a particular nested message value can have one of multiple nested message values corresponding to the nested message value. As an example, the nested message value can be a log generating component of the source device 202, such as a hardware component or a service, and the corresponding nested message value can be a subject of a log message generated by or related to the component or a subject of the log message other than a unique identifier in the subject.

[0052] When the dictionary 212 is a nested dictionary, a comparison of message values obtained from a log message with the dictionary 212 can be performed in multiple steps, where the number of steps is equal to the number of layers in the dictionary 212. For example, as shown, consider that the dictionary 212 has two layers, a first layer has a first message value and a second message value, and a second layer has a third message value to a sixth message value. Thus, two message values can be obtained from an incoming log message. For example, a portion of the log message or a value derived therefrom can be one message value, and another portion of the log message or a value derived therefrom can be another message value. One of the message values can be compared with the nested message values in the dictionary 212, such as the first message value and the second message value. If the message value matches one of the nested message values, such as the first message value, another message value from the log message can be compared with the nested message values in the nested message value, such as the third message value and the fourth message value. If the another message value matches the nested message value, a count associated with a combination of the nested message value and the nested message value, such as a combination of the first message value and the third message value, can be incremented.

[0053] The nesting of the dictionary 212 and the comparison in the steps explained above reduces the number of comparisons performed. For example, consider that the message values are arranged in the dictionary 212 in the form of a list. Thus, the plurality of message bodies generated by the first component and the plurality of message bodies generated by the second component are listed in the dictionary 212. Thus, if a log message generated by the second component arrives, the message value from the log message can be compared with the plurality of message values listed in the dictionary 212, including the plurality of message values generated by the first component. On the other hand, if the dictionary 212 is a nested dictionary, the component and the body specified in the log message can be obtained as separate message values. Further, the component specified in the log message can be compared with the first layer of message values in the dictionary 212. Once the component specified in the log message matches the second component mentioned in the dictionary 212, then the body specified in the log message is compared with the bodies nested in the second component.

[0054] FIG. 3(c) illustrates the dictionary 212, in accordance with an example implementation of the present subject matter. The dictionary 212 can include additional information about the log messages with the message values. The additional information can include metrics indicating the temporal distribution of the incoming log messages corresponding to the message values in the first time window. The metrics, for example, can include the length of the time window, the last seen timestamp (LST) within the time window at which the log message corresponding to the message value was received, and the mean inter-arrival time (MIT) between the log messages corresponding to the message values. Similar to the counts, other values stored in the dictionary 212 can also be updated based on the incoming log messages. For example, the LST value corresponding to the first message value can be updated when a log message with the first message value arrives.

[0055] The dictionary 212 can also include a specification about whether the dictionary is a nested dictionary or not. Further, the dictionary 212 can include a specification of the type of the message value (MV), i.e., whether the message value is the entire log message, the entire log message except the timestamp in the log message, a hash of the log message or a portion thereof, a portion of the log message except the unique identifier in the log message, and the like.

[0056] The information stored on the dictionary 212 can be transmitted as part of the tagged message and can be used by the backend device 204 to determine the root cause of the generation of the log message and take corrective action.

[0057] In an example, the system 100 can determine the length of the subsequent time window in which the subsequent de-essentialization is performed based on the de-duplication rate, the number of different message values received in the previous time window, the rate at which log messages are received, or how they are combined. Further, in an example, the message values and other entries in the dictionary 212 can be selectively evicted based on the last seen timestamp of the log message with the message value being received. Still further, in an example, the system 100 can prioritize log messages based on attributes of the log messages and can process log messages of different priorities independently of each other. These aspects will be explained with the help of the flowcharts below.

[0058] Figures 4 to 10 Methods 400-1000 for adaptive time window based log message de-duplication performance are shown in accordance with example implementations of the present subject matter. The order in which the methods 400-1000 are described is not intended to be construed as a limitation, and any number of the described process blocks can be combined in any order to implement the methods 400-1000, or alternate methods. Furthermore, the methods 400-1000 can be implemented by a processing resource(s) or computing device(s) through any suitable hardware, or hardware executing instructions encoded on a non-transitory machine-readable medium.

[0059] It can be appreciated that the blocks of the methods 400-1000 can be executed by a programmed computing device and can be performed based on instructions stored in a non-transitory computer-readable medium. The non-transitory computer-readable medium may, for example, include digital memories, magnetic storage media such as magnetic disks and magnetic tapes, hard disk drives, or optically readable digital data storage media. Further, although the methods 400-1000 can be implemented in various systems, for ease of explanation, the methods 400-1000 are described with reference to the system 100. In an example, the blocks of the methods 400-1000 can be executed by a processing resource such as the processor 102.

[0060] Reference Figure 4 , Figure 4 Methods 400-1000 for adaptive time window based log message de-duplication performance are shown in accordance with example implementations of the present subject matter. The order in which the methods 400-1000 are described is not intended to be construed as a limitation, and any number of the described process blocks can be combined in any order to implement the methods 400-1000, or alternate methods. Furthermore, the methods 400-1000 can be implemented by a processing resource(s) or computing device(s) through any suitable hardware, or hardware executing instructions encoded on a non-transitory machine-readable medium.

[0061] In block 402, a first time window starts. The dictionary 212 can be initialized at the start of the time window. Alternatively, the dictionary 212 can be initialized at the expiration of a previous time window. The initialization of the dictionary 212 can include clearing the dictionary 212, i.e., evicting the message values and counts in the dictionary 212.

[0062] In block 406, a log message is received. In block 408, a message value is obtained from the log message. The obtained message value can be a first message value. Accordingly, in block 410, the first message value is compared to the message values in the dictionary 212. Since the dictionary 212 is empty, in block 412, it is determined that the obtained first message value does not match any of the message values in the dictionary 212. Accordingly, in block 414, the first message value can be added to the dictionary 212. Further, in block 416, a count associated with the first message value is set to 1. Since the log message is a first log message having the first message value received in the first time window, in block 418, the log message is transmitted to the backend device 204.

[0063] If a log message having the first message value is subsequently received in block 406, in block 412, it can be determined that the message value obtained from the log message matches a message value in the dictionary 212. Accordingly, in block 420, the count associated with the first message value can be incremented. However, because the log message having the first message value has already been transmitted to the backend device 204 in the first time window, this log message is not transmitted to the backend device 204.

[0064] If a log message having a second message value is received in block 406, in block 412, it can be determined that the message value obtained from the log message does not match any of the message values in the dictionary 212. Accordingly, in blocks 414 and 416, respectively, the second message value can be added to the dictionary 212 and a count associated with the second message value can be set to 1. Further, in block 418, the log message having the second message value is transmitted to the backend device 204 because it is a first log message received in the first time window and has the second message value. In this way, when a message value is obtained from an incoming log message, the log message corresponding to each message value that is different from the message values in the dictionary 212 is transmitted. Accordingly, for each message value, one log message is transmitted to the backend device 204 without delay. Thus, the backend device 204 can initiate taking a corrective action corresponding to the log message without any delay.

[0065] In block 422, it can be determined whether the first time window has expired. If so, in block 424, one log message corresponding to each message value obtained in the first time window is transmitted to the backend device 204. In addition, the count associated with each message value can also be transmitted. Further, additional information about the log messages with the message values can be transmitted. The additional information can include metrics indicative of the temporal distribution of the incoming log messages corresponding to the message values in the first time window. The metrics, for example, can include the length of the first time window, the last timestamp within the first time window at which a log message corresponding to a message value was received, and the average inter-arrival time between log messages corresponding to the message values, as previously explained. The additional information can be used by the backend device 204 to determine the root cause of the generation of the log messages and to take corrective action. Thus, the transmission of the additional information helps to provide various information at the backend device 204 for taking corrective action without burdening the backend device 204 with redundant incoming log messages.

[0066] The log message corresponding to a message value at the expiration of the first time window, the count corresponding to the message value, and the metrics indicative of the temporal distribution of the incoming log messages corresponding to the message value can be collectively referred to as a marker message corresponding to the message value. Thus, for one time window, two messages (one log message and one marker message) corresponding to a message value are transmitted to the backend device 204, regardless of the number of log messages with the message value received in the time window, thereby achieving deduplication and avoiding overwhelming the backend device 204.

[0067] In block 426, the length of a second time window is determined, the second time window being subsequent to the first time window and in which a subsequent count of the log messages associated with the message values is performed. In an example, the start of the second time window can overlap with the end of the first time window. The determination of the length of the second time window is explained with reference to Figure 5 Further, in block 428, the dictionary 212 can be purged.

[0068] Although the method 400 is explained with reference to a dynamic dictionary, the method 400 can also be applicable to a static dictionary. For example, the initialization of the dictionary 212 can involve resetting the count associated with each message value in the dictionary 212, instead of purging the dictionary 212. Further, if a message value obtained from a log message has a corresponding count of 0 in the dictionary 212, the log message can be transmitted to the backend device 204. Further still, the count associated with a message value can be reset at the expiration of the first time window.

[0069] Figure 5A method 500 for determining a length of a second time window is shown in accordance with example implementations of the present subject matter. The method 500 can be performed by the system 100, such as by the processor 102.

[0070] The length of the time window can affect the number of redundant messages (i.e., messages having the same message value) transmitted to the backend device 204. As previously mentioned, this is because for each time window, two messages (one log message and one marker message) corresponding to one message value are transmitted to the backend device 204. Thus, if the length of the time window is short, multiple messages corresponding to a message value can be transmitted. Further, if the incoming rate of log messages having a particular message value is high within the time window, multiple messages can be transmitted. For example, consider that the incoming rate of log messages having a first message value is 100 log messages per minute, and the length of each time window is 5 seconds. Thus, 20 log messages and 20 marker messages corresponding to the first message value are transmitted to the backend device 204 in one minute. Although the number of messages transmitted is reduced, the number of messages can still be large enough to overwhelm the backend device 204 and the system 100, as the generation of the marker messages also consumes resources. Thus, in the present subject matter, the length of the subsequent time window is adapted to control the number of redundant log messages transmitted to the backend device 204.

[0071] The length of the second time window can be determined based on the number of log messages received in the first time window and the number of different message values obtained in the first time window. The number of log messages and the number of different message values are indicative of the number of messages transmitted for a particular number of incoming log messages. The ratio of the number of log messages received to the number of messages transmitted to the backend device 204 can be referred to as the deduplication rate (D), and can be mathematically represented as follows:

[0072] D = I / 2H,

[0073] where I is the number of log messages received in the first time window, and H is the number of different message values received in the first time window. Here, the value H is multiplied by 2 because two messages (one log message and one marker message) corresponding to each different message value are transmitted. It can be noted that if the dictionary 212 is a dynamic dictionary, H is equal to the size of the dictionary 212. If the dictionary 212 is a static dictionary, H is equal to the number of message values in the dictionary 212 having a corresponding non-zero count.

[0074] A small value of the deduplication rate indicates a relatively large number of messages are being transmitted, which can overwhelm the backend device 204. Thus, if the deduplication rate is low, the length of the second time window can be made longer than the length of the first time window, such that more deduplication is performed and a smaller number of messages are transmitted to the backend device 204. Thus, in block 504, the deduplication rate D is calculated based on the ratio of the number of messages received to the number of distinct message values. In block 506, the deduplication rate D is compared to a first threshold value. If the deduplication rate is less than the first threshold value, it can be determined that the length of the second time window is longer than the first time window, such that fewer messages are transmitted to the backend device. Thus, in block 508, the length of the second time window can be set to a length that is greater than the first time window. In an example, the length of the second time window can be set to a value that is twice the length of the first time window.

[0075] If the deduplication rate is greater than the first threshold value, in block 510, the deduplication rate can be compared to a second threshold value that is greater than the first threshold value. If it is determined in block 512 that the deduplication rate is greater than the second threshold value, it can be determined that the number of messages transmitted is less than the number of messages that the backend device 204 can handle, and more messages can be transmitted to the backend device 204 without overwhelming it. Thus, in block 514, the length of the second time window can be set to a length that is shorter than the first time window. In an example, the length of the second time window can be set to a value that is half the length of the first time window. If the deduplication rate is less than the second threshold value, in block 516, it can be determined that the second time window can have the same length as the first time window.

[0076] Figure 6 A method 600 for determining the length of the second time window is shown, in accordance with example implementations of the present subject matter. The method 600 can be performed by the system 100. Here, in addition to the deduplication rate, the number of distinct message values obtained during the first time window is also considered for determining the length of the second time window. Since a large number of distinct message values in the first time window indicates a larger number of messages are transmitted to the backend device 204, the length of the second time window can have to be increased for a large number of distinct message values. Correspondingly, the length of the second time window can have to be decreased for a small number of distinct message values. Thus, the length of the second time window will be proportional to the number of distinct message values obtained in the first time window. Further, as explained earlier, the length of the second time window is inversely proportional to the deduplication rate.

[0077] Thus, in block 602, a factor can be calculated as the ratio of the number of distinct message values to the deduplication rate, i.e., H / D. This factor can be referred to as the first factor or the H / D factor, and can be multiplied with the length of the first time window to calculate the length of the second time window.

[0078] In an example, the length of the second time window can be determined based on the incoming rate of log messages in the previous time window, as the incoming rate indicates an increase or decrease in the number of log messages received as time elapses. Thus, using the incoming rate of log messages, the presence of a flooding scenario can be identified. To determine the increase or decrease in the number of log messages received, in block 604, a ratio of the incoming rate of log messages in the first time window to the incoming rate of log messages in the zeroth time window, which is prior to the first time window, can be computed. This ratio can be referred to as a second factor or an incoming message rate (IMR) factor. A high value of the IMR factor indicates an increase in the incoming rate of log messages over time. Thus, for a high value of the IMR factor, the length of the second time window will be greater than the length of the first time window to avoid overwhelming the backend device 204 due to the increase in the incoming rate. Thus, the length of the second time window will be proportional to the IMR factor. Thus, the length of the second time window can be achieved by multiplying the IMR factor with the length of the first time window.

[0079] In an example, both the IMR factor and the H / D factor can be used to determine the length of the second time window. For example, in block 606, the length of the second time window can be computed based on a product of the length of the first time window, the IMR factor, and the H / D factor. The length of the second time window can be mathematically represented as follows:

[0080] L2 = L1 x IMR factor x H / D factor x A

[0081] where L1 and L2 are the lengths of the first time window and the second time window, respectively. Further, 'A' can be an adjustment factor that can be used to adjust the value of the subsequent time window. For example, the value of 'A' can be configured based on observations about the performance of the backend device 204 for various lengths of time windows. In an example, machine learning techniques can be used to observe the performance of the backend device 204 for various values of A and adjust the value of A accordingly.

[0082] In the above example, the dynamic dictionary is interpreted to be cleared upon expiration of the time window and repopulated during the subsequent time window. Although the clearing and repopulating can keep the dictionary size small, in some cases, message values can be frequently evicted and added back to the dictionary 212. For example, consider that at expiration of the first time window, a first message value is evicted from the dictionary 212 as part of the clearing. Also consider that a log message having the first message value is received shortly after the second time window begins. In this case, the first message value is removed and added back to the dictionary 212 within a short time span, resulting in a waste of resources. To prevent the overhead due to frequently removing and adding message values to the dictionary 212, in an example, the message values can be retained in the dictionary 212 even after expiration of the first time window, as will be explained below.

[0083] Figure 7 A method 700 for transmitting tagged messages according to an example implementation of the present subject matter is shown. The method 700 can be performed by the system 100. In block 702, a second time window is initialized upon expiration of a first time window. At this time, the dictionary 212 can have all message values obtained during the first time window. Further, the dictionary 212 can have a last seen timestamp associated with each message value, which indicates a timestamp at which a latest log message having the message value was received in the first time window. However, the count associated with the message value can be zero, such that the count can start afresh in the second time window.

[0084] In block 704, it can be determined whether an interval has lapsed since the second time window began. The interval can be a mathematical factor of the length of the second time window. For example, if the length of the second time window is five minutes, the interval can be one minute.

[0085] In block 706, message values in the dictionary 212 are identified that correspond to no log messages being received in a duration of time in the past. The length of the duration of time can be the same as the length of the second time window. For example, if the length of the second time window is five minutes, it can be determined whether a first message value (present in the dictionary 212) is obtained from a log message received in the past five minutes. If a message value is identified such that no log message corresponding to the message value was received in the duration of time, the message value is evicted from the dictionary 212. In an example, the identification of the message values in block 706 can be based on the last seen timestamps in the dictionary 212.

[0086] The eviction of message values in block 708 can be repeated at a regular interval. For example, in block 710, it can be determined whether an interval has elapsed since the last eviction of message values. If so, the identification and eviction of message values in blocks 706 and 708 are repeated. Thus, multiple sets of identification and eviction can be performed in a second time window. The regular eviction of message values ensures that message values corresponding to log messages that were not received are not stored in dictionary 212. Thus, there are no stale message values in dictionary 212. Further, since message values in dictionary 212 are not evicted based on time window expiration, the overhead associated with removing and adding message values in a short time span can be avoided.

[0087] In an example, if it is determined in block 710 that the interval has elapsed, in block 712, it can be determined that the second time window has expired. Further, if the second time window has not expired, the identification in block 706 can be performed. If the second time window has expired, method 700 ends in block 714.

[0088] In an example, the interval specified in block 710 can be the same as the interval specified in block 704, and can be a mathematical factor of the length of the second time window. By making the interval a mathematical factor of the length of the second time window, the eviction can be performed at the expiration of the second time window. For example, consider that the length of the second time window is five minutes, and the interval is two minutes. In this case, while the eviction can be performed at the end of the second and fourth minutes into the second time window, the eviction can not be performed at the end of the fifth minute. Thus, the message values retained in dictionary 212 for a third time window after the second time window can include some stale message values. In contrast, if the interval is one minute (a factor of the length of the second time window), the eviction can also be performed at the end of the fifth minute, thereby preventing stale messages from being retained in dictionary 212.

[0089] In an example, the system 100 can prioritize the log messages into different priorities and can process log messages of one priority differently from log messages of another priority for deduplication. The priority of a log message can refer to the severity of the issue that the log message conveys. In an example, the system 100 can prioritize the log messages based on the severity of the log messages. The severity of a log message can be, for example, an error, a warning, or an info (also referred to as info). Thus, a log message can be referred to as having a first priority if it is an error log message, a second priority if it is a warning log message, or a third priority if it is an info log message. A log message of the first priority (FP) can be referred to as an FP log message. Similarly, a log message of the second priority (SP) and a log message of the third priority (TP) can be referred to as an SP log message and a TP log message, respectively. In the following, the processing of log messages is explained with reference to two priorities: FP and SP, where FP is higher than SP. However, the explanations provided below can also apply to scenarios with more priorities.

[0090] The system 100 can process the FP log messages independently of the SP log messages for the transmission of the marked messages. For example, the system 100 can count the number of FP log messages and the number of different message values from the FP log messages independently of the SP log messages. Further, the time window in which the counting of the log messages and the different message values is performed can differ between the two priorities. For example, while the counting of the FP log messages can be performed during a first time window, the counting of the SP log messages can be performed during a third time window that can partially overlap with the first time window. Thus, the length of the time window in which the counting is performed can differ between one priority and another priority.

[0091] Figure 8 A method 800 for processing log messages of different priorities is shown, in accordance with an example implementation of the present subject matter. The method 800 can be performed by the system 100.

[0092] In an example, the message values obtained from the log messages can be different for different priorities. For example, the message value can be a timestamp of the log message other than the FP log message, and the message value can be a hash of the log message other than the timestamp of the SP log message. Thus, while the dictionary 212 can store the FP log messages other than the timestamp as the message values, the dictionary 212 can store the hash values of the SP log messages other than the timestamp. Further, in an example, the system 100 can utilize different dictionaries corresponding to the log messages of different priorities. For example, the dictionary 212 can be used for comparison with the FP log messages, and another dictionary (not shown) can be used for comparison with the SP log messages. Figure 8The dictionary 212 (not shown) is used for comparison with the SP log messages. For clarity, the dictionary 212 can be referred to as a first dictionary 212 and the other dictionary can be referred to as a second dictionary. The first dictionary 212 can store the FP log messages except for their timestamps and the second dictionary can store the hash values of the SP log messages except for their timestamps.

[0093] Accordingly, when a log message is received in block 802, the priority of the log message is determined in block 804. In an example, the priority can be determined based on the severity attribute of the log message. If the log message is an FP log message, in block 806, the first dictionary 212 is fetched. Further, in block 808, the timestamp is removed from the log message to obtain the message value of the log message. Thereafter, the message value in the first dictionary 212 is compared with the message value from the log message. Subsequently, as explained above, the count is incremented in the first dictionary 212 and the tagged message is transmitted. If the log message is an SP log message, in block 810, the second dictionary is fetched. Further, in block 812, the hash value of the log message except for the timestamp is obtained and compared with the hash values in the second dictionary to increment the count of the corresponding message value.

[0094] As mentioned earlier, the length of the time window can be different for log messages of different priorities, wherein the count is performed and the tagged message is transmitted upon expiration thereof. Further, the length of the time window can be smaller for higher priorities. For example, the time window corresponding to the FP log messages can have a smaller length as compared to the time window corresponding to the SP log messages. To achieve a smaller length of the time window for higher priorities, the threshold value for comparison of the deduplication rate to adjust the length of the time window can also be different for different priorities. For example, the value of the first threshold value can be smaller for SP than for FP. Accordingly, the increase in the length of the time window can be performed more frequently for SP. Similarly, the value of the second threshold value can be smaller for FP. Accordingly, the decrease in the length of the time window can be performed more frequently for FP. Thus, the tagged message corresponding to FP can be transmitted more frequently. The threshold values corresponding to FP and SP can be referred to as FP threshold value and SP threshold value, respectively.

[0095] Although the priority of the log message is explained to be determined based on the severity of the log message, in some examples, other attributes of the log message can be used to prioritize the log message. The other attributes can be, for example, the log generating component of the source device 202. The log messages generated by the kernel can be FP log messages and the log messages generated by the user space can be SP log messages.

[0096] Figure 9A method 900 for performing log message deduplication based on an adaptive time window according to an example implementation of the present subject matter is shown. The method 900 may be performed by the system 100.

[0097] Referring to method 900, in block 902, a first priority (FP) log message may be received in a first time window. Each FP log message has a first priority. The FP log message may be, for example, a reference Figure 8 Interpreted FP log message. In an example, the FP log message may be an error log message.

[0098] In block 904, an FP message value may be obtained from each FP log message. The message value may be, for example, the entire FP log message, the entire log message excluding a specific portion of the log message, an identifier of the FP log message, a hash of the FP log message, a portion of the FP log message excluding a unique identifier in the FP log message, or a combination thereof.

[0099] In block 906, a count associated with each of a plurality of distinct FP message values ​​in a first dictionary is determined. The determination may be performed upon expiration of a first time window. The plurality of distinct message values ​​may be message values ​​obtained from log messages received in the first time window. The count associated with the FP message value indicates the number of FP log messages from which the FP message value was obtained. The first dictionary may be, for example, dictionary 212. The FP message values ​​stored in the first dictionary are referred to as distinct message values ​​because the first dictionary stores a single copy of the message value, as previously explained. The determination in block 906 may be performed by counting the number of FP log messages received in the first time window before the FP message value is obtained from the FP log message, as described with reference to FIG. Figure 2 and 3(a) Explained.

[0100] In block 908, upon expiration of the first time window, a log message corresponding to each FP message value obtained in the first time window and a count associated with the FP message value are transmitted to the backend device. The log message corresponding to the FP message value is a log message for which the FP message value can be obtained, as described in reference to FIG. Figure 2 In an example, the log message and the count may be sent as part of a marker message, which may also include a metric indicating the temporal distribution of the incoming log messages, as previously explained.

[0101] In block 910, the length of the second time window is determined in which subsequent counting of FP log messages is to be performed. This determination may be based on the length of the first time window, the number of FP log messages received in the first time window, and the number of distinct FP message values ​​obtained in the first time window, which may be referenced toFigure 5 As explained with reference to FIG. 2, the length of the second time window is determined based on the number of FP log messages received in the first time window and the number of different FP message values obtained in the first time window. The number of FP log messages received in the first time window is determined based on the number of FP log messages received in the first time window and the number of different FP message values obtained in the first time window. The number of different FP message values obtained in the first time window is determined based on the number of FP log messages received in the first time window and the number of different FP message values obtained in the first time window. Figure 5 As explained with reference to FIG. 2, the length of the second time window is determined based on the number of FP log messages received in the first time window and the number of different FP message values obtained in the first time window. The number of FP log messages received in the first time window is determined based on the number of FP log messages received in the first time window and the number of different FP message values obtained in the first time window. The number of different FP message values obtained in the first time window is determined based on the number of FP log messages received in the first time window and the number of different FP message values obtained in the first time window. Figure 8 As explained with reference to FIG. 2, the length of the second time window is determined based on the number of FP log messages received in the first time window and the number of different FP message values obtained in the first time window. The number of FP log messages received in the first time window is determined based on the number of FP log messages received in the first time window and the number of different FP message values obtained in the first time window. The number of different FP message values obtained in the first time window is determined based on the number of FP log messages received in the first time window and the number of different FP message values obtained in the first time window.

[0102] In an example, the counts associated with the plurality of different FP message values are reset in the first dictionary upon expiry of the first time window. However, the FP message values in the first dictionary are retained. In the second time window, the FP message values in the first dictionary are identified that correspond to no log messages being received in a duration. The length of the duration is equal to the length of the second time window. Subsequently, the identified FP message values can be evicted from the first dictionary. The identification and eviction of the FP message values can be performed at a regular interval. As explained with reference to FIG. 2, the interval at which the identification and eviction of the FP message values is performed can be a mathematical factor of the length of the second time window. Figure 6 As explained with reference to FIG. 2, the length of the second time window is determined based on the number of FP log messages received in the first time window and the number of different FP message values obtained in the first time window. The number of FP log messages received in the first time window is determined based on the number of FP log messages received in the first time window and the number of different FP message values obtained in the first time window. The number of different FP message values obtained in the first time window is determined based on the number of FP log messages received in the first time window and the number of different FP message values obtained in the first time window.

[0103] In an example, the first dictionary is a nested dictionary and stores third and fourth FP message values nested in the first FP message value, as shown in FIG. 3(b). Further, in response to receiving the FP log message, the method 900 can include obtaining a plurality of FP message values from the FP log message. The plurality of FP message values may, for example, be the component that generated the FP log message or is related to the FP log message and the body of the FP log message other than the unique identifier in the body. One of the plurality of FP message values is compared with the first FP message value. If the obtained FP message value matches the first message value, another of the plurality of FP message values is compared with the third FP message value. If the other FP message value matches the third FP message value, the count associated with the combination of the first and third FP message values is incremented, as explained with reference to FIG. 3(b).

[0104] In an example, another priority of log messages can be received and processed independently of the FP log messages, as will be explained below.

[0105] Figure 10A method for counting and transmitting tagged messages corresponding to second priority (SP) log messages according to example implementations of the present subject matter is shown. In block 1002, second priority (SP) log messages can be received in a third time window. The SP log messages can be, for example, informational log messages. The third time window can be different from the first time window and the second time window, and can partially overlap with the first time window and / or the second time window.

[0106] In block 1004, SP message values can be obtained from each of the SP log messages. In an example, the SP message values can be different from the FP message values. For example, while the FP message values can be the FP log messages or hashes of the FP log messages other than their timestamps, the SP message values can be the SP log messages or hashes of the SP log messages other than their timestamps, as explained with reference to Figure 8 Thus, obtaining the SP message values from the SP log messages can involve removing the timestamps from the SP log messages and computing a hash of the remaining portion of the SP log messages, i.e., the SP log messages other than the timestamps.

[0107] In block 1006, counts associated with each of a plurality of different SP message values stored in a second dictionary are determined. The second dictionary can correspond to the SP, while the first dictionary can correspond to the FP. If the obtained FP message values are the FP log messages other than their timestamps, and the obtained SP message values are hashes of the SP log messages other than their timestamps, the first dictionary can store a plurality of FP log messages other than their timestamps, and the second dictionary can store a plurality of hashes of the SP log messages other than their timestamps. Thus, the method 1000 can include comparing the hashes of the SP log messages other than their timestamps with the hashes in the second dictionary.

[0108] In block 1008, upon expiration of the third time window, a tagged message corresponding to each of the SP message values obtained in the third time window and the count associated with the SP message value are transmitted to a backend device.

[0109] In block 1010, a length of a fourth time window for which a subsequent count of SP log messages is to be performed is determined. The determination can be based on the length of the third time window, the number of SP log messages received in the third time window, and the number of different SP message values obtained in the third time window. In an example, determining the length of the fourth time window includes determining a second ratio of the number of SP log messages received in the third time window to the number of different SP message values obtained in the first time window. Further, the second ratio is compared to a first SP threshold and a second SP threshold. The first SP threshold and the second SP threshold can correspond to SPs, and can be different from the first FP threshold and the second FP threshold corresponding to FP. The first SP threshold can be less than the first FP threshold. Thus, the increase in the length of the time window can be performed more frequently for SPs. Similarly, the second FP threshold can be less than the second SP threshold. Thus, the decrease in the length of the time window can be performed more frequently for FP.

[0110] Figure 11 A computing environment 1100 is shown that implements a non-transitory computer readable medium for performing adaptive time window based log message deduplication in accordance with example implementations of the present subject matter. In an example, the non-transitory computer readable medium 1102 can be utilized by a system, such as the system 100. In an example, the computing environment 1100 can include a processing resource 1104 communicatively coupled to the non-transitory computer readable medium 1102 by a network link 1106. The processing resource 1104 may, for example, be the processor 102.

[0111] The non-transitory computer readable medium 1102 may, for example, be an internal memory device or an external memory device. In an example, the communication link 1106 can be a direct communication link, such as any memory read / write interface. In another example, the communication link 1106 can be an indirect communication link, such as a network interface. In this case, the processing resource 1104 can access the non-transitory computer readable medium 1102 over a network 1108. The network 1108 can be a single network or a combination of multiple networks, and can use a variety of different communication protocols.

[0112] The processing resource 1104 and the non-transitory computer readable medium 1102 can also be communicatively coupled to a backend device 1110, such as the backend device 204, and a source device 1112, such as the source device 202. In an example, the non-transitory computer readable medium 1102 can be used by the source device 1112, such that the functions for counting and transmitting of messages can be performed by the source device 1112.

[0113] In an example implementation, the non-transitory computer-readable medium 1102 includes a set of computer-readable instructions for counting and transmitting log messages. The set of computer-readable instructions can be accessed by the processing resource 1104 over the communication link 1106 and subsequently executed.

[0114] With reference to Figure 11 In an example, the non-transitory computer-readable medium 1102 includes instructions 1114 that cause the processing resource 1104 to receive log messages in a first time window. The non-transitory computer-readable medium 1102 includes instructions 1116 that cause the processing resource 1104 to obtain a message value from each log message.

[0115] The non-transitory computer-readable medium 1102 includes instructions 1118 that cause a count of the number of log messages received in the first time window from which a first message value is obtained. The count can be performed with the aid of a dictionary, such as the dictionary 212.

[0116] The non-transitory computer-readable medium 1102 includes instructions 1120 that cause one log message corresponding to the log messages from which the first message value is obtained and the counted number to be transmitted to the backend device 1110 upon expiration of the first time window. In an example, the instructions cause a metric indicative of a temporal distribution of incoming log messages corresponding to the first message value in the first time window to be transmitted upon expiration of the first time window. The metric includes a length of the first time window, a last timestamp within the first time window at which a log message corresponding to the first message value was received, an average inter-arrival time of log messages corresponding to the first message value, and a histogram indicative of the temporal distribution, as explained in block 324. The metric, the log message, and the count can be collectively referred to as a marker message.

[0117] The non-transitory computer-readable medium 1102 includes instructions 1122 that cause a length of a second time window in which a subsequent count of log messages is to be performed to be determined based on a rate at which log messages are received in the first time window and a rate at which log messages are received in a zeroth time window preceding the first time window, as explained with reference to Figure 6 In an example, to determine the length of the second time window, the instructions are executable to compute a ratio of the rate at which log messages are received in the first time window to the rate at which log messages are received in the zeroth time window preceding the first time window. The ratio can be an IMR factor. Further, the length of the second time window is determined based on a product of the ratio and the length of the first time window.

[0118] In an example, the instructions are executable to determine the length of the second time window based on a number of different message values obtained from the log messages received in the first time window, as explained with reference toFigure 4 and 5 As explained. In an example, the instruction is executable to transmit to the backend device log messages that are first received in the first time window and for which the first message value is available, before transmitting the log messages and the count number to the backend device, as explained in block 318.

[0119] The present subject matter reduces the number of redundant log messages transmitted to the backend device, thereby preventing the backend device from becoming overloaded. Further, the messages can be processed by the backend device in near real-time. The deduplication technique of the present subject matter is lightweight and can be performed at the source device that generates the log messages. Resource consumption of the source device that generates the log messages can also be reduced as the number of log messages to be converted for transmission to the backend device is reduced.

[0120] The technique can also be implemented in an intermediate device between the backend device and the source device. Implementation in the intermediate device simplifies loading of settings such as threshold settings, time window settings, and time period settings, as well as loading of the dictionary, as the loading is not performed by each source device.

[0121] Using the technique of the present subject matter, network traffic generated by the transmission of log messages can be significantly reduced. Further, the technique of the present subject matter can be implemented with little change to the backend device and components that generate log messages. Adjustment of the length of the subsequent time window based on various parameters ensures that the number of redundant messages received by the backend device can be reduced even in a flooding scenario of random duration.

[0122] While examples and implementations of the present subject matter have been described in structural and / or method language herein, it is to be understood that the present subject matter is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed and explained in the context of some example implementations of the present subject matter.

Claims

1. A system for adaptive time window based log message deduplication, comprising: a processor; and a non-transitory storage medium comprising instructions executable on the processor to: receive a plurality of log messages in a first time window; obtain a message value from each of the plurality of log messages; count a number of log messages from which a first message value is obtained among the plurality of log messages received in the first time window; in response to expiration of the first time window, transmit a first log message from which the first message value is obtained and the counted number to a backend device; and determine a length of a second time window in which a subsequent count of log messages is to be performed, the determination of the length of the second time window based on a number of the plurality of log messages received in the first time window and a number of different message values obtained from the plurality of log messages in the first time window.

2. The system of claim 1, wherein to determine the length of the second time window, the instructions are executable on the processor to: calculate a ratio of the number of the plurality of log messages received in the first time window to the number of different message values obtained in the first time window; in response to the ratio being less than a first threshold, set the length of the second time window to be greater than a length of the first time window; and in response to the ratio being greater than a second threshold, set the length of the second time window to be less than the length of the first time window, the second threshold being greater than the first threshold. to determine the length of the second time window, the instructions are executable on the processor to: calculate a deduplication rate of the number of the plurality of log messages received in the first time window to the number of different message values obtained in the first time window; 3. The system of claim 1, wherein, calculate a first factor based on a ratio of the number of the different message values obtained in the first time window to the deduplication rate; calculate a second factor based on a ratio of a rate at which log messages are received in the first time window to a rate at which log messages are received in a zeroth time window prior to the first time window; and determine a length of the second time window based on a product of the length of the first time window, the first factor, and the second factor.

4. The system of claim 1, wherein the instructions are executable on the processor to: in response to the expiration of the first time window, transmit a metric indicative of a temporal distribution of incoming log messages corresponding to the first message value in the first time window, the metric comprising a length of the first time window, a last timestamp within the first time window in which a log message corresponding to the first message value was received, and an average inter-arrival time of log messages corresponding to the first message value. to count the number of received log messages from which the first message value is obtained, the instructions are executable on the processor to: ​ ​ 5. The system of claim 1, comprising a memory to store a dictionary containing a plurality of message values, the plurality of message values including the first message value, wherein, ​ compare the message value obtained from each log message to the message values in the dictionary; and in response to the obtained message value matching the first message value, increment a count associated with the first message value in the dictionary.

6. The system of claim 5, wherein, in response to a respective message value obtained from a given log message not matching each message value in the dictionary, the instructions are executable on the processor to: transmit the given log message to the backend device; add the respective message value obtained from the given log message to the dictionary; and set a count associated with the added respective message value to one.

7. The system of claim 5, wherein, in response to expiration of the first time window, the instructions are executable on the processor to: evict the first message value from the dictionary; or reset the count associated with the first message value.

8. The system of claim 1, wherein the instructions are executable on the processor to: increase a length of the second time window to increase deduplication of log messages received in the second time window.

9. The system of claim 1, further comprising a memory to store a dictionary containing different message values, wherein the instructions are executable on the processor to: compare a message value from a second log message to the different message values in the dictionary; in response to the message value from the second log message matching the first message value in the dictionary, determine not to transmit the second log message to the backend device to apply deduplication of log messages.

10. The system of claim 1, wherein the instructions are executable on the processor to: determine a rate at which the plurality of log messages are received in the first time window; and determine the length of the second time window further based on the rate.

11. The system of claim 10, wherein the instructions are executable on the processor to: calculate a factor based on the rate, wherein the factor indicates that a rate of incoming log messages increases over time; and increase the length of the second time window based on the factor.

12. A non-transitory computer-readable medium comprising instructions executable by a processing resource to: receive a plurality of log messages in a first time window; obtain a message value from each log message in the plurality of log messages; count a number of log messages in the plurality of log messages received in the first time window from which a first message value is obtained; in response to expiration of the first time window, transmit the counted number and a single log message from which the first message value is obtained to a backend device; and determine a length of a second time window in which a subsequent count of log messages is to be performed based on a rate at which log messages are received in the first time window and a rate at which log messages are received in a zeroth time window prior to the first time window.

13. The non-transitory computer-readable medium of claim 12, wherein to determine the length of the second time window, the instructions are executable by the processing resource to: a ratio of the rate at which log messages are received in the first time window to the rate at which log messages are received in the zeroth time window; and determining the length of the second time window based on a product of the ratio and a length of the first time window.

14. The non-transitory computer-readable medium of claim 12, wherein the instructions are executable by the processing resource to determine the length of the second time window further based on a number of distinct message values obtained from the plurality of log messages received in the first time window.

15. The non-transitory computer-readable medium of claim 12, wherein the instructions are executable by the processing resource to: in response to expiration of the first time window, transmit a metric indicative of a temporal distribution of incoming log messages corresponding to the first message value in the first time window, the metric comprising a length of the first time window, a last timestamp within the first time window at which a log message corresponding to the first message value was received, and an average inter-arrival time of log messages corresponding to the first message value.

16. The non-transitory computer-readable medium of claim 12, wherein the instructions are executable by the processing resource to: adjust the length of the second time window to alter an amount of deduplication of log messages in the second time window.

17. A method performed by a system comprising a hardware processor, comprising: receiving, at the system, a plurality of log messages in a first time window; obtaining a message value from each log message of the plurality of log messages; counting a number of log messages of the plurality of log messages received in the first time window from which a first message value is obtained; transmitting, from the system to a backend device, the counted number and a first log message from which a first message value is obtained; and determining a length of a second time window in which a subsequent count of log messages is to be performed, the determination of the length of the second time window based on a number of the plurality of log messages received in the first time window and a number of distinct message values obtained from the plurality of log messages in the first time window.

18. The method of claim 17, further comprising: storing a dictionary containing the distinct message values in a memory; comparing a message value from a second log message to the distinct message values in the dictionary; in response to the message value from the second log message matching the first message value in the dictionary, determining not to transmit the second log message to the backend device to apply deduplication of log messages.

19. The method of claim 17, further comprising: determining a rate at which the plurality of log messages are received in the first time window; and determining the length of the second time window further based on the rate.

20. The method of claim 19, further comprising: calculating a factor based on the rate, wherein the factor indicates that a rate of incoming log messages increases over time; and increasing the length of the second time window based on the factor.

Images