Methods, apparatuses, computing devices, and media for processing a log

Abstract

The present disclosure provides a method for processing logs, comprising: obtaining a plurality of logs to be output received in a traffic detection period; determining semantic similarity between the plurality of logs to be output in a case where a number of the plurality of logs to be output exceeds a traffic detection threshold; merging the plurality of logs to be output according to the semantic similarity between the plurality of logs to be output, to obtain at least one target log; and outputting the at least one target log. The present disclosure also provides an apparatus for processing logs, a computing device, a medium, and a computer program product.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer technology / Internet technology / electronic technology, and more particularly, to a method and apparatus for processing logs, a computing device, a medium, and a computer program product. BACKGROUND

[0002] Log data is used to reflect the trajectory generated in the process of user interaction with a computer system. A log system is used to collect and monitor the running of a program to monitor the exceptions generated by an application in real time. The log system can effectively help developers understand the status of system running and troubleshoot system exceptions. Therefore, an efficient and stable log system is of great significance.

[0003] Log4j is a log operation package that provides log recording and control functions and is flexible. The log library system of the related art learns from the concept of Log4j and prints logs to files asynchronously through Log4j.

[0004] The log library system of the related art has at least the following problems in design:

[0005] 1) The input and output (IO) operation of the log library console will block the execution of the main thread. More specifically, since the log library system of the related art uses single-thread design, the business program needs to wait for the IO operation of the log data to be completed before continuing to execute when executing, which seriously affects the real-time performance of the business.

[0006] 2) A large number of repeated abnormal logs are contained in the flood peak traffic, and a large number of IO operations will be triggered for these large number of repeated abnormal logs, which will impact the disk and seriously affect the system performance. SUMMARY

[0007] Therefore, the present disclosure provides a method and apparatus for processing logs, a computing device, a medium, and a computer program product.

[0008] One aspect of the present disclosure provides a method for processing logs, comprising: obtaining a plurality of to-be-output logs received within a traffic detection period; in a case where the number of the plurality of to-be-output logs exceeds a traffic detection threshold, determining semantic similarity between the plurality of to-be-output logs; merging the plurality of to-be-output logs according to the semantic similarity between the plurality of to-be-output logs to obtain at least one target log; and outputting the at least one target log.

[0009] According to an embodiment of the present disclosure, the determining the semantic similarity between the plurality of logs to be output includes: determining content similarity and / or type similarity between the plurality of logs to be output, respectively; and determining the semantic similarity between the plurality of logs to be output according to the content similarity and / or the type similarity.

[0010] According to an embodiment of the present disclosure, the determining the content similarity between the plurality of logs to be output includes: determining a same word sequence contained in any two logs to be output in the plurality of logs to be output; determining a word sequence containing the most words in the same word sequence as a longest common subsequence; and determining content similarity of the two logs to be output according to the longest common subsequence and the number of words contained in each of the two logs to be output.

[0011] According to an embodiment of the present disclosure, the determining the type similarity between the plurality of logs to be output includes: determining a type feature set of each log to be output in any two logs to be output in the plurality of logs to be output, wherein the type feature set includes at least one type feature; determining the number of same type features contained in the type feature sets of the two logs to be output; and determining the type similarity between the two logs to be output according to the number of same type features.

[0012] According to an embodiment of the present disclosure, the merging the plurality of logs to be output according to the semantic similarity between the plurality of logs to be output includes: dividing the plurality of logs to be output into at least one log set according to the semantic similarity between the plurality of logs to be output, wherein the semantic similarity between the logs to be output contained in each log set in the at least one log set exceeds a similarity threshold; and determining any one log to be output in each log set as the target log.

[0013] According to an embodiment of the present disclosure, the method further includes: counting the number of logs to be output contained in each log set; and outputting the number of logs to be output contained in each log set.

[0014] Another aspect of the present disclosure provides an apparatus for processing logs, comprising: an obtaining module configured to obtain a plurality of logs to be output received in a traffic detection period; a determining module configured to determine semantic similarity between the plurality of logs to be output in a case where a quantity of the plurality of logs to be output exceeds a threshold of a request number; a merging module configured to merge the plurality of logs to be output according to the semantic similarity between the plurality of logs to be output, to obtain at least one target log; and an outputting module configured to output the at least one target log.

[0015] Another aspect of the present disclosure provides a computing device, comprising: one or more processors; a storage device configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method as described above.

[0016] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, implement the method as described above.

[0017] According to yet another aspect of embodiments of the present disclosure, a computer program product is provided, comprising computer programs / instructions that, when executed by a processor, implement the method according to embodiments of the present disclosure.

[0018] According to embodiments of the present disclosure, by merging logs according to semantic similarity of the logs in a case where a quantity of logs received in a traffic detection period exceeds a traffic detection threshold, the quantity of output logs can be reduced, and the output performance of logs can be ensured. In addition, the processing operation and the output operation of logs are executed asynchronously, and a business program does not need to wait for completion of an IO operation of log data when being executed, thereby ensuring real-time performance of the business. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0020] Figure 1 An exemplary system architecture to which the method for processing logs according to embodiments of the present disclosure can be applied is schematically shown;

[0021] Figure 2 A flowchart of the method for processing logs according to embodiments of the present disclosure is schematically shown;

[0022] Figure 3 A fixed window period division schematic diagram according to embodiments of the present disclosure is schematically shown;

[0023] Figure 4A flowchart of a method of processing logs in an abnormal scenario is schematically illustrated according to an embodiment of the present disclosure;

[0024] Figure 5 A flowchart of a method of processing logs is schematically illustrated according to another embodiment of the present disclosure;

[0025] Figure 6 A block diagram of an apparatus is schematically illustrated according to an embodiment of the present disclosure; and

[0026] Figure 7 A block diagram of a computer system suitable for implementing the method described above is schematically illustrated according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that the description which follows is merely illustrative and is not intended to limit the scope of the present disclosure. In the following detailed description of embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that one or more embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present disclosure.

[0028] The terms used herein are merely used to describe specific embodiments and are not intended to limit the present disclosure. The terms "include", "comprise", and the like used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of the present specification, and should not be interpreted in an idealized or overly formal manner.

[0030] In the case of using expressions similar to "at least one of A, B, and C, etc.", it is generally understood that the expression is used to include at least one of A, B, or C or any combination thereof, unless otherwise specifically stated. In the case of using expressions similar to "at least one of A, B, or C, etc.", it is generally understood that the expression is used to include at least one of A, B, or C or any combination thereof, unless otherwise specifically stated.

[0031] Embodiments of the present disclosure provide a method for processing logs and an apparatus capable of applying the method. The method comprises obtaining a plurality of to-be-output logs received within a flow detection period; in a case where a number of the plurality of to-be-output logs exceeds a flow detection threshold, determining semantic similarity between the plurality of to-be-output logs; according to the semantic similarity between the plurality of to-be-output logs, merging the plurality of to-be-output logs to obtain at least one target log; and outputting the at least one target log.

[0032] Figure 1 An exemplary system architecture 100 to which the method for processing logs according to embodiments of the present disclosure can be applied is schematically shown. It should be noted that, Figure 1 The shown is only an example of a system architecture to which embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0033] As Figure 1 The system architecture 100 according to the embodiment can include log generation apparatuses 101, 102, 103, a log processing apparatus 104, and a log receiving apparatus 105, as shown.

[0034] The log generation apparatuses 101, 102, 103 can be used to generate logs and send the generated logs to the log processing apparatus 104, which can include network devices, security devices, databases, business systems, etc.

[0035] The log processing apparatus 104 can be used to monitor the number of received logs in a set time window as a period, in a case where the number of logs received within the flow detection period exceeds the flow detection threshold, determine the semantic similarity between the logs, merge the logs according to the semantic similarity between the logs, and then output the merged logs to the log receiving apparatus 105.

[0036] The log receiving device 105 can be configured to perform subsequent processing on the logs, such as storing, analyzing, presenting, forwarding, etc.

[0037] It should be understood that Figure 1 The number of log generating devices, log processing devices and log receiving devices in the system 100 is only illustrative. According to the needs of implementation, there can be any number of log generating devices, log processing devices and log receiving devices.

[0038] Figure 2 A flowchart of a method of processing logs according to an embodiment of the present disclosure is illustratively shown.

[0039] As Figure 2 shown, the method comprises, in operation S210, obtaining a plurality of to-be-output logs received within a flow detection period.

[0040] According to an embodiment of the present disclosure, the flow detection period can be set according to actual needs. The following formula illustratively shows the setting method of the flow detection period. Wherein, period is the flow detection period parameter. As shown in the following formula, if the value of the flow detection period parameter x≤0, no flow control is performed, and the to-be-output logs are directly output; if x>0, the flow detection period is x (the unit can be, for example, second).

[0041]

[0042] Then, in operation S220, in the case where the number of the plurality of to-be-output logs exceeds a flow detection threshold, the semantic similarity between the plurality of to-be-output logs is determined.

[0043] According to an embodiment of the present disclosure, the flow detection threshold can be set according to actual needs. The following formula illustratively shows the setting method of the flow detection threshold. Wherein, threadhold is the maximum number of log outputs allowed in the flow detection stage, i.e. the flow detection threshold. As shown in the following formula, if the flow detection threshold y≤0, flow control is always performed; if y>0, the flow detection threshold is y (the unit is, for example, number).

[0044]

[0045] According to an embodiment of the present disclosure, if the number of logs exceeds the flow detection threshold, operations S230-S240 are performed to control the to-be-output logs. If the number of the plurality of to-be-output logs does not exceed the flow detection threshold, operations S230-S240 are not performed, and the to-be-output logs are directly output.

[0046] In operation S230, the plurality of to-be-output logs are merged according to the semantic similarity between the plurality of to-be-output logs, to obtain at least one target log.

[0047] According to an embodiment of the present disclosure, in the log similarity calculation stage T2, the content similarity and / or the type similarity between the plurality of to-be-output logs can be determined respectively, and then the semantic similarity between the plurality of to-be-output logs is determined according to the content similarity and the type similarity. Next, the plurality of to-be-output logs are divided into at least one log set according to the semantic similarity between the plurality of to-be-output logs, wherein the semantic similarity between the to-be-output logs contained in each log set in the at least one log set exceeds a similarity threshold. Any one to-be-output log in each log set is determined as a target log.

[0048] According to an embodiment of the present disclosure, the logs with the semantic similarity exceeding the similarity threshold can be considered as the same kind of logs, the plurality of to-be-output logs are divided into at least one log set according to the semantic similarity between the plurality of to-be-output logs, and any one to-be-output log in each log set is determined as a target log, so that the same kind of logs can be merged into one target log, thereby reducing the number of output logs and ensuring the log output performance.

[0049] For example, determining the content similarity between the plurality of to-be-output logs can include determining the same word sequence contained in any two to-be-output logs in the plurality of to-be-output logs, then determining the word sequence with the largest number of contained words as the longest common subsequence, and next determining the content similarity between the two to-be-output logs according to the longest common subsequence and the number of words contained in each of the two to-be-output logs.

[0050] In addition, determining the type similarity between the plurality of to-be-output logs can include, for any two to-be-output logs in the plurality of to-be-output logs, determining the type feature set of each to-be-output log in the two to-be-output logs, wherein the type feature set includes at least one type feature, then determining the number of same type features contained in the type feature sets of the two to-be-output logs, and next determining the type similarity between the two to-be-output logs according to the number of same type features.

[0051] In operation S240, the at least one target log is output.

[0052] According to another embodiment of the present disclosure, the number of to-be-output logs contained in each log set can also be counted, and the number of to-be-output logs contained in each log set can also be output.

[0053] According to the embodiments of the present disclosure, by merging the logs according to the semantic similarity of the logs in the case that the number of logs received in the flow detection period exceeds the flow detection threshold, the number of output logs can be reduced, and the output performance of the logs can be ensured. In addition, the processing operation and the output operation of the logs are executed asynchronously, and the business program does not need to wait for the IO operation of the log data to be completed when being executed, thereby ensuring the real-time performance of the business.

[0054] The method shown in Figures 3-4 will be further described below in conjunction with another specific embodiment. It can be understood by those skilled in the art that the following example embodiments are only for understanding the present disclosure, and the present disclosure is not limited thereto. Figure 2

[0055] According to the embodiments of the present disclosure, a fixed window period can be set, and the detection and control of the log flow are periodically performed according to the fixed window period. Figure 3 A schematic diagram of the division of the fixed window period according to the embodiments of the present disclosure is shown schematically. As shown in Figure 3 , period is a flow detection period parameter, and threadhold is a flow detection threshold. The fixed time window may, for example, include a flow log detection period and a flow log control execution period. The flow log detection period may, for example, include a flow detection stage T1, and the flow log control execution period may, for example, include a log similarity calculation stage T2 and a log record flow rate control stage T3.

[0056] According to the embodiments of the present disclosure, the number of received logs can be detected in the flow detection stage T1. If the number of logs exceeds the flow detection threshold, the log similarity calculation stage T2 and the log record flow rate control stage T3 are entered to perform flow control. If the number of the plurality of logs to be output does not exceed the flow detection threshold, the next flow log detection period is entered.

[0057] According to the embodiments of the present disclosure, in the log similarity calculation stage T2, the longest common subsequence algorithm can be used to determine the semantic similarity between the logs.

[0058] According to the longest common subsequence algorithm of the embodiments of the present disclosure, when calculating the semantic similarity between two logs, each word in the logs is taken as a basic unit, and each word in the two logs is matched in turn. If the words are the same, the word is added to the common subsequence until the next word is not the same. Then, from all the obtained common subsequences, the common subsequence containing the most words is determined as the longest common subsequence. Next, the content similarity is calculated based on the number of words in the longest common subsequence and the total number of words in the two logs.

[0059] ​For example, assuming two logs to be matched are l' and l, each of which contains m and n words respectively, the content similarity calculation formula of the two logs is

[0060]

[0061] wherein SS(l, l') represents the content similarity of the two logs, and |LSC(l, l')| represents the number of matching words of the longest common subsequence of the two logs.

[0062] Table 1 schematically shows the longest common subsequence log examples according to the embodiments of the present disclosure. As shown in Table 1, the logs to be matched include l1, l2, l3, and the similarity threshold is set to 0.8, wherein the first 13 characters "No alive provider! The key is jsf: / / com.jd.b2b.workflow.sdk.WorkflowReadProvider" of l1 and l2 are matched, and S(l1, l2) = 13*2 / (14+14) = 0.928 > 0.8 is calculated according to the formula, which indicates that l1 and l2 are the same kind of logs. Similarly, l1 and l3 do not have matching words, i.e., the number of matching words is 0, and thus the similarity of l1 and l3 is S(l1, l3) = 0 < 0.8, which indicates that l1 and l3 are different kinds of logs. Similarly, l2 and l3 are also different kinds of logs.

[0063]

[0064] Table 1

[0065] According to the embodiments of the present disclosure, the longest common subsequence matching only measures the frequency of the largest characteristic item appearing in the log text, and ignores the semantic difference between the texts. Based on this, the type characteristics of the logs can be extracted, and the semantic similarity between the logs can be determined according to the type characteristics.

[0066] According to the embodiments of the present disclosure, the type characteristics of the logs can be extracted from the log output statements of the logs. For convenience of comparison, the extracted type characteristics can be stored in a log type template.

[0067] Table 2 exemplarily shows the log type template according to the embodiments of the present disclosure. As shown in Table 2, the storage structure of the log type template is key-value, the key is the index, and the value is the element in the log type template, which is used to record the type characteristics of the logs.

[0068] key value 001 No alive provider*for com.jd.b2b.* 002 connet*error xxxx ...

[0069] Table 2

[0070] Then, the similarity of the two logs is determined based on the degree of overlap of the types of the two logs, and the calculation formula of the type similarity is:

[0071]

[0072] wherein, tn l，l′ represents the number of the same type features contained in the two logs, and tmin(l, l') represents the smaller value of the type feature number of l' and the type feature number of l. The value range of TS is [0, 1], which represents the degree of overlap of the two logs, and the higher the value, the greater the semantic similarity of the two logs, and the lower the value, the smaller the semantic similarity of the two logs. According to the embodiment of the present disclosure, the longest common subsequence matching and the log type matching can be combined, and the improved similarity measurement formula is obtained based on this:

[0073] Sim(l, l') = SS(l, l') * TS(l, l')

[0074] wherein, Sim(l, l') is the semantic similarity of the logs l and l', SS(l, l') represents the content similarity of the logs l and l', and TS(l, l') represents the type similarity of the logs l and l'.

[0075] After obtaining the semantic similarity between the to-be-output logs through the log similarity calculation stage T2, the log record flow rate control stage T3 is entered to control the flow of the to-be-output logs.

[0076] According to the preset order, a log is obtained from the to-be-output logs, and it is judged whether the log belongs to the recorded log category. If not, the log category of the log is recorded. If yes, the number of the log category to which the log belongs is increased by one.

[0077] For example, in the embodiment, the preset order can be, for example, the time sequence of receiving the logs, and the logs received earlier are in the front order and the logs received later are in the rear order. In the actual application process, other orders can also be used as the preset order, which is not limited in the present disclosure.

[0078] In the embodiment, the log category can be recorded in the HashTable. The number of occurrences of the log category can also be recorded in the HashTable. In the actual application process, the log category and the number of occurrences of the log category can also be stored in other data structures, which is not limited in the present disclosure.

[0079] Figure 4 The method for processing logs in an abnormal scenario according to the embodiment of the present disclosure is schematically shown. As shown in FIG. 4, the method comprises the following stages: Figure 4As shown. Taking error alarm logs as an example, when a large number of errors occur suddenly, the log library receives a large number of error alarm logs, i.e., exception logs. The log library uses the error number as the key to write each type of exception log into a HashTable. If the exception log belongs to a type that has already been recorded in the HashTable, only the log count of that log type is increased, and it is not written again.

[0080] After the T3 phase of log recording flow control ends, the records in the HashTable can be output to the log file, and the next round of flow control detection cycle can be started.

[0081] According to embodiments of this disclosure, through the traffic detection phase T1, the log similarity calculation phase T2, and the log recording flow rate control phase T3, traffic control can be performed on the logs when peak traffic arrives, thereby ensuring the output performance of the logs. Furthermore, the log traffic control operation and output operation are executed asynchronously, ensuring that the real-time performance of the business logic is not constrained by the disk I / O performance of the log files.

[0082] The following is for reference. Figure 5 In conjunction with another specific embodiment, Figure 2 The method shown will be further explained. Those skilled in the art will understand that the following example embodiments are only for understanding this disclosure, and this disclosure is not limited thereto.

[0083] Figure 5 A flowchart illustrating a method for processing logs according to another embodiment of this disclosure is shown schematically. Figure 5 As shown, the log processing method according to embodiments of this disclosure can perform time-slice splitting, dividing the processing into multiple fixed window periods. Real-time traffic, i.e., the number of requests generating real-time logs, is detected within each fixed window period, and the traffic logs generated by these real-time requests are obtained.

[0084] If the number of traffic logs received within a traffic detection period exceeds a threshold, the received traffic logs are processed asynchronously. Each traffic log is read sequentially, and a log semantic matching algorithm is used to determine if there is already matching data of the same log type in the log buffer. If not, the log type is recorded in the hashtable of the log buffer. If it exists, the next log is retrieved and the above operation is repeated. Specifically, for newly received traffic logs outside of the traffic detection period, the similarity between the traffic log and the logs in the log buffer is calculated and compared. If they are the same, they are merged; if they are different, the log is processed asynchronously separately.

[0085] Figure 6 A block diagram of an apparatus according to an embodiment of the present disclosure is shown schematically. Figure 6As shown, the apparatus 600 includes an obtaining module 610, a determining module 620, a merging module 630, and an outputting module 640.

[0086] The obtaining module 610 is configured to obtain a plurality of to-be-output logs received in a traffic detection period.

[0087] The determining module 620 is configured to determine semantic similarity between the plurality of to-be-output logs in a case where a quantity of the plurality of to-be-output logs exceeds a request quantity threshold.

[0088] The merging module 630 is configured to merge the plurality of to-be-output logs according to the semantic similarity between the plurality of to-be-output logs, to obtain at least one target log.

[0089] The outputting module 640 is configured to output the at least one target log.

[0090] According to an embodiment of the present disclosure, by merging logs according to semantic similarity of the logs in a case where a quantity of logs received in a traffic detection period exceeds a traffic detection threshold, the quantity of output logs can be reduced, and the output performance of the logs can be ensured.

[0091] Any one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure, or at least part of the functions of any one or more of the modules, sub-modules, units, sub-units can be implemented in one module. Any one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure can be split into multiple modules for implementation. Any one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure can be implemented at least in part as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in software, hardware, and firmware in any one or in an appropriate combination of any several of the three implementation manners. Alternatively, one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure can be implemented at least in part as computer program modules that can perform corresponding functions when the computer program modules are run.

[0092] For example, any plurality of the acquisition module 610, determination module 620, merging module 630, and output module 640 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present disclosure, at least one of the acquisition module 610, determination module 620, merging module 630, and output module 640 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 610, determination module 620, merging module 630, and output module 640 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0093] Figure 7 A block diagram of a computer system suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Figure 7 The computer system shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0094] like Figure 7 As shown, a computer system 700 according to an embodiment of this disclosure includes a processor 701, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0095] In the RAM 703, various programs and data required for the operation of the system 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other via the bus 704. The processor 701 performs various operations of the method flow according to the embodiments of the present disclosure by executing the programs in the ROM 702 and / or the RAM 703. It should be noted that the programs can also be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 can also perform various operations of the method flow according to the embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0096] According to the embodiments of the present disclosure, the system 700 can further include an input / output (I / O) interface 705, which is also connected to the bus 704. The system 700 can further include one or more of the following components connected to the I / O interface 705: an input part 706 including a keyboard, a mouse, and the like; an output part 707 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage part 708 including a hard disk, and the like; and a communication part 709 including a network interface card such as a LAN card, a modem, and the like. The communication part 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as necessary. A removable recording medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 710 as necessary, so that a computer program read therefrom is installed in the storage part 708 as necessary.

[0097] Embodiments of the present disclosure also include a computer program product including a computer program containing program codes for executing the method provided by the embodiments of the present disclosure, which is used to cause the computing device to implement the method of processing the log provided by the embodiments of the present disclosure when the computer program product is run on the computing device.

[0098] The above-described functions defined in the system / apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 701. According to the embodiments of the present disclosure, the system, apparatus, module, unit, and the like described above can be implemented by computer program modules. In one embodiment, the computer program can rely on a tangible storage medium of an optical storage device, a magnetic storage device, and the like. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium, and installed and executed through the communication part 709, and / or installed from the removable recording medium 711. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, and the like, or any appropriate combination of the above.

[0099] According to an embodiment of the present disclosure, program code for execution of the computer programs provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using a high-level procedural and / or object-oriented programming language, and / or an assembly / machine language. The programming languages include, but are not limited to, a programming language such as Java, C++, python, “C” language, or a similar programming language. The program code can be executed entirely on a user computing device, partially on a user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected through the Internet by using an Internet service provider).

[0100] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present disclosure.

[0101] According to an embodiment of the present disclosure, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a portable compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer readable storage medium can include the ROM 702 and / or the RAM 703 described above, and / or one or more memories other than the ROM 702 and the RAM 703.

[0102] The computer program product of the first aspect can include one or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform the operations of the first aspect. The one or more non-transitory computer-readable media can include, for example, magnetic media such as one or more magnetic disks, magnetic tapes or cassettes; optical media such as one or more compact discs (CD), optical discs or discs (for example, DVD, Blu-ray Disc®, digital video disc, ultra density disc, ultra-compact disc, any optical media, etc.); semiconductor media such as solid state hard drives (for example, flash memory, solid state USB drives, etc.); any other suitable medium; or any suitable combination of media.

[0103] Those skilled in the art will understand that features of the various embodiments and / or claims of the present disclosure can be combined or / and integrated with one another, even though such a combination or integration is not expressly disclosed in the present disclosure. In particular, the features of the various embodiments and / or claims of the present disclosure can be combined and / or integrated with one another in any number of ways, without departing from the spirit and scope of the present disclosure. All such combinations and / or integrations are within the scope of the present disclosure.

[0104] The above describes embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although the above describes each embodiment separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and all such substitutions and modifications shall fall within the scope of the present disclosure.

Claims

1. A method for processing logs, comprising: setting a plurality of fixed window periods, the fixed window periods comprising a traffic detection period and a traffic log control execution period, the traffic log control execution period comprising a log similarity calculation stage and a log record flow rate control stage; for each of the fixed window periods, obtaining a plurality of to-be-output logs received in the traffic detection period; in a case where a number of the to-be-output logs exceeds a traffic detection threshold, entering the log similarity calculation stage to determine semantic similarities between the to-be-output logs; entering the log record flow rate control stage to merge the to-be-output logs according to the semantic similarities between the to-be-output logs, to obtain at least one target log, wherein the target log is determined from the merged to-be-output logs; and outputting the at least one target log; determining whether the target log belongs to a log category recorded in a hash table, and if not, recording a log category of the target log, and if so, increasing a number of the log category to which the target log belongs by one; outputting records in the hash table to a log file; and starting a traffic detection period in a next fixed window period. the determining the semantic similarities between the to-be-output logs comprises: determining content similarities and / or type similarities between the to-be-output logs, respectively; and determining the semantic similarities between the to-be-output logs according to the content similarities and / or the type similarities. the determining the content similarities between the to-be-output logs comprises: determining same word sequences contained in any two of the to-be-output logs; determining a word sequence containing the most words in the same word sequences as a longest common subsequence; and determining content similarities of the two to-be-output logs according to the longest common subsequence and a number of words contained in each of the two to-be-output logs. the determining the type similarities between the to-be-output logs comprises: determining type feature sets of each of any two of the to-be-output logs, wherein the type feature sets comprise at least one type feature; determining a number of same type features contained in the type feature sets of the two to-be-output logs; and determining a type similarity between the two to-be-output logs according to the number of same type features. the merging the to-be-output logs according to the semantic similarities between the to-be-output logs to obtain at least one target to-be-output log comprises: dividing the to-be-output logs into at least one log set according to the semantic similarities between the to-be-output logs, wherein to-be-output logs contained in each of the at least one log set have semantic similarities exceeding a similarity threshold; and determining any one of to-be-output logs in each of the log sets as the target log. ​ ​ ​ ​ ​ ​ ​ 2. The method of claim 1, wherein, ​ ​ ​ 3. The method of claim 2, wherein, ​ ​ ​ ​ 4. The method of claim 2 or 3, wherein, ​ ​ ​ ​ 5. The method of claim 1, wherein, ​ ​ ​ 6. The method of claim 5, further comprising: counting a number of logs to be outputted contained in each of the log sets; and outputting the number of logs to be outputted contained in each of the log sets.

7. An apparatus for processing logs, comprising: the apparatus is configured to set a plurality of fixed window periods, the fixed window periods comprising a traffic detection period and a traffic log control execution period, the traffic log control execution period comprising a log similarity calculation stage and a log record flow rate control stage; an obtaining module configured to, for each of the fixed window periods, obtain a plurality of logs to be outputted received in the traffic detection period; a determining module configured to, in a case that a number of the logs to be outputted exceeds a threshold of a request number, enter the log similarity calculation stage, and determine semantic similarities between the logs to be outputted; a merging module configured to, enter the log record flow rate control stage, and merge the logs to be outputted according to the semantic similarities between the logs to be outputted, to obtain at least one target log, wherein the target log is determined from the merged logs to be outputted; and an outputting module configured to output the at least one target log; the apparatus is further configured to determine whether the target log belongs to a log category recorded in a hash table, and if not, record a log category of the target log, and if so, increase a number of the log category to which the target log belongs by one; output records in the hash table to a log file; and start a traffic detection period in a next fixed window period.

8. A computing device, comprising: one or more processors; a storage device for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to implement the method of any one of claims 1 to 6.

10. A computer program product comprising computer executable instructions that, when executed, implement the method of any one of claims 1 to 6. ​

Images