Dynamic abnormal Chinese log early warning method and device and medium

Abstract

The embodiment of the invention discloses a dynamic abnormal Chinese log early warning method and device and a medium. The method comprises the following steps: regularly reading abnormal logs of a plurality of continuous time periods; the abnormal log quantity ratio of each time period to the first time period is calculated; if the abnormal log quantity ratio is higher than a proportion threshold value corresponding to each time period, abnormal early warning is carried out, and each proportion threshold value is determined according to the abnormal log growth proportion of each time period and the first time period in the safe state; otherwise, no abnormal early warning is carried out. According to the embodiment, the accuracy and efficiency of log detection are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of log detection, and in particular to a dynamic abnormal Chinese log early warning method, device and medium. Background Art

[0002] Program logs refer to information that records the running status of a program, and are mainly used for program debugging, information persistence, and monitoring the execution process of a program. Usually, Chinese program logs include both English and Chinese content.

[0003] For abnormal log inspection, the existing technical solution is to set the keywords of the abnormal log in advance, search the log by the keywords, and notify when the keywords of the abnormal log appear or reach a certain number; or browse manually.

[0004] This method has the following defects: the keywords of the abnormal logs need to be set in advance, and the logs without set keywords cannot be detected, resulting in missed reports; the threshold of the keyword appearance needs to be set in advance, and the threshold estimation is difficult to determine; manual retrieval is inefficient. Summary of the invention

[0005] The embodiments of the present invention provide a dynamic abnormal Chinese log early warning method, device and medium to solve at least one of the above technical problems.

[0006] In a first aspect, an embodiment of the present invention provides a dynamic abnormal Chinese log early warning method, comprising:

[0007] Regularly read abnormal logs for multiple consecutive periods;

[0008] Calculate the ratio of abnormal log volume in each period to the first period respectively;

[0009] If the ratio of each abnormal log volume is higher than the ratio threshold corresponding to each time period, an abnormal warning is issued, wherein each ratio threshold is determined according to the growth ratio of abnormal logs in each time period and the first time period under the safe state;

[0010] Otherwise, no abnormal warning is issued.

[0011] In a second aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0012] one or more processors;

[0013] a memory for storing one or more programs,

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement the dynamic abnormal Chinese log warning method described in any embodiment.

[0015] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the dynamic abnormal Chinese log warning method described in any embodiment is implemented.

[0016] In summary, the embodiment of the present invention provides a dynamic abnormal Chinese log warning method, which determines whether an obvious abnormality or serious abnormality has occurred and issues a warning by calculating the ratio of the daily log volume of each period in multiple consecutive periods to the first period. This method does not need to set keywords to search the logs, but dynamically perceives abnormal Chinese content logs through the log growth trend, and has the ability to detect unknown abnormal types, thereby improving the accuracy of detection and warning; this method also does not need to set a quantity threshold, and the sensitivity can be adjusted, thereby improving the stability of detection and warning.

[0017] In particular, compared with the ratio of the log volume between every two consecutive time periods, this embodiment uses the ratio of the log volume in each time period to the log volume in the first time period, which provides a unified benchmark for the comparison of thresholds of multiple consecutive time periods, so that the ratio of multiple consecutive time periods can be used to reflect the change pattern of the abnormal log volume over a long period of time, and the long-term change pattern can be used to judge the anomalies that really need warnings or human intervention, avoiding misidentification of short-term changes between adjacent time periods as anomalies that need warnings and cannot self-recover, thereby improving the accuracy of anomaly detection and warning and reducing the false alarm rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of a dynamic abnormal Chinese log early warning method provided by an embodiment of the present invention;

[0020] Figure 2 It is a flow chart of another dynamic abnormal Chinese log early warning method provided by an embodiment of the present invention;

[0021] Figure 3 This is a flow chart of another dynamic abnormal Chinese log early warning method provided by an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0024] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0025] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0026] Figure 1 The present invention provides a flowchart of a dynamic abnormal Chinese log early warning method. The method is applicable to the case of abnormal early warning according to the Chinese abnormal log, and is executed by an electronic device.

[0027] like Figure 1 As shown, the method specifically includes:

[0028] S110. Regularly read abnormal logs of multiple consecutive time periods.

[0029] The logs in this embodiment refer to the descriptive logs of the program, not the structured logs. The descriptive logs include normal logs and abnormal logs. This embodiment processes the daily logs in the descriptive logs. Optionally, the abnormal logs can be obtained through an existing log storage service, or by calling a third-party interface, which is not specifically limited in this embodiment.

[0030] More specifically, N segments of log content can be obtained from the log storage service at fixed time intervals, and the time interval is the length of each of the above time periods. For example, if the length of each time period is 5 minutes, then the abnormal logs in a series of intervals such as 0-5 minutes, 6-10 minutes, 11-15 minutes, etc. in the past can be obtained at a certain moment as the object of this processing. After a period of time after the processing is completed, the abnormal logs in a series of intervals such as 0-5 minutes, 6-10 minutes, 11-15 minutes, etc. in the past at that moment are obtained as the object of the next processing. The "this time" and "next time" constitute a cyclical periodic detection.

[0031] S120, respectively calculating the ratio of the abnormal log volume in each time period to the first time period.

[0032] After obtaining abnormal logs of multiple consecutive periods in this detection, the abnormal logs of each period can be preprocessed first to remove the log content that has no impact, non-important server logs and information logs. Then, the ratio of the amount of abnormal logs after preprocessing in each period to the amount of abnormal logs after preprocessing in the first period is calculated.

[0033] S130. If the ratio of each abnormal log volume is higher than the ratio threshold corresponding to each time period, an abnormal warning is issued, wherein each ratio threshold is determined according to the growth ratio of the abnormal logs in each time period and the first time period under a safe state.

[0034] When the program is running normally, the amount of abnormal logs in multiple consecutive time periods is relatively stable, and even if there is an increase, it will be controlled within a certain range. If the program has obvious or serious abnormalities, the abnormal logs will accumulate and increase in a short period of time, and will continue for a long time with a high amount of abnormal logs. In order to screen such abnormalities, this embodiment sets an abnormal log ratio threshold for each extracted continuous time period according to the growth ratio of abnormal logs in each time period to the first time period under the safe state. For example, the growth law of abnormal logs under the information security state will not exceed the exponential power of 2, that is, assuming that the amount of abnormal logs in the first time period is n, the amount of abnormal logs in the following consecutive time periods will not exceed n×2, n×2, etc. 2 , n×2 3 …. Using this rule, we can set the ratio thresholds of each time period to 1, 2, 2 in chronological order. 2 , 2 3 …, each daily log ratio threshold can form a detection curve.

[0035] After the detection curve is determined, the ratio of the abnormal log volume in each time period to the first time period is compared with the detection curve. If the ratio in each time period is higher than the threshold of the corresponding time period in the detection curve, it can be determined that a more obvious or serious abnormality has occurred, and an abnormal warning is required.

[0036] S140: If any of the abnormal log volume ratios is lower than or equal to the ratio threshold of the corresponding time period, no abnormal warning is issued.

[0037] On the contrary, if, among the multiple time periods, the ratio of the abnormal log volume of a certain time period to the abnormal log volume of the first time period is lower than or equal to the ratio threshold of the certain time period, it is determined that the current abnormality is not a relatively obvious or relatively prominent abnormality, and no abnormal warning is required. Specifically, if it is lower than or equal to the ratio threshold of the current time period, it means that the abnormal log volume of the current time period is relatively small and no special processing is required. This method can reduce the misjudgment of many obvious abnormalities. For example, among the multiple time periods, the abnormal log ratios of the first few time periods are higher than the detection curve, but the abnormal log ratios of the later few time periods return to below the detection curve, which indicates that the abnormal logs of the time period have been restored to normal levels through program self-repair, etc., and no additional warning is required.

[0038] For obvious or serious anomalies, we can analyze the logs corresponding to the anomalies and summarize the log flag fields, such as the IP of the log server, and take out the top three with the highest number; or summarize the log description content, such as classifying and sorting the Chinese alarm information, and take out the top three with the highest number. Then, we will notify the business engineer of the warning and analysis situation through WeChat SMS. The above process can also be combined with Figure 2 The flowchart shown is understood.

[0039] As described in S110, after the current abnormality detection is completed, the next abnormality detection can be performed regularly. The multiple time periods in the next detection can be seamlessly connected with the multiple time periods in this detection, or they can be partially repeated, or they can be separated by a certain time, which is not specifically limited in this embodiment. Of course, the detection sensitivity of obvious abnormalities or serious abnormalities in the above three cases is also gradually reduced. In addition, the sensitivity of abnormal warning can be improved by at least one of the following methods: lowering each ratio threshold; reducing the interval length of each adjacent time period; reducing the number of the multiple consecutive time periods read regularly.

[0040] In summary, this embodiment provides a dynamic abnormal Chinese log warning method, which determines whether an obvious abnormality or serious abnormality has occurred and issues a warning by calculating the ratio of the daily log volume of each period in multiple consecutive periods to the first period. This method does not need to set keywords to search the logs, but dynamically perceives abnormal Chinese content logs through the log growth trend, and has the ability to detect unknown abnormal types, thereby improving the accuracy of detection and warning; this method also does not need to set a quantity threshold, and the sensitivity can be adjusted, thereby improving the stability of detection and warning.

[0041] In particular, compared with the ratio of the log volume between every two consecutive time periods, this embodiment uses the ratio of the log volume in each time period to the log volume in the first time period, which provides a unified benchmark for the comparison of thresholds of multiple consecutive time periods, so that the ratio of multiple consecutive time periods can be used to reflect the change pattern of the abnormal log volume over a long period of time, and the long-term change pattern can be used to judge the anomalies that really need warnings or human intervention, avoiding misidentification of short-term changes between adjacent time periods as anomalies that need warnings and cannot self-recover, thereby improving the accuracy of anomaly detection and warning and reducing the false alarm rate.

[0042] Furthermore, the abnormal log volume in the above embodiment may refer to the total abnormal log volume, but when there are many abnormal types or a refined abnormal warning is required, the abnormal log volume may also refer to the abnormal log volume of each abnormal type. In a specific implementation, when calculating the ratio of the abnormal log volume of each time period to the first time period in S120, multiple Chinese fields in the abnormal log of each time period may be first identified; then, for each Chinese field, the number of times the current Chinese field appears in the abnormal log of each time period is recorded, that is, the abnormal log volume corresponding to the current Chinese field; finally, the ratio of the number of times the current Chinese field appears in the abnormal log of each time period to the number of times the current Chinese field appears in the abnormal log of the first time period is calculated.

[0043] Correspondingly, when judging whether the ratio of each abnormal log volume is higher than the ratio threshold corresponding to each time period in S130, it can be judged whether the ratio corresponding to the current Chinese field in each time period is higher than the ratio threshold corresponding to each time period. If the ratio of each time period is higher than the ratio threshold of the corresponding time period, it indicates that an obvious abnormality corresponding to the current Chinese field has occurred. The abnormal type of the current warning can be judged according to the text content of the current Chinese field, and the abnormal type can be added to the abnormal warning. If one of the time periods does not meet the above conditions, no warning is required. The above process can also be combined with Figure 3 The flowchart shown is understood.

[0044] In the above embodiment, each Chinese field is assumed to be a possible abnormal type, and for each possible abnormal type, the number of logs that appear in the abnormal type is counted, the abnormal type ratio is calculated, and the threshold comparison is performed to determine each possible obvious abnormality, and finally the type of obvious abnormality is determined by the field content. In this method, there is no need to set abnormal keywords, but each group of recognizable continuous Chinese characters (these characters often correspond to descriptions of abnormal types, and other descriptions are mostly English characters) is used as a recording target, and the total amount of logs containing the field is recorded, so that obvious abnormalities are determined by threshold comparison, which will not cause the omission of unknown abnormal keywords, but can also filter out short-term abnormalities that can be self-recovered, and can also distinguish abnormal types by field content, further improving the accuracy and efficiency of detection.

[0045] Furthermore, since the above steps treat the continuous Chinese characters in the exception log as a Chinese field, if the field is too long, it may include some auxiliary texts in addition to the exception type. For example, the Chinese field read is "Sorry, the network cannot be connected", in which "Sorry" is the auxiliary text, and "The network cannot be connected" is the content about the exception type. At this time, when recording the number of times the current Chinese field appears in the exception log of each time period, it may appear that the number of times the Chinese field appears in the exception log of the subsequent time periods is 0. If this happens, the Chinese field can be segmented, such as the above Chinese field is segmented into two fields, "Sorry" and "The network cannot be connected"; then return to the operation of recording the number of field occurrences, and record the number of times the two Chinese fields after segmentation appear in the exception log of each time period. At this time, the valid field "The network cannot be connected" can be correctly counted and obvious / serious anomalies can be detected.

[0046] This embodiment uses continuous Chinese fields to distinguish the types of anomalies that may be included in the exception log, counts the number of occurrences of each new Chinese field and calculates the ratio, and detects and warns the type of anomalies that may correspond to each Chinese field. For the Chinese fields that need to be warned, the true type of anomaly is determined by the field content. This method also does not require the setting of abnormal keywords, which will not cause the omission of unknown abnormal keywords, and can filter out short-term anomalies that can self-recover. At the same time, it can also distinguish the types of anomalies, further improving the accuracy and efficiency of detection. In particular, for longer Chinese fields, the auxiliary content that may be contained is fully considered, and the long fields are semantically segmented by word segmentation, and the effective field content is extracted for ratio detection, which further improves the accuracy of detection.

[0047] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 4As shown, the device includes a processor 60, a memory 61, an input device 62 and an output device 63; the number of processors 60 in the device can be one or more. Figure 4 A processor 60 is taken as an example; the processor 60, the memory 61, the input device 62 and the output device 63 in the device can be connected by a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0048] The memory 61 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the dynamic abnormal Chinese log early warning method in the embodiment of the present invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 61, that is, realizing the above-mentioned dynamic abnormal Chinese log early warning method.

[0049] The memory 61 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 61 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include a memory remotely arranged relative to the processor 60, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0050] The input device 62 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 63 may include a display device such as a display screen.

[0051] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the dynamic abnormal Chinese log early warning method of any embodiment is implemented.

[0052] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, a device or a device or used in combination with it.

[0053] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0054] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0055] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A dynamic abnormal Chinese log early warning method, characterized in that: include: Regularly read abnormal logs for multiple consecutive periods; Calculate the ratio of abnormal log volume in each period to the first period respectively; If the ratio of each abnormal log volume is higher than the ratio threshold corresponding to each time period, an abnormal warning is issued, wherein each ratio threshold is determined according to the growth ratio of abnormal logs in each time period and the first time period under the safe state; Otherwise, no abnormal warning is issued.

2. The method according to claim 1, characterized in that The calculating of the ratio of the abnormal log volume in each time period to the first time period respectively includes: Pre-process abnormal logs in each period to remove irrelevant log content, non-important server logs and information logs; Calculate the ratio of abnormal log volume in each period to that in the first period after preprocessing.

3. The method according to claim 1, characterized in that The calculating of the ratio of the abnormal log volume in each time period to the first time period respectively includes: Identify multiple Chinese fields in abnormal logs for each period; For each Chinese field, record the number of times the current Chinese field appears in the exception logs of each period; The ratio of the number of times the current Chinese field appears in the exception log of each time period to the number of times the current Chinese field appears in the exception log of the first time period is calculated respectively.

4. The method according to claim 3, characterized in that If the ratio of each abnormal log volume is higher than the ratio threshold corresponding to each time period, an abnormal warning is issued, including: If the ratio corresponding to each time period is higher than the ratio threshold corresponding to each time period, an abnormal warning is issued.

5. The method according to claim 3, characterized in that: After the abnormal warning is performed, the method further includes: The abnormal type of the current warning is determined according to the text content of the current Chinese field.

6. The method according to claim 3, characterized in that The identifying multiple Chinese fields in the abnormal logs of each time period includes: for the abnormal logs of any time period, reading continuous Chinese characters in the logs, and treating the continuous Chinese characters as a Chinese field; Correspondingly, the recording of the number of times the current Chinese field appears in the exception log of each time period includes: recording the number of times the Chinese field appears in the exception log of each time period; if the number of times it appears in all logs is 0, then segmenting the Chinese field, and recording the number of times the segmented Chinese field appears in the exception log of each time period.

7. The method according to claim 1, characterized in that The ratio threshold of each time period increases exponentially with the time sequence number of each time period.

8. The method according to claim 1, characterized in that Also includes: Improve the sensitivity of abnormal warning by at least one of the following methods: Lower the thresholds for each ratio; Reduce the length of the interval between adjacent time periods; The number of the plurality of consecutive time periods that are periodically read is reduced.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the dynamic abnormal Chinese log warning method described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the dynamic abnormal Chinese log early warning method described in any one of claims 1-8 is implemented.

Images