Log collection and analysis method and system based on weight

Abstract

The invention discloses a weight-based log collection and analysis method and system, and belongs to the technical field of log collection and analysis. The problem that in the prior art, due to the fact that a traditional log collection and analysis system does not consider the weights of different analysis entries, the analysis result accuracy is low is solved. A system registration module is adopted to carry out system registration on a third-party application system, and a registration code of the third-party application system is generated; starting the transmission code to pull the log data, and performing encryption compression to obtain pulled log data; a data storage module is adopted to store the log data pulled from the third-party application system into a local memory of the log acquisition and analysis system; and calling the log data pulled in the data storage module by adopting a log analysis module, introducing an automatic adjustment weight to compare and analyze the log data, and displaying an obtained log data analysis result. The accuracy of log collection and analysis results is effectively improved, and the method can be applied to collection and analysis of log data.

Description

Technical Field

[0001] The present invention relates to a log collection and analysis method and system, and in particular to a weight-based log collection and analysis method and system, belonging to the technical field of log collection and analysis. Background Art

[0002] As a key tool for modern information technology infrastructure, the log collection and analysis system realizes functions such as troubleshooting, performance optimization, security auditing, and automated operation and maintenance through in-depth mining and analysis of system logs. At home and abroad, log collection and analysis systems are widely used in many industries such as finance, telecommunications, and medical care, significantly improving operation and maintenance efficiency and business insight capabilities. Especially in China, with the deepening of digital transformation, the market for log collection and analysis systems has flourished and has become an important means for enterprises to ensure system stability and security.

[0003] The current log collection and analysis system does not consider the weight of different analysis items during the analysis process. That is, the data information of all systems has the same importance during the analysis process, but in some real applications, some systems are less important than other systems, so their weight in the analysis process is lower. Failure to consider the system weight issue will lead to inaccurate analysis results, or the analysis results lack weight factors.

[0004] In summary, a weight-based log collection and analysis method and system is needed. Summary of the invention

[0005] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0006] In view of this, in order to solve the problem of low accuracy of analysis results caused by failure to consider the weights of different analysis items in the traditional log collection and analysis system in the prior art, the present invention provides a weight-based log collection and analysis method and system.

[0007] Technical solution 1 is as follows: A weight-based log collection and analysis method, comprising the following steps:

[0008] S1. Use the system registration module to enter the registration information of the third-party application system to register the third-party application system and generate a registration code for the third-party application system;

[0009] S2. Use the transmission module to encrypt and compress the registration code, the packaged transmission code, the configuration file required for the transmission code, and the execution file, send it to the third-party application system deployment address for decompression, modify the transmission code configuration file and the execution file, start the transmission code to pull the log data and encrypt and compress it, and obtain the pulled log data;

[0010] S3 uses a data storage module to store log data pulled from a third-party application system to the local memory of the log collection and analysis system;

[0011] S4. Use the log analysis module to call the log data pulled from the data storage module, perform comparative analysis on it, and display the obtained log data analysis results.

[0012] Further, in said S2, the transmission code is used to pull the log data of the deployment machine where the third-party application system is located;

[0013] After the registration code, the packaged transmission code, the configuration file required by the transmission code, and the execution file are sent to the deployment machine where the third-party application system is located, i.e., the deployment address of the third-party application system, the registration code in the configuration file is modified to the registration code of the third-party application system, and the database location in the execution file is modified to the actual link database of the third-party application system;

[0014] After the transmission code is started, it is determined whether the registration code of the third-party application system in the configuration file is correct. If it is correct, the log data pulling program is run to pull log data according to the custom pulling frequency. Otherwise, the log data pulling program is terminated and automatically exited.

[0015] When the registration code of the third-party application system is verified, the above registration code is used as the random number seed of the encryption algorithm key to generate an encryption key, and the pulled log data is encrypted using the encryption method of the encryption algorithm.

[0016] Furthermore, in S4, the comparative analysis includes frequency comparative analysis and quantity comparative analysis. For the frequency comparative analysis, the frequency of occurrence of the event is compared to calculate the frequency of occurrence of the event P = {p1, p2, ..., p N}, where N is the number of items involved in the comparison, p N is the frequency of the Nth entry, P∈{times / second, times / minute, times / hour, etc.}, and the events are sorted according to the frequency P of the event;

[0017] For quantitative comparison analysis, the total number of occurrences of the comparison event S = {s1, s2, …, s N}, sort the events according to the total number of times S that the events occur;

[0018] According to the general comparison formula, automatic adjustment weights are introduced for comparative analysis;

[0019] The general comparison formula is expressed as:

[0020]

[0021] in, is the final comparison value, α is the weight value, and α={α1,α2,…,α N},0<α N <1, C is the initial comparison value, which is the frequency P of the event or the total number of times the event occurs S;

[0022] When performing frequency comparison analysis, C = P, Among them, α N is the weight value of the Nth entry. When comparing by number of times, C = S. where α N Represents the weight value of the Nth entry;

[0023] Automatic adjustment of weights is expressed as follows: according to the ranking without weight, i.e., the frequency of event occurrence P and the total number of event occurrence S, the weight rate value is determined. When the ranking of an item is improved for several consecutive days, and the number of consecutive change days reaches the cumulative number of days threshold, the weight value of the item is increased, and the upper limit of the increase is 1, that is, the importance of the item is increased. When the ranking of an item drops for several consecutive days, the weight value of the item is reduced, and the lower limit is 0. If a reverse change occurs during the continuous change process, the cumulative number of days is reset to zero and counted again. Among them, the cumulative number of days threshold and the weight rate value are preset values.

[0024] According to the final comparison value Get the log data analysis results sorted {E1, E2, ..., E N}, where E N is an entry ranked N, and the final comparison value corresponding to each entry is in, is the final comparison value corresponding to the entry ranked N,

[0025] Technical Solution 2: A weight-based log collection and analysis system, used to implement a weight-based log collection and analysis method described in Technical Solution 1, comprising a system registration module, a transmission module, a data storage module and a log analysis module;

[0026] The system registration module, transmission module, data storage module and log analysis module are connected in sequence;

[0027] The system registration module is used to register the third-party application system;

[0028] The transmission module is used to encrypt and compress the registration code, the packaged transmission code, the configuration file required by the code, and the execution file, and send them to the deployment address of the third-party application system;

[0029] The data storage module is used to store the log data pulled from the third-party application system into the local storage of the log collection and analysis system;

[0030] The log analysis module is used to analyze log data and display key information in the log data.

[0031] The beneficial effects of the present invention are as follows: the present invention uses a system registration module to input the registration information of a third-party application system to perform system registration on the third-party application system, generates a unique registration code for the third-party application system to identify the third-party application system, starts a transmission code to pull log data and encrypts and compresses it, obtains the pulled log data, and performs comparative analysis on it, displays the obtained log data analysis results, and realizes log collection and analysis; the present invention introduces automatic adjustment weights when performing comparative analysis, increases analysis factors, and provides a log collection and analysis system with more accurate analysis results, which can more accurately discover potential problems, improve the company's operation and maintenance efficiency and business insight capabilities, and provide strong guarantees for the company's stable operation and sustainable development. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0033] Figure 1 It is a flowchart of a log collection and analysis method based on weight;

[0034] Figure 2 It is a structural diagram of a weight-based log collection and analysis system;

[0035] Figure 3 A schematic diagram of the process flow of an embodiment for pulling log data;

[0036] Figure 4 A flowchart of an embodiment of analyzing log data is shown in FIG.

[0037] Figure numbers: 1. System registration module; 2. Transmission module; 3. Data storage module; 4. Log analysis module. DETAILED DESCRIPTION

[0038] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0039] Example 1: Reference Figure 1-Figure 4 The present embodiment is described in detail, a log collection and analysis method based on weights, specifically comprising the following steps:

[0040] S1. Use the third-party application system of the system registration module to register the system and generate a registration code;

[0041] S2. Use the transmission module to encrypt and compress the registration code, the packaged transmission code, the configuration file required for the transmission code, and the execution file, send it to the third-party application system deployment address for decompression, modify the transmission code configuration file and the execution file, start the transmission code to pull the log data and encrypt and compress it, and obtain the pulled log data;

[0042] S3 uses a data storage module to store log data pulled from a third-party application system to the local memory of the log collection and analysis system;

[0043] S4. Use the log analysis module to call the log data pulled from the data storage module, perform comparative analysis on it, and display the obtained log data analysis results.

[0044] Specifically, in this embodiment, the present invention is applied to the field of smart water affairs, specifically a smart water affairs system, which includes a smart water-saving system, a smart drainage system, and a smart water supply system. The execution language is Java, the data is relational data, the database is MySQL, and the encryption algorithm is AES256.

[0045] The registration information of the smart water system includes the name of the smart water system, the source of the smart water system, etc. After the system is registered, a unique registration code of the smart water system is generated as the identification of the smart water system and as the subsequent determination information of the smart water system. At the same time, the registration code will also participate in the generation of the data transmission encryption key;

[0046] First, the administrator needs to determine the details of the smart water system. Then the administrator will include the registration information of the smart water system and the initial weight of the system analysis (the initial weight is between 0 and 1). The initial weight of the analysis is entered according to the specific analysis items. For example, the module activity analysis needs to enter the weight of each module of the smart water system. The initial weight value of each module of the smart water system defaults to 1. Generate a unique registration code uuid to identify the smart water system;

[0047] The local storage is a database that can store relational data or non-relational data. The specific database form needs to be determined according to the specific form of the log data. The storage process is completed through the mybatis framework. At the same time, since a large amount of log data is generated every day, the MySQL distributed deployment method will be used to store the log data to improve the storage capacity and disaster recovery capability of the log collection and analysis system. In the MySQL distributed deployment process, the log data segmentation solution selects horizontal segmentation to ensure the concurrency and response speed of the log data. The master-slave database synchronization method selects a dual-master mutual backup solution.

[0048] Furthermore, in S2, the transmission code is an independent deployment code for pulling log data of the machine where the smart water system is deployed;

[0049] When the registration code, the packaged transmission code, the configuration file required by the transmission code, and the execution file are sent to the deployment machine where the smart water system is located, i.e., the deployment address of the smart water system, the registration code in the configuration file is modified to the registration code of the smart water system, and the database location in the execution file is modified to the actual link database of the smart water system;

[0050] After starting the transmission code, determine whether the registration code of the smart water system in the configuration file is correct. If it is correct, run the log data pulling program to pull log data according to the custom pulling frequency. Otherwise, terminate the log data pulling program and exit automatically.

[0051] When the registration code of the smart water system is verified, the registration code is used as the random number seed of the encryption algorithm key to generate an encryption key, and the pulled log data is encrypted using the encryption method of the encryption algorithm.

[0052] Specifically, for encryption compression, you can use a compression tool to compress and add a compression password during compression. Specific compression formats include zip, rar, and 7z.

[0053] The key information of the configuration file required for code transmission includes database driver type, database connection address, database user name, database login password and registration code of the smart water system;

[0054] refer to Figure 3, start the transmission code Logsys.jar, first determine whether the registration code of the smart water system in the configuration file is correct, for example, by determining whether the local database of the log collection and analysis system has the registration code of the smart water system. If it does not exist, it is regarded as an incorrect registration code. If it is correct, start pulling the log data of the smart water system according to the custom pulling frequency. The custom frequency defaults to pulling once a day at midnight. If the registration code is wrong, the log data pulling program will terminate and exit automatically;

[0055] When the registration code of the smart water system is verified, the random number seed used as the AES256 encryption algorithm key is used to generate an encryption key, and then the AES256 encryption algorithm is used to encrypt the pulled log data;

[0056] The AES256 encryption process is as follows:

[0057] (1) Generate a secret key (use the registration code of the smart water system as a random number seed to generate the secret key):

[0058] Generate a key generator;

[0059] Generate a key using the registration code of the smart water system as a random number seed;

[0060] Key formatting: format the generated key into a byte array or Base64 string for easy storage and transmission;

[0061] (2) Encryption:

[0062] Use the secret key to initialize the encryptor and determine the encryption method;

[0063] Get the byte array of encrypted log data;

[0064] Encrypt the log data byte array according to the encryptor initialization method;

[0065] For data transmission, this embodiment uses the Kafka publish-subscribe messaging system. After the log data is encrypted, the secret key and the encrypted data will be published to the Kafka message queue and further encrypted through Kafka encrypted transmission. The log collection and analysis system will subscribe to the encrypted message and pull the encrypted message, that is, the encrypted log data, from the Kafka message queue. Then, the data is decrypted by AES256 decryption method to obtain the actual log data and store it in the local storage of the log collection and analysis system.

[0066] The AES256 decryption process is as follows:

[0067] Initialize the decryptor using the secret key;

[0068] Get the encrypted log data;

[0069] Use the decryptor to decrypt the encrypted log data to obtain the actual log data.

[0070] Furthermore, in S4, the comparative analysis includes frequency comparative analysis and quantity comparative analysis. For frequency comparative analysis, the comparison is performed based on the frequency of event occurrence, for example, times / second, times / minute, times / hour, etc., and the frequency of event occurrence P = {p1, p2, ..., p N}, where N is the number of items involved in the comparison, p N is the frequency of the Nth entry, P∈{times / second, times / minute, times / hour, etc.}, and the events are sorted according to the frequency P of the event;

[0071] For quantitative comparison analysis, the total number of occurrences of the comparison event S = {s1, s2, …, s N}, sort the events according to the total number of times S that the events occur;

[0072] According to the general comparison formula, automatic adjustment weights are introduced for comparative analysis;

[0073] The general comparison formula is expressed as:

[0074]

[0075] in, is the final comparison value, α is the weight value, and α={α1,α2,…,α N},0<α N <1, C is the initial comparison value, which is the frequency P of the event or the total number of times the event occurs S;

[0076] When performing frequency comparison analysis, C = P, Among them, α N is the weight value of the Nth entry. When comparing by number of times, C = S. where α N Represents the weight value of the Nth entry;

[0077] Automatic adjustment of weights is expressed as follows: according to the ranking without weight, i.e., the frequency of event occurrence P and the total number of event occurrence S, the weight rate value is determined. When the ranking of an item is improved for several consecutive days, and the number of consecutive change days reaches the cumulative number of days threshold, the weight value of the item is increased, and the upper limit of the increase is 1, that is, the importance of the item is increased. When the ranking of an item drops for several consecutive days, the weight value of the item is reduced, and the lower limit is 0. If a reverse change occurs during the continuous change process, the cumulative number of days is reset to zero and counted again. Among them, the cumulative number of days threshold and the weight rate value are preset values.

[0078] According to the final comparison value Get the log data analysis results sorted {E1, E2, ..., E N}, where E N is an entry ranked N, and the final comparison value corresponding to each entry is in, is the final comparison value corresponding to the entry ranked N,

[0079] Specifically, in order to avoid deviations in artificially set weight values, the present invention introduces automatic weight adjustment to solve the above problem. In the process of displaying the analysis results, the present invention uses the Vue front-end framework and the Element-ui framework as the basis to display the analysis results in a web manner, and at the same time combines the Echarts framework to draw the required display charts, and displays the analysis results in an intuitive and clear graphical manner;

[0080] In this embodiment, reference Figure 4 , the analysis items include system stability analysis, module activity analysis, and module health analysis. The specific analysis methods are as follows:

[0081] System stability analysis (frequency comparison analysis): First, calculate the monthly maintenance frequency of each smart water system. The maintenance frequency of the smart water-saving system, smart drainage system, and smart water supply system are n1, n2, and n3 respectively. The maintenance frequency of the smart water-saving system, smart drainage system, and smart water supply system are p respectively. wd1 、p wd2 and p wd3 ,but Among them, d start1 d start2 and d start3 are the earliest time when the stored log data of the current smart water-saving system, smart drainage system, and smart water supply system are generated, respectively. end1 d end2 and d end3 are the times of the current smart water-saving system, smart drainage system and smart water supply system respectively, and the maintenance frequency P of the smart water system is obtained. wd , P wd ={p wd1 ,p wd2 ,p wd3}, and then sort them according to maintenance frequency, that is, analyze the system stability by the frequency of problems in the smart water system, and the comparison process is carried out between each smart water system;

[0082] Module activity analysis (quantity comparison analysis): First, calculate the total amount of operation logs of each module of the smart water system, that is, calculate the number of operations of each module. Taking the smart water-saving system as an example, there are three modules, and the number of operations of the three modules is s cz1 、s cz2 and cz3 , then the operation times S of the smart water-saving system is obtained cz , S cz ={s cz1 ,s cz2 ,s cz3}, then sort the operation times, analyze the module activity by the module usage times, and compare the internal modules of each smart water system separately during the comparison process;

[0083] Module health analysis (quantity comparison analysis): First, calculate the total number of abnormal operation logs in the operation log of each module of the smart water system, that is, calculate the number of abnormal operation times of each module. Taking the smart water-saving system as an example, there are three modules, and the number of abnormal operation times of the three modules are s and s, respectively. yc1 、s yc2 and yc3 , then the number of abnormal operation times S of the smart water-saving system is obtained yc , S yc ={s yc1 ,s yc2 ,s yc3}, then sort the operation times, analyze the module activity by the module usage times, and compare the internal modules of each smart water system separately during the comparison process;

[0084] According to the general comparison formula, a comparative analysis of the three scenarios of this embodiment is as follows:

[0085] System stability analysis (frequency comparison analysis), that is, P in the general comparison formula is P in this scenario. wd ;

[0086] Module activity analysis (quantity comparison analysis) means that the S in the general comparison formula is S in this scenario. cz ;

[0087] Module health analysis (quantity comparison analysis) means that the S in the general comparison formula is S in this scenario. yc ;

[0088] Finally, the analysis results of the three analysis items are displayed in Echarts stacked bar charts.

[0089] Example 2: Reference Figure 2The present embodiment is described in detail, a weight-based log collection and analysis system, which is used to implement a weight-based log collection and analysis method described in embodiment 1, comprising a system registration module 1, a transmission module 2, a data storage module 3 and a log analysis module 4;

[0090] The system registration module 1, the transmission module 2, the data storage module 3 and the log analysis module 4 are connected in sequence;

[0091] The system registration module 1 is used to register the smart water system;

[0092] The transmission module 2 is used to encrypt and compress the registration code, the packaged transmission code, the configuration file required by the code, and the execution file, and send them to the deployment address of the smart water system;

[0093] The data storage module 3 is used to store the log data pulled from the smart water system into the local memory of the log collection and analysis system;

[0094] The log analysis module 4 is used to analyze log data and present key information in the log data in an intuitive manner.

[0095] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. A weight-based log collection and analysis method, characterized in that: The following steps are involved: S1. Use the system registration module to enter the registration information of the third-party application system to register the third-party application system and generate a registration code for the third-party application system; S2. Use the transmission module to encrypt and compress the registration code, the packaged transmission code, the configuration file required for the transmission code, and the execution file, send it to the third-party application system deployment address for decompression, modify the transmission code configuration file and the execution file, start the transmission code to pull the log data and encrypt and compress it, and obtain the pulled log data; S3 uses a data storage module to store log data pulled from a third-party application system to the local memory of the log collection and analysis system; S4. Use the log analysis module to call the log data pulled from the data storage module, perform comparative analysis on it, and display the obtained log data analysis results.

2. According to the weight-based log collection and analysis method of claim 1, it is characterized in that: In S2, the transmission code is used to pull the log data of the deployment machine where the third-party application system is located; After the registration code, the packaged transmission code, the configuration file required by the transmission code, and the execution file are sent to the deployment machine where the third-party application system is located, i.e., the deployment address of the third-party application system, the registration code in the configuration file is modified to the registration code of the third-party application system, and the database location in the execution file is modified to the actual link database of the third-party application system; After the transmission code is started, it is determined whether the registration code of the third-party application system in the configuration file is correct. If it is correct, the log data pulling program is run to pull log data according to the custom pulling frequency. Otherwise, the log data pulling program is terminated and automatically exited. When the registration code of the third-party application system is verified, the above registration code is used as the random number seed of the encryption algorithm key to generate an encryption key, and the pulled log data is encrypted using the encryption method of the encryption algorithm.

3. According to the weight-based log collection and analysis method of claim 2, it is characterized in that: In S2 and S4, the comparative analysis includes frequency comparative analysis and quantity comparative analysis. For the frequency comparative analysis, the frequency of occurrence of the event is compared to calculate the frequency of occurrence of the event P = {p1, p2, ..., p N }, where N is the number of items involved in the comparison, P N is the frequency of the Nth entry, P∈{times / second, times / minute, times / hour, etc.}, and the events are sorted according to the frequency P of the event; For quantitative comparison analysis, the total number of occurrences of the comparison event S = {s1, s2, …, s N }, sort the events according to the total number of times S that the events occur; According to the general comparison formula, automatic adjustment weights are introduced for comparative analysis; The general comparison formula is expressed as: in, is the final comparison value, α is the weight value, and α={α1,α2,…,α N },0<α N <1, C is the initial comparison value, which is the frequency P of the event or the total number of times the event occurs S; When performing frequency comparison analysis, C = P, Among them, α N is the weight value of the Nth entry. When comparing by number of times, C = S. where α N Represents the weight value of the Nth entry; Automatic adjustment of weights is expressed as follows: according to the ranking without weight, i.e., the frequency of event occurrence P and the total number of event occurrence S, the weight rate value is determined. When the ranking of an item is improved for several consecutive days, and the number of consecutive change days reaches the cumulative number of days threshold, the weight value of the item is increased, and the upper limit of the increase is 1, that is, the importance of the item is increased. When the ranking of an item drops for several consecutive days, the weight value of the item is reduced, and the lower limit is 0. If a reverse change occurs during the continuous change process, the cumulative number of days is reset to zero and counted again. Among them, the cumulative number of days threshold and the weight rate value are preset values. According to the final comparison value Get the log data analysis results sorted {E1, E2, ..., E N }, where E N is an entry ranked N, and the final comparison value corresponding to each entry is in, is the final comparison value corresponding to the entry ranked N, 4. A weight-based log collection and analysis system, characterized in that: A weight-based log collection and analysis method for implementing any one of claims 1 to 3, comprising a system registration module (1), a transmission module (2), a data storage module (3) and a log analysis module (4); The system registration module (1), the transmission module (2), the data storage module (3) and the log analysis module (4) are connected in sequence; The system registration module (1) is used to register the third-party application system; The transmission module (2) is used to encrypt and compress the registration code, the packaged transmission code, the configuration file required by the code, and the execution file, and send them to the deployment address of the third-party application system; The data storage module (3) is used to store the log data pulled from the third-party application system into the local storage of the log collection and analysis system; The log analysis module (4) is used to analyze log data and present key information in the log data.

Images