A cloud server monitoring system

Abstract

The present invention relates to the field of server monitoring, and in particular to a cloud server monitoring system. The present invention comprises a data acquisition module, an explicit feature analysis module, a storage module, a log extraction module and an abnormality analysis module, which acquire log records and load records of corresponding running programs executed by each server node, and calculate the appearance difference characterization parameters by fitting with a preset formula in combination with the load change of the corresponding running program executed by the server node, and mark the appearance difference category of each log record according to the calculated appearance difference characterization parameters, so as to determine the extraction method of each log record according to the marking result, specifically extract the corresponding fragments, and process the extracted fragments, and process some log records in a targeted manner, so as to effectively save computing resources and further improve the processing efficiency of the log records.

Description

Technical Field

[0001] The present invention relates to the field of server monitoring, and in particular to a cloud server monitoring system. Background Art

[0002] In the daily maintenance process of servers, once a problem occurs, maintenance personnel usually query logs to locate the problem. In today's Internet era with huge amounts of data, data interaction usually needs to be completed in a server cluster environment. In a cluster environment, each server will store logs in its own memory. Maintenance personnel need to check logs one by one. If there are dozens or even hundreds of servers in the cluster, it is conceivable that the efficiency is extremely low.

[0003] Chinese patent publication number: CN112882913A, discloses a log recording method, including: determining a second event to be monitored that is associated with a first event to be monitored; if the first event to be monitored is detected to occur, generating a first log message; wherein the first log message is used to record relevant information of the first event to be monitored; if the first log message has been generated and the second event to be monitored is detected to occur, generating a second log message; wherein the second log message is used to record relevant information of the second event to be monitored;

[0004] However, the following problems still exist in the prior art. In the prior art, when a large number of cloud servers are monitored, the data processing volume is high when processing the corresponding log records of the cloud servers, which occupies computing power and affects the processing efficiency of the log records. Summary of the invention

[0005] To this end, the present invention provides a cloud server monitoring system to overcome the problem in the prior art that when monitoring a large number of cloud servers, the data processing volume is high when processing the corresponding log records of the cloud servers, which occupies computing power and affects the processing efficiency of the log records.

[0006] To achieve the above object, the present invention provides a cloud server monitoring system.

[0007] A data collection module is used to collect log records and load records of each server node executing the corresponding running program;

[0008] An explicit feature analysis module, which is connected to the data acquisition module, is used to determine the difference in the appearance features between the log record corresponding to the server node and the historical log record, and calculate the appearance difference characterization parameter in combination with the load of the corresponding running program executed by the server node to classify the appearance difference category of the log record;

[0009] A storage module, which is connected to the dominant feature analysis module, and is used to store the log records of each server node and mark the appearance difference category of each log record;

[0010] The log extraction module is connected to the data acquisition module and the storage module respectively, and includes a classification extraction unit and a supplementary extraction unit. The classification extraction unit is used to extract log records based on the appearance difference category, including:

[0011] Construct a load change time domain curve according to the load record, identify a specific time domain segment based on the difference between each time domain curve segment and the corresponding average peak value of the adjacent time domain curve segment, extract all log records in the specific time domain segment, determine the extraction ratio to extract the log record fragments in the specific time domain segment; or, extract all log records;

[0012] The supplementary extraction unit is used to determine whether to extract all log records stored in each cycle based on the quantity ratio of the representation difference categories marked by the storage module in each cycle.

[0013] Furthermore, the explicit feature analysis module determines that the surface features used to determine the log records corresponding to the server nodes include:

[0014] To determine the average length of a single line of code for logging;

[0015] The number of lines of code to determine the number of lines of code to log.

[0016] Furthermore, the dominant feature analysis module calculates the image difference characterization parameter according to formula (1):

[0017]

[0018] In formula (1), B is the representation difference characterization parameter, Le is the average number of code lines in historical log records, Lc is the number of code lines in the current log record, Me is the average value of the average code length of historical log records, Mc is the average code length of the current log record, Ie is the average value of the load change amplitude corresponding to the historical log records, Ic is the load change amplitude of the current log record, α is the preset representation execution difference coefficient, β is the preset representation byte difference coefficient, and γ is the preset load feature difference coefficient.

[0019] Furthermore, the dominant feature analysis module divides the log record into appearance difference categories based on the appearance difference characterization parameters, including:

[0020] If the appearance difference characterization parameter is less than or equal to the preset appearance difference characterization parameter, controlling the storage module to mark the log record as a weak appearance difference category;

[0021] If the appearance difference characterization parameter is greater than the preset appearance difference characterization parameter, the storage module is controlled to mark the log record as a strong appearance difference category.

[0022] Furthermore, the log extraction module extracts log records based on the appearance difference category, including:

[0023] If the log record is of the weak appearance difference category, identify the specific time domain segment based on the difference amount, extract all log records in the specific time domain segment, and extract non-specific log record segments belonging to the segment at a predetermined ratio;

[0024] If the log record is of the strong appearance difference category, all log records of a single server node are extracted.

[0025] Further, the supplementary extraction unit determines whether to extract all log records stored in the period including:

[0026] Determine the ratio of the number of strong appearance difference categories and the number of weak appearance difference categories marked by the storage module in each cycle of a single server node;

[0027] If the difference category ratio is less than or equal to the preset ratio threshold, the extraction range of the log records for a single server is not updated;

[0028] If the difference category ratio is greater than a preset ratio threshold, all log records stored in the single server node during the period are extracted.

[0029] Furthermore, the log extraction module identifies the specific time domain segment including:

[0030] If the difference between the average peak value of the wave peaks corresponding to a time domain curve segment and an adjacent time domain curve segment is greater than a preset peak difference threshold, the time domain segment corresponding to the time domain curve segment is marked as a specific time domain segment.

[0031] Furthermore, the classification and extraction unit determines the extraction ratio of the log record segments within the specific time domain segment based on the image difference representation parameter.

[0032] The extraction ratio determined by the classification extraction unit is proportional to the image difference characterization parameter

[0033] Furthermore, it also includes an abnormality analysis module, which is connected to the log extraction module to determine whether the extracted log records and / or log record fragments have abnormalities.

[0034] Furthermore, the exception analysis module is also used to mark the abnormal log records and / or abnormal codes in the log record segments where abnormalities exist.

[0035] Compared with the prior art, the present invention acquires the log records and load records of the corresponding running programs executed by each server node by setting a data acquisition module, determines the difference in appearance features between the corresponding log records of the server nodes and the historical log records by setting an explicit feature analysis module, and calculates the appearance difference characterization parameters by fitting with a preset formula in combination with the change in the load of the server node executing the corresponding running program, and marks the appearance difference category of each log record according to the calculated appearance difference characterization parameters, determines the extraction method of each log record according to the marking result, extracts the corresponding fragments specifically, processes the extracted fragments, and processes some log records in a targeted manner, thereby effectively saving computing resources and further improving the processing efficiency of log records.

[0036] Furthermore, the image difference characterization parameters are calculated to determine the difference between the number of lines of code in the current log record and the average number of lines of code in the historical log records. In actual situations, each server node is usually configured with a specific operating level. During stable operation, its corresponding log records may be repeated and relatively close in explicit features. Therefore, the present invention considers determining the image features from an explicit perspective. Under stable operation conditions, the image features of the log records corresponding to a single server node will not differ significantly, and similarly, its load will not fluctuate significantly. Therefore, the present invention calculates the image difference characterization parameters to characterize the tendency of the log records to be abnormal, and the extraction of the image features is relatively concise from the data level and does not occupy computing power. Furthermore, the image difference characterization parameters are used to divide the image difference categories of the log records, providing data support for the subsequent log extraction module to adaptively extract the log records required for analysis. Furthermore, while ensuring reliability, the amount of data calculation is reduced, and the log records corresponding to a large number of server nodes are analyzed to improve the analysis efficiency of the log records.

[0037] Furthermore, the appearance difference categories of each log record are divided based on the appearance difference characterization parameters, and the specific extraction method for the log record is further determined according to the completed classification category. If it is a weak appearance difference category, only the log record fragments within the specific time domain segment are extracted and analyzed. If it is a strong appearance difference category, all log records of a single server node are extracted and analyzed, and the log records with abnormal risks are specifically determined, and the extracted log records are processed in a targeted manner. While effectively improving the stability of the cloud server operation, the analysis efficiency of the log records is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A module block diagram of a cloud server monitoring system according to an embodiment of the invention;

[0039] Figure 2A logical decision diagram for classifying the appearance difference categories of log records for an embodiment of the invention;

[0040] Figure 3 A logical decision diagram for extracting log records based on appearance difference categories according to an embodiment of the invention;

[0041] Figure 4 An embodiment of the present invention is a logical decision diagram for extracting log records within a single server based on difference category ratios. DETAILED DESCRIPTION

[0042] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0044] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0045] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0046] See also Figure 1 , Figure 2 , Figure 3 as well as Figure 4 As shown, they are respectively a module block diagram of a cloud server monitoring system according to an embodiment of the present invention, a logic decision diagram for classifying the appearance difference categories of log records, a logic decision diagram for extracting log records based on the appearance difference categories, and a logic decision diagram for extracting log records in a single server based on the difference category ratio; a cloud server monitoring system according to an embodiment of the present invention comprises:

[0047] A data collection module is used to collect log records and load records of each server node executing the corresponding running program;

[0048] An explicit feature analysis module, which is connected to the data acquisition module, is used to determine the difference in the appearance features between the log record corresponding to the server node and the historical log record, and calculate the appearance difference characterization parameters in combination with the load of the corresponding running program executed by the server node to classify the appearance difference categories of the log record; the appearance features include the average length of a single line of code in the log record and the number of lines of code;

[0049] A storage module, which is connected to the dominant feature analysis module, and is used to store the log records of each server node and mark the appearance difference category of each log record;

[0050] The log extraction module is connected to the data acquisition module and the storage module respectively, and includes a classification extraction unit and a supplementary extraction unit. The classification extraction unit is used to extract log records based on the appearance difference category, including:

[0051] Construct a load change time domain curve according to the load record, identify a specific time domain segment based on the difference between each time domain curve segment and the corresponding average peak value of the adjacent time domain curve segment, extract all log records in the specific time domain segment, determine the extraction ratio to extract the log record fragments in the specific time domain segment; or, extract all log records;

[0052] The supplementary extraction unit is used to determine whether to extract all log records stored in each cycle based on the quantity ratio of the representation difference categories marked by the storage module in each cycle.

[0053] By setting up a data acquisition module, the log records and load records of the corresponding running programs executed by each server node are obtained. By setting up an explicit feature analysis module, the difference in the appearance features between the corresponding log records of the server nodes and the historical log records is determined. In combination with the change in the load of the server node executing the corresponding running program, the appearance difference characterization parameters are calculated by fitting with a preset formula, and each log record is marked with an appearance difference category according to the calculated appearance difference characterization parameters. The extraction method of each log record is determined according to the marking result, so as to specifically extract the corresponding fragments, and process the extracted fragments. Some log records are processed in a targeted manner, which effectively saves computing resources while further improving the processing efficiency of log records.

[0054] Specifically, the log records of the server nodes refer to the log records automatically generated by the server nodes during operation. These log records record in detail the running status of each running program executed by the server nodes. Log records are an important part of server operation and maintenance, and their forms are various. This is the existing technology and will not be repeated here.

[0055] Specifically, the load record is the change in the load of the server node when executing each running program. This is a prior art and will not be described in detail.

[0056] Specifically, the present invention does not limit the specific structures of the data acquisition module, the dominant feature analysis module, the storage module, and the log extraction module, which can all be composed of logical components. The logical components can be field programmable components, computers, and microprocessors in the computers, etc., which will not be repeated here.

[0057] Specifically, the dominant feature analysis module calculates the image difference representation parameter according to formula (1):

[0058]

[0059] In formula (1), B is the representation difference characterization parameter, Le is the average number of code lines in historical log records, Lc is the number of code lines in the current log record, Me is the average value of the average code length of historical log records, Mc is the average code length of the current log record, Ie is the average value of the load change amplitude corresponding to the historical log records, Ic is the load change amplitude of the current log record, α is the preset representation execution difference coefficient, β is the preset representation byte difference coefficient, and γ is the preset load feature difference coefficient.

[0060] The image difference characterization parameters are calculated to determine the difference between the number of code lines in the current log record and the average number of code lines in the historical log record. In actual situations, each server node is usually configured with a specific operating level. During stable operation, its corresponding log records may be repeated and relatively close in explicit features. Therefore, the present invention considers determining the image features from an explicit perspective. Under stable operation conditions, the image features of the log records corresponding to a single server node will not differ significantly, and similarly, its load will not fluctuate significantly. Therefore, the present invention calculates the image difference characterization parameters to characterize the tendency of the log records to be abnormal. In addition, the extraction of image features is relatively concise from the data level and does not occupy computing power. Furthermore, the image difference characterization parameters are used to divide the image difference categories of the log records, providing data support for the subsequent log extraction module to adaptively extract the log records required for analysis. Furthermore, under the premise of ensuring reliability, the amount of data calculation is reduced, and the log records corresponding to a large number of server nodes are analyzed to improve the analysis efficiency of the log records.

[0061] Specifically, α is the preset representation execution difference coefficient of 0.35, β is the preset representation byte difference coefficient of 0.35, and γ is the preset load characteristic difference coefficient of 0.3.

[0062] Specifically, the selected range of historical log records can be freely set by those skilled in the art, and it is only necessary to achieve the purpose of identifying each cyclic running program when a single server node executes each corresponding cyclic running program. This is a prior art and will not be elaborated on.

[0063] Specifically, the dominant feature analysis module divides the log record into appearance difference categories based on the appearance difference characterization parameters, including:

[0064] If the appearance difference characterization parameter is less than or equal to the preset appearance difference characterization parameter, controlling the storage module to mark the log record as a weak appearance difference category;

[0065] If the appearance difference characterization parameter is greater than the preset appearance difference characterization parameter, the storage module is controlled to mark the log record as a strong appearance difference category.

[0066] The preset image difference characterization parameter B0 is selected in the interval [0.15, 0.3].

[0067] Specifically, the log extraction module extracts log records based on the appearance difference category, including:

[0068] If the log record is of the weak appearance difference category, identify the specific time domain segment based on the difference amount, extract all log records in the specific time domain segment, and extract the log record fragments in the non-specific time domain segment at a predetermined ratio;

[0069] If the log record is of the strong appearance difference category, all log records of a single server node are extracted.

[0070] Based on the appearance difference characterization parameters, the appearance difference categories of each log record are divided, and the specific extraction method for the log record is further determined according to the completed category. If it is a weak appearance difference category, only the log record fragments within the specific time domain segment are extracted and analyzed. If it is a strong appearance difference category, all log records of a single server node are extracted and analyzed. The log records with abnormal risks are specifically determined, and the extracted log records are processed in a targeted manner. While effectively improving the stability of the cloud server operation, the analysis efficiency of the log records is further improved.

[0071] Specifically, the supplementary extraction unit determines whether to extract all log records stored in the period, including:

[0072] Determine the ratio of the number of strong appearance difference categories and the number of weak appearance difference categories marked by the storage module in each cycle of a single server node;

[0073] If the difference category ratio is less than or equal to the preset ratio threshold, the extraction range of the log records for a single server is not updated;

[0074] If the difference category ratio is greater than a preset ratio threshold, all log records stored in the single server node during the period are extracted.

[0075] Specifically, the log extraction module identifies a specific time domain segment including:

[0076] If the difference between the average peak value of the wave peaks corresponding to a time domain curve segment and an adjacent time domain curve segment is greater than a preset peak difference threshold, the time domain segment corresponding to the time domain curve segment is marked as a specific time domain segment.

[0077] The preset peak difference threshold is selected within the interval [0.11Vp, 0.15Vp], where Vp is the average peak value of the peaks of each historical time domain curve segment of a single server node.

[0078] Specifically, the classification and extraction unit determines the extraction ratio of the log record segments within the specific time domain segment based on the image difference representation parameter.

[0079] The extraction ratio determined by the classification extraction unit is proportional to the image difference representation parameter.

[0080] In this embodiment, optionally,

[0081] The image difference characterization parameter B is compared with the first preset image comparison threshold B1 and the second preset image comparison threshold B2,

[0082] If B≤B1, the extraction ratio is determined as the first extraction ratio T1, and T1=1.11T0;

[0083] If B1<B≤B2, the extraction ratio is determined as the second extraction ratio T2, and T1=1.17T0;

[0084] If B>B2, the extraction ratio is determined as the third extraction ratio T3, and T3=1.22T0;

[0085] Wherein, T0 is the initial extraction ratio, 20%<T0<40%, B1=0.75B0, B2=0.85B0.

[0086] Furthermore, it also includes an abnormality analysis module, which is connected to the log extraction module to determine whether the extracted log records and / or log record fragments have abnormalities.

[0087] Furthermore, the exception analysis module is also used to mark the abnormal log records and / or abnormal codes in the log record segments where abnormalities exist.

[0088] Specifically, the present invention does not limit the specific structure of the abnormality analysis module, which can be composed of logic components. The logic components can be field programmable components, computers, and microprocessors in the computers, etc., which will not be repeated here.

[0089] Specifically, the exception analysis module can be any program that can determine the abnormal conditions of extracted log records and / or log record fragments, such as Log Tank Service (LTS), Simple Log Service (SLS), etc., or it can be any algorithm that can determine the abnormal conditions of log records and / or log record fragments. This is the prior art and will not be repeated here.

[0090] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A cloud server monitoring system, characterized in that: include: A data collection module is used to collect log records and load records of each server node executing the corresponding running program; An explicit feature analysis module, which is connected to the data acquisition module, is used to determine the difference in the appearance features between the log record corresponding to the server node and the historical log record, and calculate the appearance difference characterization parameter in combination with the load of the corresponding running program executed by the server node to classify the appearance difference category of the log record; A storage module, which is connected to the dominant feature analysis module, and is used to store the log records of each server node and mark the appearance difference category of each log record; The log extraction module is connected to the data acquisition module and the storage module respectively, and includes a classification extraction unit and a supplementary extraction unit. The classification extraction unit is used to extract log records based on the appearance difference category, including: Construct a load change time domain curve according to the load record, identify a specific time domain segment based on the difference between each time domain curve segment and the corresponding average peak value of the adjacent time domain curve segment, extract all log records in the specific time domain segment, determine the extraction ratio to extract the log record fragments in the specific time domain segment; or, extract all log records; The supplementary extraction unit is used to determine whether to extract all log records stored in each cycle based on the quantity ratio of the appearance difference categories marked by the storage module in each cycle; The dominant feature analysis module calculates the image difference characterization parameter according to formula (1), In formula (1), B is the representation difference characterization parameter, Le is the average number of code lines in historical log records, Lc is the number of code lines in the current log record, Me is the average value of the average code length of historical log records, Mc is the average code length of the current log record, Ie is the average value of the load change amplitude corresponding to the historical log records, Ic is the load change amplitude of the current log record, α is the preset representation execution difference coefficient, β is the preset representation byte difference coefficient, and γ is the preset load feature difference coefficient.

2. The cloud server monitoring system according to claim 1, characterized in that: The explicit feature analysis module determines that the surface features used to determine the log records corresponding to the server node include: To determine the average length of a single line of code for logging; The number of lines of code to determine the number of lines of code to log.

3. The cloud server monitoring system according to claim 2, characterized in that: The dominant feature analysis module divides the log records into appearance difference categories based on the appearance difference characterization parameters, including: If the appearance difference characterization parameter is less than or equal to the preset appearance difference characterization parameter, controlling the storage module to mark the log record as a weak appearance difference category; If the appearance difference characterization parameter is greater than the preset appearance difference characterization parameter, the storage module is controlled to mark the log record as a strong appearance difference category.

4. The cloud server monitoring system according to claim 3, characterized in that: The log extraction module extracts log records based on the appearance difference category, including: If the log record is of the weak appearance difference category, identify the specific time domain segment based on the difference amount, extract all log records in the specific time domain segment, and extract the log record fragments in the non-specific time domain segment at a predetermined ratio; If the log record is of the strong appearance difference category, all log records of a single server node are extracted.

5. The cloud server monitoring system according to claim 1, characterized in that: The supplementary extraction unit determines whether to extract all log records stored in the period, including: Determine the ratio of the number of strong appearance difference categories and the number of weak appearance difference categories marked by the storage module in each cycle of a single server node; If the difference category ratio is less than or equal to the preset ratio threshold, the extraction range of the log records for a single server is not updated; If the difference category ratio is greater than a preset ratio threshold, all log records stored in the single server node during the period are extracted.

6. The cloud server monitoring system according to claim 1, characterized in that: The log extraction module identifies a specific time domain segment including: If the difference between the average peak value of the wave peaks corresponding to a time domain curve segment and an adjacent time domain curve segment is greater than a preset peak difference threshold, the time domain segment corresponding to the time domain curve segment is marked as a specific time domain segment.

7. The cloud server monitoring system according to claim 1, characterized in that: The classification and extraction unit determines the extraction ratio of the log record segments within the specific time domain segment based on the image difference representation parameter, wherein: The extraction ratio determined by the classification extraction unit is proportional to the image difference representation parameter.

8. The cloud server monitoring system according to claim 1, characterized in that: It also includes an abnormality analysis module, which is connected to the log extraction module and is used to determine whether the extracted log records and / or log record fragments have abnormalities.

9. The cloud server monitoring system according to claim 8, characterized in that: The exception analysis module is also used to mark the log records with exceptions and / or the exception codes in the log record segments.

Images