Distributed application monitoring method and monitoring system based on data analysis

Abstract

The present invention relates to the technical field of data anomaly monitoring, and specifically to a distributed application monitoring method and monitoring system based on data analysis. First, the historical call count data segment of the server for each service type is obtained, and the change in the call count is analyzed, and the optimal change data segment is divided to reveal the change pattern of the call count. Based on the temporal distance between the current moment and the historical moment in the optimal change data segment, the matching change data segment and the moment difference value are determined as an indicator to characterize whether the current moment is abnormal. Further, based on the degree of deviation between the number of calls at the current moment and the number of calls in the matching change data segment, the number anomaly factor is obtained as another anomaly indicator. Finally, based on these two indicators, the abnormal situation of each server in the distributed application is monitored. This method can distinguish between the call surge of malicious requests and the normal call surge, thereby improving the monitoring credibility.

Description

Technical Field

[0001] The present invention relates to the technical field of data anomaly monitoring, and in particular to a distributed application monitoring method and a monitoring system based on data analysis. Background Art

[0002] Distributed application refers to an application that is distributed on different computing nodes and works through the network to achieve a common goal. As the main component of computing nodes, the server plays a key role in processing requests and providing services. In distributed application monitoring, tracing data anomalies refer to abnormal behaviors or data detected in the distributed tracing system. These anomalies can reveal performance bottlenecks, errors or other potential problems in the system, which is conducive to improving the reliability and performance of the system.

[0003] Since malicious attacks usually occupy resources by sending a large number of requests to the server at the same time, causing the service to be unavailable, existing methods usually use the number of calls to the service type by each server as the basis. When the number of calls exceeds the set threshold, the server status is considered abnormal. However, in actual applications, normal request data may surge in certain periods of time, and malicious attacks are usually disguised as normal requests to bypass defense mechanisms. Therefore, by simply setting a threshold for the number of calls, it is impossible to effectively distinguish malicious attacks from normal surges in request data, which will lead to low credibility in monitoring distributed applications and easily lead to misjudgment. Summary of the invention

[0004] In order to solve the problem that normal request data may surge in certain periods of time, and malicious attacks are usually disguised as normal requests to bypass defense mechanisms, it is impossible to effectively distinguish malicious attacks from normal surges in request data by simply setting a threshold for the number of calls, which will lead to low credibility in monitoring distributed applications and easy misjudgment. The purpose of the present invention is to provide a distributed application monitoring method and monitoring system based on data analysis, and the technical solutions adopted are as follows:

[0005] For each server in the distributed application, obtain a plurality of unit data segments of the number of historical calls to each service type;

[0006] Under each service type, each unit data segment is divided based on the change of the number of calls in the unit data segment at historical moments, so as to obtain the optimal change data segment; based on the time series distance between the current moment and the historical moment of the optimal change data segment in each unit data segment, the matching change data segment corresponding to the current moment and the time difference value between the current moment and the matching change data segment are determined;

[0007] In all unit data segments, the call count at the current moment is adjusted using the fluctuation of the call count at the historical moment, and the difference between the adjusted value and the call count at the historical moment in the matching change data segment is analyzed to determine the abnormality factor of the number at the current moment;

[0008] Based on the current time difference value and the number of abnormal factors under all service types of each server, the abnormal situation of each server in the distributed application is monitored.

[0009] Furthermore, the method for obtaining the optimal change data segment includes:

[0010] Under each service type, analyze the changes in the number of calls within a unit data segment at each historical moment, and determine the growth change coefficient corresponding to each historical moment;

[0011] In each unit data segment, the optimal change data segment is obtained by dividing each unit data segment according to the growth change coefficient and the number of calls corresponding to the historical moment.

[0012] Furthermore, the method for obtaining the growth variation coefficient includes:

[0013] In all unit data segments, based on the fluctuation of the number of calls at the same historical moment, determine the floating factor of the number of calls corresponding to the same historical moment;

[0014] In each unit data segment, a timing curve corresponding to all call times is obtained, and on the timing curve, a slope value at each historical moment is obtained;

[0015] Based on the call count floating factor and slope value corresponding to each historical moment, the growth variation coefficient corresponding to each historical moment is determined, and the call count floating factor is negatively correlated with the growth variation coefficient, and the slope value is positively correlated with the growth variation coefficient.

[0016] Furthermore, the method for obtaining the call count floating factor includes:

[0017] In all unit data segments, all historical moments at the same position are sorted according to the time sequence of the unit data segments to obtain a sorted sequence;

[0018] The sorted sequence is forward differentiated to obtain a first-order difference sequence, and the absolute value of the cumulative sum of all values ​​in the first-order difference sequence is used as the calling number floating factor.

[0019] Furthermore, in each unit data segment, according to the growth change coefficient and the number of calls corresponding to the historical moment, each unit data segment is divided to obtain the optimal change data segment, including:

[0020] Different numbers of changed data segments are preset. In each unit data segment, for any number of changed data segments, a preset number of historical moments are randomly selected as breakpoints, and the preset number is an integer multiple of the number of changed data segments;

[0021] Two breakpoints adjacent in time series are regarded as a breakpoint combination, and there is no intersection between all breakpoint combinations. The two breakpoints in each breakpoint combination and the number of calls at the historical moments between the two breakpoints are combined into an initial change data segment, and all initial change data segments are obtained;

[0022] In each initial change data segment, the priority factor of each initial change data segment is determined based on the numerical characteristics of the growth change coefficients of the two breakpoints and the average level of the number of calls at all historical moments between the two breakpoints;

[0023] The normalized value of the average of the priority factors of all initial change data segments under the number of change data segments is used as the priority value of the number of change data segments; in each unit data segment, the number of change data segments corresponding to the maximum priority value is used as the optimal number; the initial change data segment of each unit data segment under the optimal number is used as the optimal change data segment.

[0024] Furthermore, the method for obtaining the priority factor includes:

[0025] The average of the growth change coefficients of the two breakpoints is taken as the first priority coefficient; the average of the number of calls at all times between the two breakpoints is taken as the second priority coefficient;

[0026] The product of the first priority coefficient and the second priority coefficient is used as the priority factor of each change data segment.

[0027] Further, the determining of the matching change data segment corresponding to the current moment and the time difference value between the current moment and the matching change data segment based on the time sequence distance between the current moment and the historical moment of the optimal change data segment in each unit data segment includes:

[0028] The historical moment at the same position as the current moment in each unit data segment is used as the comparison moment;

[0029] In each unit data segment, if the comparison moment exists in a certain optimal change data segment, the moment difference value corresponding to the current moment is 0, and the optimal change data segment is the matching change data segment corresponding to the current moment;

[0030] If the comparison time does not exist in any optimal change data segment, obtain the median of the historical moments in each optimal change data segment, and use the absolute value of the difference between the median and the comparison time as the time distance between the current moment and each optimal change data segment. Among all optimal change data segments, use the minimum time distance as the time difference value corresponding to the current moment, and use the optimal change data segment corresponding to the minimum time distance as the matching change data segment corresponding to the current moment.

[0031] Furthermore, the method for obtaining the number of abnormal factors includes:

[0032] In all unit data segments, the variance of the number of calls at the current moment and the number of calls at the same historical moment is normalized as an adjustment factor, and the number of calls at the current moment is adjusted based on the adjustment factor to obtain an adjustment value, and the adjustment value is positively correlated with the number of calls, and negatively correlated with the adjustment factor;

[0033] The average of the maximum number of calls at historical moments in all matching change data segments is used as the floating upper limit value. If the adjustment value at the current moment is less than or equal to the floating upper limit value, the number abnormality factor at the current moment is 0; if the adjustment value at the current moment is greater than the floating upper limit value, the difference between the adjustment value at the current moment and the floating upper limit value is used as the number abnormality factor at the current moment.

[0034] Furthermore, the abnormal situation of each server in the distributed application is monitored based on the time difference value at the current time and the number of abnormal factors under all service types of each server, including:

[0035] For any server, under each service type of the server, based on the moment difference value and the number of abnormal factors at the current moment, determine the abnormal coefficient at the current moment, and the moment difference value and the number of abnormal factors are both positively correlated with the abnormal coefficient;

[0036] The normalized sum of the abnormal coefficients of the server at the current moment under all service types is used as the abnormality degree value of the server at the current moment;

[0037] When the abnormality level value is greater than a preset abnormality threshold, the server is judged to be in an abnormal state and an early warning is issued.

[0038] A distributed application monitoring system based on data analysis includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and when the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor, any one of the steps of a distributed application monitoring method based on data analysis is implemented.

[0039] The present invention has the following beneficial effects:

[0040] Since it is not possible to effectively distinguish malicious attacks from normal data surges by only setting a threshold for the number of calls, but malicious attacks are indifferent in time and have no specific rules, while normal request data will have certain changing rules in each unit time, so in the present invention, firstly, multiple unit data segments of the historical number of calls of each server for each service type in the distributed application are obtained, and in the subsequent process, all unit data segments can be analyzed. Because data surges and malicious attacks will lead to an increase in the number of calls of the server to the service type, under each service type, the changes in the number of calls in the unit data segment at historical moments are analyzed to divide each unit data, so as to obtain the optimal change data segment, which characterizes the fluctuation of the number of calls in the unit data segment and can reveal the changing rules of the number of calls in the historical process. Then, based on the temporal distance between the current moment and the historical moment in the optimal change data segment, the matching change data segment at the current moment and the moment difference value at the current moment can be determined. The matching change data segment is the data segment that is closest to the current moment in time in the historical process, and the moment difference value is the deviation degree of the current moment from the data segment that is closest in time in the temporal sequence, which can be used as an indicator to characterize whether the current moment is abnormal. Further, the number of calls at the current moment is adjusted by the fluctuation of the number of calls at the historical moment in the unit data segment to eliminate the error caused by the fluctuation of the data itself. Based on the difference between the adjusted value and the number of calls at the historical moment in the matching change data segment, the deviation degree of the number of calls between the current moment and the data segment that is closest in time can be quantified to obtain the number of abnormal factors, which is also used as an indicator to characterize whether the current moment is abnormal. Finally, the abnormal situation of each server in the distributed application is monitored based on the two indicators that characterize whether the current moment is abnormal. In summary, the present invention can effectively distinguish the surge of malicious requests from normal request data based on the difference in the temporal change rules between malicious attacks and normal request data, reduce misjudgment, and thus improve the monitoring credibility of distributed applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1A method flow chart of a distributed application monitoring method based on data analysis provided by an embodiment of the present invention;

[0043] Figure 2 A method flow chart of a method for obtaining an optimal change data segment provided by an embodiment of the present invention;

[0044] Figure 3 A method flow chart of a method for obtaining a growth variation coefficient provided by an embodiment of the present invention;

[0045] Figure 4 A system block diagram of a distributed application monitoring system based on data analysis provided by an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the system structure of a distributed application monitoring system based on data analysis provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a distributed application monitoring method and monitoring system based on data analysis proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0048] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0049] The following is a detailed description of a distributed application monitoring method and a monitoring system based on data analysis provided by the present invention in conjunction with the accompanying drawings.

[0050] See also Figure 1 , which shows a method flow chart of a distributed application monitoring method based on data analysis provided by an embodiment of the present invention, the method comprising the following steps:

[0051] Step S1: For each server in the distributed application, multiple unit data segments of the historical call counts for each service type are obtained.

[0052] In distributed applications, applications distributed on different servers or different processes on the same server work together through the network to complete the same task. The services provided usually provide a unified access portal to the outside world through load balancing. In the field of distributed application monitoring, tracking data anomalies is a key link in ensuring system stability and performance.

[0053] Malicious attacks, such as DDOS attacks, usually disguise themselves as normal requests, and occupy system resources by sending a large number of disguised requests to the server at the same time, thus causing service unavailability. Therefore, existing monitoring methods mainly rely on setting a fixed call count threshold to determine whether the server status is abnormal. However, in actual applications, the number of normal requests may surge in certain periods of time, and the number of these disguised requests may be similar to that of normal requests in this case. Therefore, it is difficult to effectively distinguish malicious attacks from normal request data surges by relying solely on the call count threshold.

[0054] However, malicious attacks are indiscriminate and have no specific rules in terms of attack time, so in an embodiment of the present invention, the abnormal situation of each server in the distributed application is monitored by comparing the matching situation between the number of calls of the server at the current moment and the number of calls in the historical process.

[0055] First, in the distributed application, it is necessary to obtain the historical number of calls of each server to each service type. The number of calls can be obtained by viewing the call log of each server. Specifically, after collecting the call logs on all servers, the call logs are parsed according to the same rules. The time interval can be set to 1 minute, that is, the time recorded in the call log is classified according to 1 minute. After classification, the sum of the number of calls of each server to each service type in each 1 minute can be counted. For example, if a server calls a certain service type 5 times within 0 to 1 minute, the number of calls corresponding to this service type by the server at one minute is 5; at this time, the call status of each service type on each server can be counted, so that the time series data of the number of calls in the historical process can be obtained. Given that normal requests, that is, the number of calls to the service type, have a certain regularity, when obtaining the call status of each service type in the historical process, a unit data segment is set, that is, multiple unit data segments of the historical number of calls of each server to each service type are obtained, and each unit data segment is the time series data of the number of calls in the historical process within a unit time length.

[0056] It should be noted that, in the embodiment of the present invention, the length of the unit data segment can be in days (24 hours). When the number of unit data segments is 3, the current day can be used as the starting point to obtain historical call data within three days before the current day. Then, the current moment can be found in each unit data segment. The corresponding time, such as 13 o'clock corresponds to 13 o'clock; the current moment can also be used as the starting point to obtain historical call data within three days before the current moment. Then, the time corresponding to the current moment in each unit data segment is the time at the first position; the length and number of unit data segments can be adjusted according to the implementation scenario and are not limited here.

[0057] Step S2: Under each service type, each unit data segment is divided based on the change in the number of calls within the unit data segment at historical moments, so as to obtain the optimal change data segment; based on the temporal distance between the current moment and the historical moment of the optimal change data segment in each unit data segment, the matching change data segment corresponding to the current moment and the moment difference value between the current moment and the matching change data segment are determined.

[0058] Since the surge in the number of requests and calls of normal service types has certain regularity, for example, during the period from 9 am to 5 pm, users will be more active, resulting in a surge in the number of requests and calls of service types; therefore, the embodiment of the present invention determines whether the number of calls at the current moment is abnormal by comparing the matching degree between the call situation at the current moment and the historical call process. Therefore, in the unit data segment under each service type, each unit data segment can be first divided based on the change of the number of calls at the historical moment to obtain the optimal change data segment, which characterizes the change regularity of the number of calls under normal circumstances in the historical call process; then, based on the time series distance between the current moment and the historical moment in the optimal change data segment, the current moment is matched with the optimal change data segment, so as to determine the matching change data segment corresponding to the current moment and the time difference value between the current moment and the matching change data segment, the matching change data segment is the data segment in each unit data segment in the historical process that is closest to the current moment in time, and the time difference value is the degree of deviation in time between the current moment and the data segment that is closest in time, which can be used as an indicator to characterize whether the current moment is abnormal in the subsequent process.

[0059] Based on the above analysis, it can be seen that the normal surge in call data has a certain regularity, so the position of the corresponding data segment with the change in the number of calls should be relatively fixed in the historical process. Under each service type, each unit data segment can be divided by analyzing the change in the number of calls in the unit data segment at historical moments to obtain the optimal change data segment. The optimal change data segment represents the data segment in which the number of calls in each unit data segment has changed significantly.

[0060] Preferably, in one embodiment of the present invention, the method for obtaining the optimal change data segment includes:

[0061] See also Figure 2 , which shows a method flow chart of a method for obtaining an optimal change data segment in one embodiment of the present invention, the method comprising the following steps:

[0062] Step S201: Under each service type, analyze the changes in the number of calls within a unit data segment at historical moments, and determine the growth change coefficient corresponding to each historical moment.

[0063] See also Figure 3 , which shows a method flow chart of a method for obtaining a growth variation coefficient in one embodiment of the present invention, the method comprising the following steps:

[0064] Step S2011: In all unit data segments, based on the fluctuation of the number of calls at the same historical moment, determine the floating factor of the number of calls corresponding to the same historical moment.

[0065] By conducting a vertical analysis of all unit data segments and analyzing the fluctuations in the number of calls at the same location at historical moments, we can more comprehensively reflect the changing patterns of the number of calls in the historical process.

[0066] Therefore, in all unit data segments, all historical moments at the same position are sorted according to the time sequence of the unit data segments to obtain a sorted sequence; for example, the unit data segments are recorded as unit data segment a, unit data segment b and unit data segment c according to the time sequence, and the length of each unit data segment is 5, then the first moment in unit data segment a, the first moment in unit data segment b and the first moment in unit data segment c are the moments at the same position, and so on, 5 sorted sequences can be obtained.

[0067] Since the fluctuation can be characterized by the values ​​in the first-order difference sequence, the values ​​in the sorted sequence are forward differentiated to obtain the first-order difference sequence of the sorted sequence. At this time, the values ​​in the first-order difference sequence are positive, indicating that the number of calls at the same historical moment in the vertical direction has increased. Conversely, the values ​​in the first-order difference sequence are negative, indicating that the number of calls at the same historical moment in the vertical direction has increased.

[0068] Then, the absolute value of the cumulative sum of all values ​​in the first-order difference sequence at each identical position is taken as the call count floating factor, denoted as DF. The closer the call count floating factor is to 0, that is, the smaller it is, it means that in the vertical direction as a whole, even in different unit data segments, the overall fluctuation of the call count at the same position is smaller, and the values ​​tend to be consistent. It can also be regarded as that the same position has a higher degree of vertical alignment.

[0069] So far, each historical moment in all unit data segments has a call count floating factor, and the call count floating factors of historical moments at the same position in different unit data segments have the same value.

[0070] Step S2012: In each unit data segment, the floating factor of the number of calls at each historical moment and the change trend of the number of calls at each historical moment in the corresponding unit data segment are comprehensively considered to determine the growth change coefficient corresponding to each historical moment.

[0071] When the number of server calls for a certain service request surges, the horizontal data will show a more obvious upward trend. Therefore, in each unit data segment, the timing curve corresponding to the number of calls can be obtained based on the timing data of the unit data segment, and the slope value at each historical moment can be obtained on the timing curve. When the slope value is positive and the larger it is, the data shows an increasing trend. Conversely, when the slope value is negative and the smaller it is, the data shows a decreasing trend. It should be noted that the slope value at the last historical moment can be set to be consistent with the slope value at the previous historical moment.

[0072] Finally, based on the call count floating factor and slope value corresponding to each historical moment, the growth change coefficient corresponding to each historical moment can be determined. The formula model of the growth change coefficient includes:

[0073]

[0074] in, Indicates the growth change coefficient at each historical moment in each unit data segment; Indicates the slope value at each historical moment in each unit data segment; Indicates the floating factor of the number of calls corresponding to each historical moment in each unit data segment; Indicates the preset first parameter; Represents the normalization function.

[0075] In the formula model of the growth variation coefficient, based on the analysis of the previous steps, it can be seen that when the slope value is a positive number and the larger it is, the data shows an increasing trend; at this time, when the call count floating factor is smaller, it means that in the vertical direction as a whole, even in different unit data segments, the overall fluctuation of the call count at the same position is small, and the values ​​tend to be consistent, which can also be regarded as the vertical alignment of the same position is higher, so the call count floating factor corresponding to the historical moment is placed in the denominator, the slope value of the historical moment is placed in the numerator, and the comparison value is normalized to obtain the growth variation coefficient. When the growth variation coefficient is larger, it means that on the basis of a higher degree of vertical alignment and smaller overall fluctuation, the growth trend of the call data at the historical moment is more obvious. In view of the fact that the slope value may be positive or negative, the method for normalizing the comparison value here adopts Normalization function.

[0076] It should be noted that the first parameter is preset The function of is to prevent the denominator from being 0. The value can be 0.001. The specific value can be adjusted according to the implementation scenario and is not limited here.

[0077] Step S202: In each unit data segment, based on the growth change coefficient and the number of calls corresponding to the historical moment, each unit data segment is divided to obtain the optimal change data segment.

[0078] In each unit data segment, the growth change coefficient corresponding to each historical moment can measure the growth trend of the data called at the historical moment. Therefore, the change data segment in each unit data segment can be extracted by combining the growth change coefficient corresponding to the historical moment and the number of calls. In view of the fact that the number of change data segments and the division method are unknown, in this embodiment of the present invention, by traversing the number of change data segments, an optimal division method that can more accurately characterize the data change law is screened out from a variety of possible division methods, thereby dividing each unit data segment to obtain the optimal change data segment.

[0079] First, different numbers of changing data segments are preset. In each unit data segment, for any number of changing data segments, a preset number of historical moments are randomly selected as breakpoints. Since two breakpoints can determine a data segment, the preset number is twice the number of the changing data segments. It should be noted that the value range of the number of changing data segments is set to [2,8]. The specific value range can be adjusted according to the implementation scenario, but the minimum value should be 2, which is not limited here; the random selection process of the breakpoint can be: based on the length of the unit data segment as a range, a preset number of random numbers are generated to determine the corresponding breakpoint.

[0080] Then, two breakpoints adjacent in time series can be regarded as a breakpoint combination, and there is no intersection between all breakpoint combinations. Then, the call counts of the two breakpoints and the historical moments between the two breakpoints in each breakpoint combination can form an initial change data segment, and thus all initial change data segments can be obtained. For example: if the number of historical moments in a unit data segment is 10, when the number of change data segments is 2, if the randomly selected breakpoints are historical moment 2, historical moment 5, historical moment 6, and historical moment 8, then historical moment 2 and historical moment 5 are a breakpoint combination, historical moment 6 and historical moment 8 are a breakpoint combination, and the call counts corresponding to historical moment 2 to historical moment 5 form an initial change data segment, and the call counts corresponding to historical moment 6 to historical moment 8 form an initial change data segment.

[0081] Next, since the data increase trend at the two endpoints of the data segment with a significant surge in the number of calls should be more obvious, and the average level of the number of calls between the two endpoints should also be higher; and the data increase trend can be represented by the growth change coefficient, in each initial change data segment, based on the numerical characteristics of the growth change coefficients of the two breakpoints and the average level of the number of calls at all historical moments between the two breakpoints, the priority factor of each initial change data segment is determined: the mean of the growth change coefficients of the two breakpoints is taken as the first priority coefficient. The larger the first priority coefficient is, the greater the data growth trend at the two endpoints of the initial change data segment is, and the more likely it is to be a data segment with a significant increase in the number of calls; and the mean of the number of calls at all moments between the two breakpoints is taken as the second priority coefficient. Similarly, the larger the second priority coefficient is, the higher the average level of all calls in the initial data segment is, which can be regarded as the more likely that the initial data segment is a data segment with a significant increase in the number of calls; the product of the first priority coefficient and the second priority coefficient is taken as the priority factor of each initial change data segment. Based on the above analysis, it can be seen that the larger the priority factor is, the more the initial change data segment can represent the special time period with a surge in the number of calls.

[0082] Finally, the average value of the priority factors of all initial change data segments under the number of change data segments is normalized as the priority value of the number of change data segments. The larger the priority value, the more the initial change data segments divided under the number of change data segments can represent the data change pattern in the unit time period, that is, the special time period that represents the surge in the number of calls can be obtained more accurately. It should be noted that the normalization method here can be used Normalization function.

[0083] At this point, based on the above method, the priority value corresponding to the number of each changing data segment in each unit data segment can be obtained. In each unit data segment, the number of changing data segments corresponding to the maximum priority value can be taken as the optimal number, and the initial changing data segment of each unit data segment under the optimal number can be taken as the optimal changing data segment.

[0084] In order to more accurately evaluate the abnormality of the number of calls at the current moment, the connection between the current moment and the optimal change data segment in multiple unit data segments in history can be compared. If the connection is close, it can be regarded as that the surge in the number of calls at the current moment may be a normal surge in request calls, otherwise it is considered that it is more likely to be an abnormality caused by a malicious attack. On the one hand, this connection can be characterized by the distance deviation in time series. Therefore, in this embodiment of the present invention, based on the time series distance between the current moment and the historical moment of the optimal change data segment in each unit data segment, the matching change data segment corresponding to the current moment and the time difference value between the current moment and the matching time period are determined.

[0085] Preferably, in one embodiment of the present invention, based on the temporal distance between the current moment and the historical moment of the best change data segment in each unit data segment, determining the matching change data segment corresponding to the current moment and the time difference value between the current moment and the matching change data segment includes:

[0086] Since normal data calls usually have certain regularity in multiple unit time periods, the historical moment at the same position as the current moment in each unit data segment is used as the comparison moment. Then, the analysis of the connection between the current moment and the optimal change data segment in multiple historical unit data segments can be transformed into the analysis of the connection between the comparison moment and the optimal change data segment in the unit data segment to which it belongs.

[0087] Then in each unit data segment, if the comparison moment exists in a certain optimal change data segment, it is considered to have the strongest connection, and the moment difference value corresponding to the current moment is 0, and the optimal change data segment is used as the matching change data segment corresponding to the current moment.

[0088] If the comparison time does not exist in any optimal change data segment, it is necessary to calculate the time deviation between the comparison time and each optimal change data segment: obtain the median of all historical moments in each optimal change data segment, the median can represent the time center point of each optimal change data segment; take the absolute value of the difference between the median and the comparison time as the time distance between the current moment and each optimal change data segment. At this time, the comparison time and each optimal change data segment have a time distance, and the larger the time distance, the greater the time deviation between the comparison time and the optimal change data segment, that is, the longer the time distance, the weaker the connection between the current moment and the optimal change data segment.

[0089] Finally, among all the optimal change data segments, the minimum time series distance is taken as the time difference value corresponding to the current moment, recorded as , the optimal change data segment corresponding to the minimum time series distance is taken as the matching change data segment. The matching change data segment represents the optimal change data segment with the closest time series distance to the current moment in each unit data segment. Then, the change or pattern of the number of calls in the matching change data segment can be used to measure the abnormality of the number of calls at the current moment.

[0090] Step S3: In all unit data segments, the fluctuation of the number of calls at historical moments is used to adjust the number of calls at the current moment, and the difference between the adjusted value and the number of calls at historical moments in the matching change data segment is analyzed to determine the abnormal number factor at the current moment.

[0091] In step S2, the optimal change data segment closest to the current time sequence can be determined in each unit data segment, that is, the matching change data segment is determined. When comparing the connectivity between the matching change data segments in the current moment and multiple historical unit data segments, on the other hand, the difference between the number of calls can be measured to determine the number of abnormal factors at the current moment, as another indicator for evaluating whether the number of calls at the current moment is abnormal. Before determining the number of abnormal factors, the number of calls at the current moment can be adjusted in all unit data segments using the fluctuation of the number of calls at historical moments to eliminate accidental errors and make subsequent results more accurate, and then the number of abnormal factors are calculated based on the adjusted value.

[0092] Preferably, in one embodiment of the present invention, the method for obtaining the number of abnormal factors includes:

[0093] In all unit data segments, the variance of the number of calls at the current moment and the historical moment at the same location is normalized as the adjustment factor, and the number of calls at the current moment is adjusted based on the adjustment factor to obtain an adjustment value. The formula model of the adjustment value includes:

[0094]

[0095] in, Indicates the adjustment value corresponding to the current moment; Indicates the number of calls at the current moment; Indicates the variance of the number of calls between the current moment and the historical moment at the same position in all unit data segments; Indicates the preset second parameter; Represents the normalization function.

[0096] In the formula model of the adjustment value, if the variance value of the number of calls at the current moment and the historical moment at the same position in all unit data segments is smaller, it means that the fluctuation between the number of calls at the current moment and the historical moment at the same position in all unit data segments is smaller, that is, more consistent, then the possibility of accidental error in the number of calls at the current moment is smaller, and the authenticity of the number of calls is higher, so the value after the variance is normalized is used as the adjustment factor , and put the adjustment factor in the denominator to get the adjustment value corresponding to the current moment.

[0097] It should be noted that the purpose of presetting the second parameter is to prevent the denominator from being 0, and the value may be 0.001. The specific value may be adjusted according to the implementation scenario and is not limited here.

[0098] At the current moment, each unit data segment corresponds to a matching change data segment, that is, the data segment closest in time series. Then, the average of the maximum number of calls in the historical moments in all matching change data segments is taken as the floating upper limit value. The floating upper limit value represents a relatively high, but still reasonable and normal number of calls. Therefore, if the adjustment value at the current moment is less than or equal to the floating upper limit value, it is considered that the call value at the current moment is within a reasonable range, and the number anomaly factor at the current moment is 0; if the adjustment value at the current moment is greater than the floating upper limit value, it is considered that the number of calls at the current moment exceeds the reasonable range calculated based on the historical call situation. The difference between the adjustment value at the current moment and the floating upper limit value is taken as the number anomaly factor at the current moment, denoted as CY, which is used to quantify the degree of abnormality of the number of calls at the current moment. The larger the number anomaly factor, the higher the degree to which the number of calls at the current moment exceeds the normal range, that is, the higher the degree of abnormality, the greater the possibility of malicious attacks.

[0099] At this point, another indicator for measuring the abnormality of the number of calls of each server at the current moment under each service type can be obtained—the number abnormality factor.

[0100] Step S4: Based on the time difference value at the current moment and the number of abnormal factors under all service types of each server, the abnormal situation of each server in the distributed application is monitored.

[0101] Based on the above steps, it can be known that the moment difference value and the number anomaly factor at the current moment can both represent the abnormal situation of the number of calls at the current moment. Therefore, here, the moment difference value and the number anomaly factor of each server at the current moment under all service types can be comprehensively considered to monitor the abnormal situation of each server in the distributed application.

[0102] Preferably, in one embodiment of the present invention, based on the time difference value at the current moment and the number of abnormal factors under all service types of each server, the abnormal situation of each server in the distributed application is monitored, including:

[0103] For any server, under each service type of the server, the abnormal coefficient of the current moment is determined based on the time difference value and the number of abnormal factors at the current moment, and the time difference value and the number of abnormal factors are positively correlated with the abnormal coefficient. The formula model of the abnormal coefficient includes:

[0104]

[0105] in, Indicates the abnormal coefficient of each service type on each server at the current moment; Indicates the time difference value at the current time under each service type of each server; Indicates the abnormal factor of the number of times at the current moment under each service type of each server.

[0106] In the formula model of the abnormality coefficient, if the moment difference value of the current moment is smaller and the abnormal number difference factor is smaller, it means that the distance between the current moment and the data segment indicating the normal change of the number of calls in the historical call situation is closer, and the number of calls is closer to the normal and reasonable range, then the abnormality degree is lower, so the abnormality coefficient is smaller; conversely, if the moment difference value of the current moment is larger and the abnormal number difference factor is larger, it means that the distance between the current moment and the data segment indicating the normal change of the number of calls in the historical call situation is farther, and the degree to which the number of calls exceeds the normal and reasonable range is greater, then the abnormality degree is higher, so the abnormality coefficient is larger.

[0107] Finally, the sum of the abnormality coefficients of the server under all service types at the current moment is normalized and used as the abnormality degree value of the server at the current moment; when the abnormality degree value is larger, it means that the number of calls to the service type by the server at the current moment has increased abnormally, resulting in a large amount of resources being occupied, and it is more likely to be attacked maliciously. Therefore, when the abnormality degree value is greater than the preset abnormality threshold, it is judged that the server is in an abnormal state and an early warning is required.

[0108] It should be noted that the normalization method here uses Normalization function, and the preset abnormal threshold is 0.7. The specific value can be adjusted according to the implementation scenario and is not limited here.

[0109] In summary, since it is impossible to effectively distinguish malicious attacks from normal data surges by only setting a threshold for the number of calls, but malicious attacks are indifferent in time and have no specific rules, and normal request data will have certain changing rules in each unit time, so in the present invention, firstly, multiple unit data segments of the historical number of calls of each server for each service type in the distributed application are obtained, and in the subsequent process, all unit data segments can be analyzed. Because data surges and malicious attacks will lead to an increase in the number of calls to the service type by the server, under each service type, the changes in the number of calls in the unit data segment at historical moments are analyzed to divide each unit data, so as to obtain the optimal change data segment, which characterizes the fluctuation of the number of calls in the unit data segment and can reveal the changing rules of the number of calls in the historical process. Then, based on the temporal distance between the current moment and the historical moment in the optimal change data segment, the matching change data segment at the current moment and the moment difference value at the current moment can be determined. The matching change data segment is the data segment that is closest to the current moment in time in the historical process, and the moment difference value is the deviation degree of the current moment from the data segment that is closest in time in the temporal sequence, which can be used as an indicator to characterize whether the current moment is abnormal. Further, the number of calls at the current moment is adjusted by the fluctuation of the number of calls at the historical moment in the unit data segment to eliminate the error caused by the fluctuation of the data itself. Based on the difference between the adjusted value and the number of calls at the historical moment in the matching change data segment, the deviation degree of the number of calls between the current moment and the data segment that is closest in time can be quantified to obtain the number of abnormal factors, which is also used as an indicator to characterize whether the current moment is abnormal. Finally, the abnormal situation of each server in the distributed application is monitored based on the two indicators that characterize whether the current moment is abnormal. In summary, the present invention can effectively distinguish the surge of malicious requests from normal request data based on the difference in the temporal change rules between malicious attacks and normal request data, reduce misjudgment, and thus improve the monitoring credibility of distributed applications.

[0110] The embodiment of the present invention also proposes a distributed application monitoring system based on data analysis, please refer to Figure 4 As shown in the system block diagram, the system includes: a data acquisition module 401, used to implement step S1 in the above method; a first abnormal factor analysis module 402, used to implement step S2 in the above method; a second abnormal factor analysis module 403, used to implement step S3 in the above method; an abnormal monitoring module 404, used to implement step S4 in the above method.

[0111] It should be noted that: the system provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the distributed application monitoring system based on data analysis provided in the above embodiment and the distributed application monitoring method embodiment based on data analysis belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0112] See also Figure 5 , which shows a system structure diagram of a distributed application monitoring system based on data analysis provided by an embodiment of the present invention, including a processor 500, a memory 501, a bus 502 and a communication interface 503, wherein the processor 500, the communication interface 503 and the memory 501 are connected via the bus 502; wherein the memory 501 may include a high-speed random access memory, the bus 502 may be an ISA bus, a PCI bus or an EISA bus, etc., and the processor 500 may be an integrated circuit chip with signal processing capabilities; the memory 501 stores at least one instruction, at least one program, a code set or an instruction set, and when the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor, the steps in a distributed application monitoring method based on data analysis are implemented.

[0113] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0114] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A distributed application monitoring method based on data analysis, characterized in that: The method comprises: For each server in the distributed application, obtain a plurality of unit data segments of the number of historical calls to each service type; Under each service type, each unit data segment is divided based on the change of the number of calls in the unit data segment at historical moments, so as to obtain the optimal change data segment; based on the time series distance between the current moment and the historical moment of the optimal change data segment in each unit data segment, the matching change data segment corresponding to the current moment and the time difference value between the current moment and the matching change data segment are determined; In all unit data segments, the call count at the current moment is adjusted using the fluctuation of the call count at the historical moment, and the difference between the adjusted value and the call count at the historical moment in the matching change data segment is analyzed to determine the abnormality factor of the number at the current moment; Based on the current time difference value and the number of abnormal factors under all service types of each server, the abnormal situation of each server in the distributed application is monitored; The method for obtaining the optimal change data segment includes: Under each service type, analyze the changes in the number of calls within a unit data segment at historical moments and determine the growth change coefficient corresponding to each historical moment; In each unit data segment, according to the growth change coefficient and the number of calls corresponding to the historical moment, each unit data segment is divided to obtain the optimal change data segment; The method for obtaining the growth variation coefficient includes: In all unit data segments, based on the fluctuation of the number of calls at the same historical moment, determine the floating factor of the number of calls corresponding to the same historical moment; In each unit data segment, a timing curve corresponding to all call times is obtained, and on the timing curve, a slope value at each historical moment is obtained; Based on the call count floating factor and slope value corresponding to each historical moment, determine the growth variation coefficient corresponding to each historical moment, wherein the call count floating factor is negatively correlated with the growth variation coefficient, and the slope value is positively correlated with the growth variation coefficient; The method for obtaining the call count floating factor includes: In all unit data segments, all historical moments at the same position are sorted according to the time sequence of the unit data segments to obtain a sorted sequence; The sorted sequence is forward differentiated to obtain a first-order difference sequence, and the absolute value of the cumulative sum of all values ​​in the first-order difference sequence is used as the calling number floating factor.

2. A distributed application monitoring method based on data analysis according to claim 1, characterized in that: In each unit data segment, according to the growth change coefficient and the number of calls corresponding to the historical moment, each unit data segment is divided to obtain the optimal change data segment, including: Different numbers of changed data segments are preset. In each unit data segment, for any number of changed data segments, a preset number of historical moments are randomly selected as breakpoints, and the preset number is an integer multiple of the number of changed data segments; Two breakpoints adjacent in time series are regarded as a breakpoint combination, and there is no intersection between all breakpoint combinations. The two breakpoints in each breakpoint combination and the number of calls at the historical moments between the two breakpoints are combined into an initial change data segment, and all initial change data segments are obtained; In each initial change data segment, the priority factor of each initial change data segment is determined based on the numerical characteristics of the growth change coefficients of the two breakpoints and the average level of the number of calls at all historical moments between the two breakpoints; The normalized value of the average of the priority factors of all initial change data segments under the number of change data segments is used as the priority value of the number of change data segments; in each unit data segment, the number of change data segments corresponding to the maximum priority value is used as the optimal number; the initial change data segment of each unit data segment under the optimal number is used as the optimal change data segment.

3. A distributed application monitoring method based on data analysis according to claim 2, characterized in that: The method for obtaining the priority factor includes: The average of the growth change coefficients of the two breakpoints is taken as the first priority coefficient; the average of the number of calls at all times between the two breakpoints is taken as the second priority coefficient; The product of the first priority coefficient and the second priority coefficient is used as the priority factor of each change data segment.

4. A distributed application monitoring method based on data analysis according to claim 1, characterized in that: The determining, based on the time series distance between the current moment and the historical moment of the optimal change data segment in each unit data segment, of the matching change data segment corresponding to the current moment and the time difference value between the current moment and the matching change data segment comprises: The historical moment at the same position as the current moment in each unit data segment is used as the comparison moment; In each unit data segment, if the comparison moment exists in a certain optimal change data segment, the moment difference value corresponding to the current moment is 0, and the optimal change data segment is the matching change data segment corresponding to the current moment; If the comparison time does not exist in any optimal change data segment, obtain the median of the historical moments in each optimal change data segment, and use the absolute value of the difference between the median and the comparison time as the time distance between the current moment and each optimal change data segment. Among all optimal change data segments, use the minimum time distance as the time difference value corresponding to the current moment, and use the optimal change data segment corresponding to the minimum time distance as the matching change data segment corresponding to the current moment.

5. The distributed application monitoring method based on data analysis according to claim 1, characterized in that: The method for obtaining the number of abnormal factors includes: In all unit data segments, the variance of the number of calls at the current moment and the number of calls at the same historical moment is normalized as an adjustment factor, and the number of calls at the current moment is adjusted based on the adjustment factor to obtain an adjustment value, and the adjustment value is positively correlated with the number of calls, and negatively correlated with the adjustment factor; The average of the maximum number of calls at historical moments in all matching change data segments is used as the floating upper limit value. If the adjustment value at the current moment is less than or equal to the floating upper limit value, the number abnormality factor at the current moment is 0; if the adjustment value at the current moment is greater than the floating upper limit value, the difference between the adjustment value at the current moment and the floating upper limit value is used as the number abnormality factor at the current moment.

6. A distributed application monitoring method based on data analysis according to claim 1, characterized in that: The monitoring of abnormal conditions of each server in the distributed application based on the current time difference value and the number of abnormal factors under all service types of each server includes: For any server, under each service type of the server, based on the moment difference value and the number of abnormal factors at the current moment, determine the abnormal coefficient at the current moment, and the moment difference value and the number of abnormal factors are both positively correlated with the abnormal coefficient; The normalized sum of the abnormal coefficients of the server at the current moment under all service types is used as the abnormality degree value of the server at the current moment; When the abnormality level value is greater than a preset abnormality threshold, the server is judged to be in an abnormal state and an early warning is issued.

7. A distributed application monitoring system based on data analysis, characterized in that: It includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and when the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor, the steps of a distributed application monitoring method based on data analysis as described in any one of claims 1 to 6 are implemented.

Images