Network anomaly identification method and system
Through the target flow processing framework and timing prediction model, the network interface access log is processed and predicted, and the exception threshold is automatically set, which solves the problem of poor accuracy and timeliness of network exception recognition in the existing technology, and realizes efficient identification of crawler behavior and Early warning.
Patent Information
- Application Number
- CN202410223289.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-28
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-02-28
AI Technical Summary
The prior art has set the abnormal threshold of network interface access through manual experience, resulting in poor accuracy and timeliness of network abnormal identification, which has failed to effectively solve the identification problem of crawlers in confrontation scenarios.
The target stream processing framework and timing prediction model are used to process and predict the network interface access log, and set automated anomaly thresholds based on the predicted access volume and actual access volume, and identify the abnormal access risk of the network interface through the prediction and analysis of timing data.
It improves the accuracy and timelyness of network abnormal identification, realizes accurate prediction and timely warning of abnormal access to server interfaces, and improves the efficiency of network abnormal warning.
Smart Images

Figure CN117997627B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of network security technology, and in particular to a network anomaly identification method and system. Background Art
[0002] In the scenario of fighting against crawlers, the perception of abnormal fluctuations in traffic time series is an important means of judging crawlers. As the confrontation gradually escalates, crawlers become more and more cunning, and they will try their best to hide their tracks by combining methods such as reducing the number of calls, dispersing calls, and calling only during peak hours.
[0003] In the prior art, the time series of the collected access volume of a certain network interface is usually smoothed first, and then the abnormal alarm threshold is set based on the statistical characteristics such as the mean and variance of the smoothed data and manual experience, so as to subsequently identify network anomalies based on the abnormal alarm threshold.
[0004] From the above analysis, it can be seen that there is no effective solution to the problem that the above-mentioned prior art sets the abnormal threshold of network interface access volume through manual experience, resulting in poor accuracy and timeliness in identifying network anomalies. Summary of the invention
[0005] The embodiment of the present invention provides a network anomaly identification method and system, so as to at least solve the technical problem that the prior art sets the network interface access volume anomaly threshold through manual experience, resulting in poor accuracy and timeliness in network anomaly identification.
[0006] According to one aspect of an embodiment of the present invention, a network anomaly identification method is provided, comprising:
[0007] Obtain a network interface access log of a target server; use a target stream processing framework to process text data of the network interface access log to obtain a processing result, wherein the processing result includes at least timing data of the target network interface, and the timing data is used to determine the actual access volume of the target network interface at any access time; use a target timing prediction model to predict the timing data of a first time period to obtain a prediction result, wherein the prediction result is used to determine the predicted access volume of the target network interface in a second time period, and the second time period is a time period whose timing is after the first time period and adjacent to the first time period; based on the predicted access volume and the actual access volume of the first time period, perform abnormal identification on the target network interface to obtain an identification result, wherein the identification result is used to determine whether the target network interface has an abnormal access risk in the second time period.
[0008] Optionally, the target stream processing framework is used to process the text data of the network interface access log, and the processing results obtained include: performing data cleaning on the text data to obtain cleaned text data; performing data parsing on the cleaned text data using a data stream processing function to obtain a parsing result, wherein the parsing result is used to determine the network interface name and timestamp information of each access log; and aggregating and counting the parsing results based on a preset time window to obtain a processing result.
[0009] Optionally, the above-mentioned network anomaly identification method also includes: constructing an initial timing prediction model based on a timing prediction algorithm; obtaining historical data of the target network interface in at least two time periods, wherein the historical data is used to determine the historical access volume of at least two time periods; performing data preprocessing and data segmentation on the historical data to obtain a training set and a test set; in the process of training the initial timing prediction model using the training set, repeatedly iterating and adjusting the model parameters to obtain a timing prediction model to be evaluated; evaluating the timing prediction model to be evaluated using the test set to obtain an evaluation result, wherein the evaluation result is used to determine the model parameters to be adjusted; adjusting the model parameters to be adjusted of the timing prediction model to be evaluated based on the evaluation result to obtain a target timing prediction model.
[0010] Optionally, the prediction result includes a first prediction result and a second prediction result, the first prediction result includes the predicted visits at a target time point, the target time point being two time endpoints of a second time period, the second prediction result includes the predicted visits at other time points, the other time points being time points other than the target time point in the second time period, and the target time series prediction model is used to predict the time series data of the first time period, and obtaining the prediction result includes: predicting the visits at the target time point using the target time series prediction model to obtain a first prediction result; determining the predicted visits at other time points using the predicted visits of at least one time endpoint in the first prediction result to obtain a second prediction result.
[0011] Optionally, using the predicted visits of at least one time endpoint in the first prediction result to determine the predicted visits of other time points includes: determining a time endpoint close to any other time point as a target time endpoint; determining the predicted visits of the target time endpoint as the predicted visits of any other time point.
[0012] Optionally, using the predicted visits of at least one time endpoint in the first prediction result to determine the predicted visits of other time points includes: determining the time distance between any other time point and any time endpoint; determining the weight of the predicted visits of any time endpoint based on the time distance; and performing weighted sum calculation on the predicted visits of the two time endpoints and the corresponding weights to obtain the predicted visits of any other time point.
[0013] Optionally, based on the predicted access volume and the actual access volume in the first time period, the target network interface is identified for abnormalities, and the identification result includes: calculating the difference between the predicted access volume and the actual access volume corresponding to any time point to obtain a residual sequence; calculating the standard deviation of the residual sequence; comparing the standard deviation with the abnormal access threshold to obtain a comparison result; in response to the comparison result, determining that the standard deviation is greater than the abnormal access threshold, determining that the identification result is that the target network interface has an abnormal access risk in the second time period.
[0014] Optionally, the abnormal access threshold is determined by a standard deviation of a residual sequence corresponding to the first time period.
[0015] Optionally, the above-mentioned network anomaly identification method also includes: in response to the target network interface having an abnormal access risk in the second time period, triggering and executing network anomaly protection measures.
[0016] According to another aspect of an embodiment of the present invention, a network anomaly identification device is provided, including:
[0017] An acquisition module is used to acquire a network interface access log of a target server; a processing module is used to process text data of the network interface access log using a target stream processing framework to obtain a processing result, wherein the processing result includes at least timing data of the target network interface, and the timing data is used to determine the actual access volume of the target network interface at any access time; a prediction module is used to predict the timing data of a first time period using a target timing prediction model to obtain a prediction result, wherein the prediction result is used to determine the predicted access volume of the target network interface in a second time period, and the second time period is a time period whose timing is after the first time period and adjacent to the first time period; an identification module is used to identify anomalies of the target network interface based on the predicted access volume and the actual access volume of the first time period to obtain an identification result, wherein the identification result is used to determine whether the target network interface has an abnormal access risk in the second time period.
[0018] Optionally, the above-mentioned processing module includes: performing data cleaning on the text data to obtain cleaned text data; performing data parsing on the cleaned text data using a data stream processing function to obtain a parsing result, wherein the parsing result is used to determine the network interface name and timestamp information of each access log; and aggregating and counting the parsing results based on a preset time window to obtain a processing result.
[0019] Optionally, the above-mentioned network anomaly identification device also includes: a training module, which is used to construct an initial time series prediction model based on a time series prediction algorithm; obtain historical data of the target network interface in at least two time periods, wherein the historical data is used to determine the historical access volume of at least two time periods; perform data preprocessing and data segmentation on the historical data to obtain a training set and a test set; in the process of training the initial time series prediction model using the training set, repeatedly iterate and adjust the model parameters to obtain a time series prediction model to be evaluated; use the test set to evaluate the time series prediction model to obtain an evaluation result, wherein the evaluation result is used to determine the model parameters to be adjusted; based on the evaluation result, adjust the model parameters to be adjusted of the time series prediction model to be evaluated to obtain a target time series prediction model.
[0020] Optionally, the prediction results include a first prediction result and a second prediction result, the first prediction result includes the predicted visits at a target time point, the target time point being two time endpoints of a second time period, the second prediction result includes the predicted visits at other time points, the other time points being time points other than the target time point within the second time period, the above-mentioned prediction module includes: using a target time series prediction model to predict the visits at the target time point to obtain a first prediction result; using the predicted visits at at least one time endpoint in the first prediction result to determine the predicted visits at other time points to obtain a second prediction result.
[0021] Optionally, the prediction module includes: determining a time endpoint close to any other time point as a target time endpoint; and determining the predicted visit volume of the target time endpoint as the predicted visit volume of any other time point.
[0022] Optionally, the above-mentioned prediction module includes: determining the time distance between any other time point and any time endpoint; determining the weight of the predicted visit volume of any time endpoint based on the time distance; performing weighted sum calculation on the predicted visit volume of the two time endpoints and the corresponding weights to obtain the predicted visit volume of any other time point.
[0023] Optionally, the above-mentioned identification module includes: calculating the difference between the predicted access volume and the actual access volume corresponding to any time point to obtain a residual sequence; calculating the standard deviation of the residual sequence; comparing the standard deviation with the abnormal access threshold to obtain a comparison result; in response to the comparison result, determining that the standard deviation is greater than the abnormal access threshold, determining that the identification result is that the target network interface has an abnormal access risk in the second time period.
[0024] Optionally, in the above-mentioned network anomaly identification device: the abnormal access threshold is determined by the standard deviation of the residual sequence corresponding to the first time period.
[0025] Optionally, the above-mentioned network anomaly identification device further includes: an execution module, which is used to trigger and execute network anomaly protection measures in response to the existence of abnormal access risk to the target network interface in the second time period.
[0026] According to another aspect of an embodiment of the present invention, a network anomaly identification system is provided, including a memory and a processor, wherein an executable computer program is stored in the memory, and the processor is configured to run the executable computer program to execute any one of the aforementioned network anomaly identification methods.
[0027] In an embodiment of the present invention, a network interface access log of a target server is first obtained, and then the text data of the network interface access log is processed using a target stream processing framework to obtain a processing result, wherein the processing result includes at least timing data of the target network interface, and the timing data is used to determine the actual access volume of the target network interface at any access time, and then the timing data of the first time period is predicted using a target timing prediction model to obtain a prediction result, wherein the prediction result is used to determine the predicted access volume of the target network interface in a second time period, and the second time period is a time period whose timing is after the first time period and adjacent to the first time period, and finally, based on the predicted access volume and the actual access volume of the first time period, the target network interface is identified for abnormalities to obtain an identification result, wherein the identification result is used to determine whether there is an abnormal access risk to the target network interface in the second time period.
[0028] The above-mentioned method provided by the present invention uses the trained target time series prediction model to predict the time series data of the first time period to obtain the network interface access volume of the second time period, and sets the abnormal access threshold based on the actual access volume and the predicted access volume of the first time period to identify network anomalies, thereby achieving the purpose of accurately and timely identifying abnormal access to the server interface, thereby achieving the technical effect of improving the accuracy of network anomaly prediction results and enhancing the timeliness of network anomaly identification and early warning, and further solving the technical problem that the prior art sets the abnormal threshold of network interface access volume through artificial experience, resulting in poor accuracy and timeliness of network anomaly identification. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0030] Figure 1 is a hardware structure block diagram of an optional mobile terminal for a network anomaly identification method according to an embodiment of the present invention;
[0031] Figure 2is a flow chart of a network anomaly identification method according to an embodiment of the present invention;
[0032] Figure 3 is a structural block diagram of an optional network anomaly identification device according to an embodiment of the present invention;
[0033] Figure 4 is a structural block diagram of another optional network anomaly identification device according to an embodiment of the present invention;
[0034] Figure 5 It is a structural block diagram of another optional network anomaly identification device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0037] According to an embodiment of the present invention, a method embodiment of a network anomaly identification method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0038] Figure 1 is a hardware structure block diagram of a mobile terminal for a network anomaly identification method according to an embodiment of the present invention, such as Figure 1As shown, the mobile terminal 10 (or mobile device 10) may include one or more processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, MCU) or a programmable logic device (Field Programmable Gate Array, FPGA)), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display device 110, an input / output device 108 (i.e., an I / O device), a Universal Serial Bus (USB) port (which may be included as one of the ports of a computer bus, not shown in the figure), a network interface (not shown in the figure), a power supply (not shown in the figure) and / or a camera (not shown in the figure). It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal 10. For example, the mobile terminal 10 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0039] It should be noted that the one or more processors 102 and / or other data processing circuits may be embodied in whole or in part as software, hardware, firmware or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be fully or partially integrated into any of the other components in the mobile terminal 10 (or mobile device 10).
[0040] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the network anomaly identification method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned network anomaly identification method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0041] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0042] Under the above operating environment, the embodiment of the present invention provides the following Figure 2 The network anomaly identification method shown, Figure 2 is a flow chart of a network anomaly identification method according to an embodiment of the present invention. Figure 2 As shown, the method includes the following implementation steps:
[0043] Step S201, obtaining the network interface access log of the target server;
[0044] Step S202, using the target stream processing framework to process the text data of the network interface access log to obtain a processing result, wherein the processing result at least includes the time series data of the target network interface, and the time series data is used to determine the actual access volume of the target network interface at any access time;
[0045] Step S203, using the target time series prediction model to predict the time series data of the first time period to obtain a prediction result, wherein the prediction result is used to determine the predicted access volume of the target network interface in the second time period, and the second time period is a time period whose time series is after the first time period and adjacent to the first time period;
[0046] Step S204, based on the predicted access volume and the actual access volume in the first time period, an abnormality identification is performed on the target network interface to obtain an identification result, wherein the identification result is used to determine whether there is an abnormal access risk on the target network interface in the second time period.
[0047] The information or data recorded in the network interface access log of the above-mentioned target server may include, but is not limited to: the client's Internet Protocol address, access time, access request method, uniform resource locator of the access request, response time, error message, amount of data transmitted, and server response content.
[0048] The target stream processing framework may be an open source distributed processing engine Flink, which can extract the required information (the time series data) from the input data stream. The target time series prediction model may be a trained model based on a time series prediction algorithm, and the time series prediction algorithm used by the model may be but is not limited to at least one of the following: Prophet algorithm, algorithm based on Long Short-Term Memory (LSTM), Autoregressive Integrated Moving Average (ARIMA) algorithm.
[0049] It should be noted that when performing anomaly identification, the network interface access volume anomaly threshold can be set according to the actual access volume in the first time period, so that the anomaly threshold can be updated in real time in each time period, thereby ensuring the timeliness of the anomaly threshold and the timeliness of anomaly identification.
[0050] In an embodiment of the present invention, a network interface access log of a target server is first obtained, and then the text data of the network interface access log is processed using a target stream processing framework to obtain a processing result, wherein the processing result includes at least timing data of the target network interface, and the timing data is used to determine the actual access volume of the target network interface at any access time, and then the timing data of the first time period is predicted using a target timing prediction model to obtain a prediction result, wherein the prediction result is used to determine the predicted access volume of the target network interface in a second time period, and the second time period is a time period whose timing is after the first time period and adjacent to the first time period, and finally, based on the predicted access volume and the actual access volume of the first time period, the target network interface is identified for abnormalities to obtain an identification result, wherein the identification result is used to determine whether there is an abnormal access risk to the target network interface in the second time period.
[0051] The above-mentioned method provided by the present invention uses the trained target time series prediction model to predict the time series data of the first time period to obtain the network interface access volume of the second time period, and sets the abnormal access threshold based on the actual access volume and the predicted access volume of the first time period to identify network anomalies, thereby achieving the purpose of accurately and timely identifying abnormal access to the server interface, thereby achieving the technical effect of improving the accuracy of network anomaly prediction results and enhancing the timeliness of network anomaly identification and early warning, and further solving the technical problem that the prior art sets the abnormal threshold of network interface access volume through artificial experience, resulting in poor accuracy and timeliness of network anomaly identification.
[0052] The above method of the embodiment of the present invention is further introduced below.
[0053] In an optional embodiment, in step S202, the text data of the network interface access log is processed using the target stream processing framework, and the processing results include:
[0054] Step S2021, performing data cleaning on the text data to obtain cleaned text data;
[0055] Step S2022, using a data stream processing function to perform data parsing on the cleaned text data to obtain a parsing result, wherein the parsing result is used to determine the network interface name and timestamp information of each access log;
[0056] Step S2023, aggregate statistics on the analysis results based on a preset time window to obtain processing results.
[0057] In the technical solution provided by the present invention, the text data of the network interface access log of the target server is obtained as a data source and input into the processing engine Flink, and the processing engine cleans the input data source. Then, the data flow processing function (for example: Map function, FlatMap function) in the processing engine can be used to extract the network interface name and timestamp information in each access log. Furthermore, the parsed data can be grouped according to a preset time window (for example: one minute), and the number of visits to each network interface in each time window is counted, and the number of visits and time information are aggregated according to the network interface name to obtain the time series data of each network interface.
[0058] It should be noted that the time window size of the processing engine can be adjusted according to demand. For example, when it is necessary to reduce the model input to reduce the amount of model calculation, the time window can be adjusted from one minute to ten minutes.
[0059] In an optional embodiment, the above network anomaly identification method further includes:
[0060] Step S2051, constructing an initial time series prediction model based on a time series prediction algorithm;
[0061] Step S2052, obtaining historical data of the target network interface in at least two time periods, wherein the historical data is used to determine the historical access volume in at least two time periods;
[0062] Step S2053, performing data preprocessing and data segmentation on the historical data to obtain a training set and a test set;
[0063] Step S2054, in the process of training the initial time series prediction model using the training set, repeatedly iterate and adjust the model parameters to obtain the time series prediction model to be evaluated;
[0064] Step S2055, using the test set to evaluate the time series prediction model to be evaluated, and obtaining an evaluation result, wherein the evaluation result is used to determine the model parameters to be adjusted;
[0065] Step S2056, adjusting the model parameters to be adjusted of the time series prediction model to be evaluated based on the evaluation result to obtain the target time series prediction model.
[0066] In the technical solution provided by the present invention, in order to make the model accurately fit the periodic characteristics of the data, at least two time periods of historical data are selected to train the initial time series prediction model. For example, the access volume of a certain network interface shows periodicity within a week. When training the model, at least two consecutive weeks of network interface access logs need to be selected.
[0067] During the training process of the initial time series prediction model, by continuously adjusting the model parameters and using the evaluation results to evaluate the prediction accuracy of the adjusted time series prediction model, a set of accurate model parameter target parameter values can be obtained, so that the model parameters of the model are set to the corresponding target parameter values to obtain the target time series prediction model.
[0068] In an optional embodiment, in step S203, the prediction result includes a first prediction result and a second prediction result, the first prediction result includes the predicted visit volume at a target time point, the target time point is two time endpoints of the second time period, the second prediction result includes the predicted visit volume at other time points, the other time points are time points other than the target time point in the second time period, and the target time series prediction model is used to predict the time series data of the first time period, and the prediction results obtained include:
[0069] Step S2031, predicting the number of visits at a target time point using a target time series prediction model to obtain a first prediction result;
[0070] Step S2032: Determine the predicted visits at other time points using the predicted visits at at least one time endpoint in the first prediction result to obtain a second prediction result.
[0071] As an optional implementation, assuming that the time period is ten minutes, the time window is one minute, and the first time period is from 1:00 to 1:10, the above-mentioned target time points (two time endpoints) are 1:00 and 1:10, and other time points include: 1:01, 1:02, ..., 1:09. The actual access volume of the target network interface from 0:50 to 1:00 is input into the target timing prediction model, and the predicted access volume at 1:00 and 1:10 can be obtained.
[0072] It should be noted that in order to ensure the confidence of the target time series prediction model, the time period and time window can be adjusted according to the confidence requirements. For example, when the confidence requirements are high, the time period and time window can be shortened (such as: shortening the time period from one hour to ten minutes, and shortening the time window from ten minutes to one minute), so that the target time series prediction model can accurately predict the number of visits at time points corresponding to shorter time intervals in the future.
[0073] In an optional embodiment, in step S2032, using the predicted page views of at least one time endpoint in the first prediction result to determine the predicted page views at other time points includes:
[0074] Step S20321, determining a time endpoint close to any other time point as a target time endpoint;
[0075] Step S20322: determine the predicted number of visits at the target time endpoint as the predicted number of visits at any other time point.
[0076] Still in the above optional implementation, in order to quickly determine the predicted visits at any other time point, the time endpoint closest to the any other time point can be determined first. For example, when determining the predicted visits at one point and three minutes, first determine that the time endpoint closest to it is one point. Therefore, the predicted visits at one point are determined as the predicted visits at one point and three minutes.
[0077] In an optional embodiment, in step S2032, using the predicted page views of at least one time endpoint in the first prediction result to determine the predicted page views at other time points includes:
[0078] Step S20323, determining the time distance between any other time point and any time endpoint;
[0079] Step S20324, determining the weight of the predicted visit volume at any time endpoint based on the time distance;
[0080] Step S20325, performing weighted sum calculation on the predicted visits at the two time endpoints and the corresponding weights to obtain the predicted visits at any other time point.
[0081] Still in the above optional implementation, in order to improve the accuracy of the predicted visits at other time points within the second time period, a method of weighted summation of the predicted visits at two time endpoints can be adopted. For example: when determining the predicted visits at 1:30, the time distance from 1:30 to the two time endpoints is determined to be three minutes and seven minutes, and a weight of 0.7 is assigned to the predicted visits at 1:00 and a weight of 0.3 is assigned to the predicted visits at 1:10, and then a weighted sum is performed to obtain the predicted visits at 1:30.
[0082] In an optional embodiment, in step S204, based on the predicted access volume and the actual access volume in the first time period, the target network interface is identified for abnormality, and the identification result includes:
[0083] Step S2041, calculating the difference between the predicted visit volume and the actual visit volume corresponding to any time point to obtain a residual sequence;
[0084] Step S2042, calculating the standard deviation of the residual sequence;
[0085] Step S2043, comparing the standard deviation with the abnormal access threshold to obtain a comparison result;
[0086] Step S2044: in response to the comparison result determining that the standard deviation is greater than the abnormal access threshold, determining that the identification result is that the target network interface has an abnormal access risk in the second time period.
[0087] The above abnormal access threshold can be determined by the standard deviation of the residual sequence corresponding to the first time period. For example, based on the 3sigma principle in probability statistics, the abnormal access threshold is set to the product of the standard deviation of the residual sequence corresponding to the first time period and the coefficient 3.
[0088] Optionally, the above network anomaly identification method further includes:
[0089] Step S206 , in response to the target network interface having an abnormal access risk in the second time period, triggering and executing a network abnormality protection measure.
[0090] The above-mentioned network anomaly protection measures may include but are not limited to: generating a network anomaly alarm, issuing a network anomaly alarm based on a monitoring system, and triggering and executing a network interface repair script or process.
[0091] The network anomaly identification method provided by the above embodiment of the present invention can achieve the following technical effects:
[0092] (1) The target time series prediction algorithm can accurately fit the periodicity, seasonality, trend and other characteristics of network interface access data, so as to accurately predict the access volume of the network interface in the future time period, thereby improving the accuracy of the predicted data;
[0093] (2) The abnormal access threshold is set based on the actual access volume and predicted access volume in the previous time period, which improves the timeliness of the abnormal access threshold. Based on the comparison result of the abnormal access threshold and the standard deviation of the residual sequence corresponding to the future time period, it is possible to accurately determine whether the network interface has an abnormal access risk in the future time period, and then to provide timely and accurate warning of the abnormal access risk of the network interface, and timely trigger and execute network abnormality protection measures to improve the efficiency of network abnormality maintenance;
[0094] (3) The time period and time window in the prediction process can be adjusted according to the confidence requirements of the target time series prediction model to ensure the accuracy of the prediction data.
[0095] In this embodiment, a network anomaly identification device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, a "module" is a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0096] Figure 3 is a structural block diagram of an optional network anomaly identification device according to an embodiment of the present invention. Figure 3 As shown, the device comprises:
[0097] The acquisition module 301 is used to acquire the network interface access log of the target server;
[0098] The processing module 302 is used to process the text data of the network interface access log using the target stream processing framework to obtain a processing result, wherein the processing result at least includes the time series data of the target network interface, and the time series data is used to determine the actual access volume of the target network interface at any access time;
[0099] A prediction module 303 is used to predict the time series data of the first time period using the target time series prediction model to obtain a prediction result, wherein the prediction result is used to determine the predicted access volume of the target network interface in the second time period, and the second time period is a time period whose time series is after the first time period and adjacent to the first time period;
[0100] The identification module 304 is used to identify abnormalities of the target network interface based on the predicted access volume and the actual access volume in the first time period to obtain an identification result, wherein the identification result is used to determine whether there is an abnormal access risk to the target network interface in the second time period.
[0101] Optionally, the above-mentioned processing module 302 includes: performing data cleaning on the text data to obtain cleaned text data; performing data parsing on the cleaned text data using a data stream processing function to obtain a parsing result, wherein the parsing result is used to determine the network interface name and timestamp information of each access log; and aggregating and counting the parsing results based on a preset time window to obtain a processing result.
[0102] Optionally, Figure 4 is a structural block diagram of another optional network anomaly identification device according to an embodiment of the present invention. Figure 4 As shown, in addition to the above modules, the device also includes: a training module 305, which is used to construct an initial time series prediction model based on a time series prediction algorithm; obtain historical data of the target network interface in at least two time periods, wherein the historical data is used to determine the historical access volume of at least two time periods; perform data preprocessing and data segmentation on the historical data to obtain a training set and a test set; in the process of training the initial time series prediction model using the training set, repeatedly iterate and adjust the model parameters to obtain a time series prediction model to be evaluated; evaluate the time series prediction model to be evaluated using the test set to obtain an evaluation result, wherein the evaluation result is used to determine the model parameters to be adjusted; adjust the model parameters to be adjusted of the time series prediction model to be evaluated based on the evaluation result to obtain a target time series prediction model.
[0103] Optionally, the prediction result includes a first prediction result and a second prediction result, the first prediction result includes the predicted visits at a target time point, the target time point being two time endpoints of a second time period, the second prediction result includes the predicted visits at other time points, the other time points being time points other than the target time point within the second time period, the above-mentioned prediction module 303 includes: using a target time series prediction model to predict the visits at the target time point to obtain a first prediction result; using the predicted visits at at least one time endpoint in the first prediction result to determine the predicted visits at other time points to obtain a second prediction result.
[0104] Optionally, the prediction module 303 includes: determining a time endpoint close to any other time point as a target time endpoint; and determining the predicted visit volume of the target time endpoint as the predicted visit volume of any other time point.
[0105] Optionally, the above-mentioned prediction module 303 includes: determining the time distance between any other time point and any time endpoint; determining the weight of the predicted visit volume of any time endpoint based on the time distance; performing weighted sum calculation on the predicted visit volume of the two time endpoints and the corresponding weights to obtain the predicted visit volume of any other time point.
[0106] Optionally, the above-mentioned identification module 304 includes: calculating the difference between the predicted access volume and the actual access volume corresponding to any time point to obtain a residual sequence; calculating the standard deviation of the residual sequence; comparing the standard deviation with the abnormal access threshold to obtain a comparison result; in response to the comparison result, determining that the standard deviation is greater than the abnormal access threshold, determining that the identification result is that the target network interface has an abnormal access risk in the second time period.
[0107] Optionally, in the above-mentioned network anomaly identification device: the abnormal access threshold is determined by the standard deviation of the residual sequence corresponding to the first time period.
[0108] Optionally, Figure 5 is a structural block diagram of another optional network anomaly identification device according to an embodiment of the present invention, such as Figure 5 As shown, in addition to the above modules, the device also includes: an execution module 306, which is used to trigger and execute network abnormality protection measures in response to the existence of abnormal access risks on the target network interface in the second time period.
[0109] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0110] According to another aspect of an embodiment of the present invention, a network anomaly identification system is provided, including a memory and a processor, wherein an executable computer program is stored in the memory, and the processor is configured to run the executable computer program to execute any one of the aforementioned network anomaly identification methods.
[0111] Optionally, in this embodiment, the memory may be configured to store a computer program for performing the following steps:
[0112] Step S1, obtaining the network interface access log of the target server;
[0113] Step S2, using the target stream processing framework to process the text data of the network interface access log to obtain a processing result, wherein the processing result at least includes the time series data of the target network interface, and the time series data is used to determine the actual access volume of the target network interface at any access time;
[0114] Step S3, using the target time series prediction model to predict the time series data of the first time period to obtain a prediction result, wherein the prediction result is used to determine the predicted access volume of the target network interface in the second time period, and the second time period is a time period after the first time period and adjacent to the first time period;
[0115] Step S4, based on the predicted access volume and the actual access volume in the first time period, an abnormality identification is performed on the target network interface to obtain an identification result, wherein the identification result is used to determine whether there is an abnormal access risk on the target network interface in the second time period.
[0116] Optionally, in this embodiment, the above-mentioned memory may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.
[0117] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:
[0118] Step S1, obtaining the network interface access log of the target server;
[0119] Step S2, using the target stream processing framework to process the text data of the network interface access log to obtain a processing result, wherein the processing result at least includes the time series data of the target network interface, and the time series data is used to determine the actual access volume of the target network interface at any access time;
[0120] Step S3, using the target time series prediction model to predict the time series data of the first time period to obtain a prediction result, wherein the prediction result is used to determine the predicted access volume of the target network interface in the second time period, and the second time period is a time period after the first time period and adjacent to the first time period;
[0121] Step S4, based on the predicted access volume and the actual access volume in the first time period, an abnormality identification is performed on the target network interface to obtain an identification result, wherein the identification result is used to determine whether there is an abnormal access risk on the target network interface in the second time period.
[0122] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiment and its optional implementation modes, and this embodiment will not be described in detail here.
[0123] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0124] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0125] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0126] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0127] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0128] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, magnetic disk or optical disk, etc., various media that can store program codes.
[0129] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A network anomaly identification method, characterized in that: include: Get the target server's network interface access log; Using a target stream processing framework to process the text data of the network interface access log to obtain a processing result, wherein the processing result at least includes time series data of the target network interface, and the time series data is used to determine the actual access volume of the target network interface at any access time; Using a target time series prediction model to predict the time series data of a first time period, to obtain a prediction result, wherein the prediction result is used to determine the predicted access volume of the target network interface in a second time period, the second time period being a time period whose timing is after the first time period and adjacent to the first time period; Based on the predicted access volume and the actual access volume in the first time period, performing abnormal identification on the target network interface to obtain an identification result, wherein the identification result is used to determine whether there is an abnormal access risk to the target network interface in the second time period; The target stream processing framework is used to process the text data of the network interface access log to obtain the processing result, which includes: performing data cleaning on the text data to obtain cleaned text data; performing data parsing on the cleaned text data using a data stream processing function to obtain a parsing result, wherein the parsing result is used to determine the network interface name and timestamp information of each access log; and performing aggregate statistics on the parsing result based on a preset time window to obtain the processing result; Based on the predicted access volume and the actual access volume in the first time period, the target network interface is identified for abnormalities, and the identification result obtained includes: calculating the difference between the predicted access volume and the actual access volume corresponding to any time point to obtain a residual sequence; calculating the standard deviation of the residual sequence; comparing the standard deviation with an abnormal access threshold to obtain a comparison result, wherein the abnormal access threshold is determined by the standard deviation of the residual sequence corresponding to the first time period; in response to the comparison result, determining that the standard deviation is greater than the abnormal access threshold, determining that the identification result is that the target network interface has the abnormal access risk in the second time period.
2. The network anomaly identification method according to claim 1, characterized in that: The method further comprises: Construct an initial time series prediction model based on the time series prediction algorithm; Acquire historical data of the target network interface in at least two time periods, wherein the historical data is used to determine the historical access volume of the at least two time periods; Performing data preprocessing and data segmentation on the historical data to obtain a training set and a test set; In the process of training the initial time series prediction model using the training set, iterating and adjusting model parameters repeatedly to obtain the time series prediction model to be evaluated; Using the test set to evaluate the time series prediction model to be evaluated, to obtain an evaluation result, wherein the evaluation result is used to determine the model parameters to be adjusted; The model parameters to be adjusted of the time series prediction model to be evaluated are adjusted based on the evaluation result to obtain the target time series prediction model.
3. The network anomaly identification method according to claim 1, characterized in that: The prediction result includes a first prediction result and a second prediction result, wherein the first prediction result includes the predicted visit volume at a target time point, the target time point being two time endpoints of the second time period, and the second prediction result includes the predicted visit volume at other time points, the other time points being time points other than the target time point within the second time period, and the time series data of the first time period is predicted using the target time series prediction model, and the prediction result obtained includes: Predicting the number of visits at a target time point using the target time series prediction model to obtain the first prediction result; The predicted visits at at least one time point in the first prediction result is used to determine the predicted visits at the other time points to obtain the second prediction result.
4. The network anomaly identification method according to claim 3, characterized in that: Determining the predicted number of visits at the other time points by using the predicted number of visits at the at least one time endpoint in the first prediction result includes: Determine a time endpoint close to any other time point as a target time endpoint; The predicted number of visits at the target time endpoint is determined as the predicted number of visits at any other time point.
5. The network anomaly identification method according to claim 3, characterized in that: Determining the predicted number of visits at the other time points by using the predicted number of visits at the at least one time endpoint in the first prediction result includes: Determine the time distance between any other time point and any time endpoint; Determine the weight of the predicted visit volume of any one time endpoint based on the time distance; The predicted visits at the two time endpoints and the corresponding weights are weighted summed to obtain the predicted visit volume at any other time point.
6. The network anomaly identification method according to claim 1, characterized in that: The method further comprises: In response to the target network interface having the abnormal access risk during the second time period, a network abnormality protection measure is triggered and executed.
7. A network anomaly identification system, characterized in that: The method comprises a memory and a processor, wherein the memory stores an executable computer program, and the processor is configured to run the executable computer program to execute the network anomaly identification method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Automatic identification of monitoring data anomalies at air quality stations
CN109034252A
Automatic anomaly detection method and system
CN112149860A
Abnormal traffic identification method and device, electronic equipment and storage medium
CN115118511A
Page view early warning method and device, electronic equipment and storage medium
CN115801602A
Abnormity warning method and device of business system and computer readable storage medium
CN116910343A
