Feature waveform fitting and sliding window-based frequent online and offline anomaly detection method for user and superior equipment

Abstract

The invention discloses a frequent online and offline anomaly detection method for a user and superior equipment based on characteristic waveform fitting and a sliding window, and relates to the technical field of online and offline anomaly detection of equipment, and the method comprises the steps: obtaining online and offline record data of the user, and carrying out the cleaning and format conversion; using a general rule to identify abnormal users who are frequently online and offline; carrying out feature modeling on the online and offline data of the abnormal users, counting the online and offline times of the users, constructing online and offline features of the abnormal users, carrying out waveform fitting among the users, judging the users with similar online and offline rhythms, and obtaining a fitted user group; when all or part of users have similar online and offline rhythms, counting the occurrence times and frequency of each user group by using a sliding window; and comparing the occurrence frequency of the user group with a dynamic threshold to obtain a sliding user group, and comparing the sliding user group with the fitting user group to judge whether the fault of the superior equipment causes frequent online and offline of the user. According to the invention, whether the superior equipment breaks down can be quickly determined.

Description

Technical Field

[0001] The present invention relates to the technical field of abnormal detection of device online and offline, and specifically to a method for abnormal detection of frequent online and offline of users and superior devices based on feature waveform fitting and sliding window. Background Art

[0002] In the field of operation and maintenance of telecommunication networks and wireless networks, the frequent online and offline of users and poor quality problems are extremely common and intractable fault scenarios. Once such faults occur, the network speed experienced by users will be significantly slowed down, the data transmission delay will increase substantially, and packet loss will occur frequently. In more serious cases, the stability of the network connection is damaged, resulting in users being frequently disconnected and unable to use network services normally. From the perspective of network monitoring data, the packet loss rate shows an obvious increasing trend, and the delay is also increasing continuously; while in the logs of network devices, a large number of user online and offline records can be clearly seen. These problems have extremely serious negative impacts on user experience, greatly reducing user satisfaction with network services, and may even lead to user loss.

[0003] The reasons for the frequent online and offline of users are relatively complex, and the common ones mainly fall into the following three categories:

[0004] 1. Physical layer problems: Poor connection of optical fibers or copper cables is a common cause. In actual network laying, due to factors such as improper construction, line aging, or external force pulling, the connection of optical fibers or copper cables may become loose or have poor contact, thereby affecting the stability of data transmission and causing users to frequently go online and offline. In addition, connector oxidation or damage is also a problem that cannot be ignored. Over time, network connectors are exposed to the air for a long time, prone to oxidation reactions, resulting in an increase in contact resistance and signal transmission being blocked; at the same time, human misoperations or accidental collisions may also cause connector damage, which will also cause instability of the user's network connection.

[0005] 2. Wireless signal problems: Weak or unstable signals will directly affect the connection quality between users and the network. In some areas with poor signal coverage, such as remote mountainous areas and deep inside large buildings, the wireless signal strength received by users is insufficient, resulting in a decrease in data transmission rate and even an inability to maintain a stable connection, thus frequently going online and offline. In addition, channel interference is also an important factor. Co-channel or adjacent-channel interference is relatively common in wireless network environments. When multiple wireless devices communicate using the same or similar frequencies, their signals will interfere with each other, resulting in a decrease in signal quality and frequent online and offline of the user network. For example, in some crowded places, such as office buildings and shopping malls, numerous wireless routers and mobile devices work simultaneously, easily causing serious channel interference.

[0006] 3. User-side device problems: Client device hardware failures may be caused by natural aging of the device, overheating damage, or component quality problems. For example, damage to the wireless network card or motherboard failure may cause the device to be unable to connect to the network normally, resulting in frequent online and offline phenomena. Client device software problems should not be underestimated either, and driver incompatibility is one of the common situations. When users update the operating system or install new applications, the original driver may be incompatible with the new system or software, affecting the stability of the connection between the device and the network. In addition, client device configuration errors may also cause problems. For example, users may incorrectly set network parameters, security keys, etc., resulting in the device being unable to connect to the network correctly, and frequently going online and offline.

[0007] However, the commonly used methods for detecting frequent online and offline anomalies have obvious limitations. The existing detection methods can only determine whether a single user is in an abnormal state, but cannot further determine whether a certain upper-level device in the physical layer has an abnormality. It is precisely this upper-level device abnormality that causes many users under it to collectively frequently go online and offline within a certain period of time. In actual network operation and maintenance, this limitation greatly hinders the efficiency of troubleshooting and resolution. Because it is impossible to accurately determine whether there is a problem with the upper-level device, it is impossible to conduct targeted troubleshooting and repairs. Each single user can only be checked one by one, which not only consumes a lot of time and manpower costs, but also has a wide range of troubleshooting and low efficiency.

[0008] Therefore, the urgent problem to be solved now is: to develop a detection technology that can accurately judge whether the physical layer upper-level equipment is abnormal, and determine whether its abnormality causes the downstream users to frequently go online and offline, so that after obtaining the demarcation result of the upper-level equipment, the single-user anomaly caused by the failure of the upper-level equipment can be accurately eliminated, effectively narrowing the scope of troubleshooting for user-side equipment problems and significantly improving the efficiency of network maintenance. Summary of the invention

[0009] In view of the needs and shortcomings of current technology development, the present invention provides a method for detecting frequent online and offline anomalies of users and upper-level devices based on characteristic waveform fitting and sliding windows.

[0010] The present invention provides a method for detecting the frequent online and offline anomalies of users and upper-level devices based on characteristic waveform fitting and sliding windows, and the technical solution adopted to solve the above technical problems is as follows:

[0011] A method for detecting frequent online and offline anomalies of users and upper-level devices based on characteristic waveform fitting and sliding windows, comprising the following steps:

[0012] S1. Obtain user online and offline record data;

[0013] S2. Clean and convert the format of the acquired data;

[0014] S3. Use general rules to identify abnormal users with frequent logins and logouts;

[0015] S4. Perform feature modeling on the login and logout data of abnormal users, accumulate and count the number of logins and logouts of users according to the time dimension, and construct the login and logout features of abnormal users;

[0016] S5. Perform waveform fitting between users based on the login and logout features of abnormal users, judge which users have similar login and logout rhythms, and obtain the fitted user groups;

[0017] S6. When all or some users have similar login and logout rhythms, use a sliding window to count the occurrence times and frequencies of each user group;

[0018] S7. Compare the occurrence times of the user group with the dynamic threshold to obtain the sliding user group, compare the sliding user group with the fitted user group, and judge whether the frequent logins and logouts of users are caused by the failure of the superior device.

[0019] Optionally, execute step S1 to obtain the user login and logout record data through the 3A server;

[0020] The 3A server refers to a system that provides authentication, authorization, and accounting services in a computer network. The 3A server determines the network resources and services that a user has permission to access by verifying the user's identity, and at the same time records the usage of network resources by the user, including login and logout times, data transfer volume, and session duration;

[0021] The user login and logout record data obtained through the 3A server includes the following fields: "Date", "Username", "User login time", "User logout time", "BRAS device name", "OLT device name", "PON port device name", and "Secondary splitter device name".

[0022] Optionally, step S2 specifically includes:

[0023] S2.1. Clean the user login and logout data: If there is a missing value in at least one of the fields of "Date", "Username", "User login time", "User logout time", "BRAS device name", "OLT device name", "PON port device name", and "Secondary splitter device name" for a certain user at a certain time point, use the data of other time points of this user for backfilling. If there is a missing value in the above-mentioned field for all time points of this user, the data of this user is directly discarded and does not participate in the subsequent delimitation process;

[0024] S2.2. Convert the format of user online / offline data: Set the data formats of "date", "user online time", and "user offline time" to time types, and set the data formats of "username", "BRAS device name", "OLT device name", "PON port device name", and "secondary splitter device name" to string types.

[0025] Optionally, the general rule is: There are 5 or more online and offline behaviors within one minute, and the time interval between online and offline is less than or equal to 1 minute.

[0026] If a user exhibits behaviors that conform to the above general rule at a certain time point, it is considered that the user has an abnormal behavior of frequent online / offline.

[0027] Optionally, accumulate and count the number of frequent online / offline of the abnormal users identified in step S3 according to the time dimension, and construct the online / offline characteristics of abnormal users. Among them, for a certain user, its characteristic is a 1×n vector, and n depends on the length of time and the time granularity.

[0028] Optionally, step S5 specifically includes:

[0029] S5.1. For users under the same superior device, if the superior device fails, users may exhibit abnormal behaviors of frequent online / offline. At this time, introduce formula (1), sum the corresponding dimensions of the online / offline characteristics of all users under the failed superior device and divide by the number of users to obtain the user prototype of this superior device, and then use the Pearson correlation coefficient to determine whether the online / offline characteristics of each user are similar to the prototype:

[0030]

[0031] where x is the online / offline characteristic of the user, y is the user prototype, ρ x,y is the Pearson correlation coefficient, Cov(x,y) is the covariance of x and y, σ x is the variance of x, σ y is the variance of y, E(x) is the expectation of x, E(y) is the expectation of y, and n is the number of dimensions of the characteristics;

[0032] S5.2. Set a threshold α. If the Pearson correlation coefficients of the online / offline characteristics of all users and the prototype are higher than the threshold α, it is considered that all users under this superior device have a similar online / offline rhythm; if only the Pearson correlation coefficients of the online / offline characteristics of some users and the prototype are higher than the threshold α, it is considered that some users have a similar online / offline rhythm;

[0033] S5.3. Record the above all users or some users as the fitting user group.

[0034] Optionally, step S6 is executed, the length of the sliding window is set to 1 minute, the users within each sliding window are detected, and the abnormal users within one sliding window form a user group, and then the occurrence times of each user group are counted.

[0035] Optionally, step S7 specifically includes:

[0036] S7.1. Compare the occurrence times of each user group with the dynamic threshold, and record the user group with the occurrence times higher than the dynamic threshold and the largest number of users as the sliding user group; wherein, the dynamic threshold = the number of abnormal users * β, and β ranges from 0.8 to 0.95;

[0037] S7.2. Take the intersection of the fitting user group and the sliding user group to obtain the fitting quality difference users:

[0038] a) If the number of fitting quality difference users is equal to the total number of users, it indicates that a failure of the upper-level device has caused the frequent online and offline of its subordinate users. At this time, record the name, time, and demarcation conclusion of the upper-level device as the demarcation result;

[0039] b) If the number of fitting quality difference users is less than the total number of users, it indicates that the upper-level device has not failed, but only some users under the upper-level device have frequent online and offline. At this time, record the user name, time, and demarcation conclusion as the demarcation result;

[0040] S7.3. Output all demarcation results.

[0041] A method for detecting frequent online and offline anomalies of users and upper-level devices based on feature waveform fitting and sliding windows according to the present invention has the following beneficial effects compared with the prior art:

[0042] 1. The present invention can quickly determine whether a failure of the upper-level device causes the frequent online and offline of users, changing the situation in the past where only single-user anomalies can be detected and the failure of the upper-level device cannot be determined. It can greatly improve the accuracy and efficiency of fault location, avoid blind troubleshooting among a large number of devices, thereby shortening the fault troubleshooting time, reducing the waste of manpower and material resources, improving the efficiency of overall network maintenance, enabling the network to return to normal operation faster, and reducing the impact duration of network faults on users;

[0043] 2. The present invention can quickly solve the problems of frequent online and offline of users and poor quality caused by anomalies of the upper-level device, effectively improve the network usage experience of users, so that users are no longer troubled by problems such as slow network speed, high latency, packet loss, and frequent disconnection, making the network connection more stable and data transmission more smooth, thereby enhancing users' satisfaction with network services, reducing user churn, and helping to improve the market competitiveness of operators. Description of the Drawings

[0044] AppendixFigure 1 This is the flowchart of the method of the present invention. Detailed implementation manners

[0045] In order to make the technical solution, the technical problems solved and the technical effects of the present invention clearer and more understandable, the following combines specific embodiments to clearly and completely describe the technical solution of the present invention.

[0046] Embodiment:

[0047] Combined with the attached Figure 1 , this embodiment proposes a method for detecting frequent abnormal online and offline of users and superior devices based on feature waveform fitting and sliding window, which includes the following steps:

[0048] S1. Obtain user online and offline record data through a 3A server.

[0049] The 3A server refers to a system that provides authentication, authorization, and accounting services in a computer network. The 3A server determines the network resources and services that a user has the right to access by verifying the user's identity, and at the same time records the usage of network resources by the user, including login and logout times, data transfer volume, and session duration.

[0050] In this step, the user online and offline record data obtained through the 3A server includes the following fields: "date", "username", "user online time", "user offline time", "BRAS device name", "OLT device name", "PON port device name", and "secondary splitter device name".

[0051] S2. Clean and convert the obtained data, specifically including:

[0052] S2.1. Clean the user online and offline data: If at least one of the fields of "date", "username", "user online time", "user offline time", "BRAS device name", "OLT device name", "PON port device name", and "secondary splitter device name" of a certain user at a certain time point has a missing value, use the data of other time points of this user for backfilling. If the data of all time points of this user have the above-mentioned missing value in a certain field, the data of this user will be directly discarded and will not participate in the subsequent delimiter process;

[0053] S2.2. Convert the format of user online and offline data: Set the data formats of "date", "user online time", and "user offline time" to the time type, and set the data formats of "user name", "BRAS device name", "OLT device name", "PON port device name", and "secondary splitter device name" to the string type.

[0054] S3. Use general rules to identify abnormal users with frequent online and offline activities.

[0055] The general rules preset in this step are: There are 5 or more online and offline behaviors within one minute, and the time interval between online and offline is less than or equal to 1 minute.

[0056] If a user shows behaviors that meet the above general rules at a certain time point, it is considered that the user has abnormal frequent online and offline activities.

[0057] S4. Perform feature modeling on the online and offline data of abnormal users, accumulate and count the number of online and offline times of users according to the time dimension, and construct the online and offline features of abnormal users.

[0058] For a certain user, its online and offline feature is a 1×n vector, where n depends on the length of time and the time granularity. If the data is at a 1-minute granularity for 1 hour, then n is 60.

[0059] S5. Perform waveform fitting between users based on the online and offline features of abnormal users, and determine which users have similar online and offline rhythms to obtain a fitted user group.

[0060] It should be noted in this step that for users under the same superior device, if the superior device fails, users may show abnormal behaviors of frequent online and offline, and the online and offline behaviors between users will be relatively similar.

[0061] The specific implementation process of step S5 is as follows:

[0062] S5.1. Introduce formula (1), sum the corresponding dimensions of the online and offline features of all users under the faulty superior device and divide by the number of users to obtain the user prototype of this superior device, and then use the Pearson correlation coefficient to determine whether the online and offline features of each user are similar to the prototype:

[0063]

[0064] Among them, x is the online and offline feature of the user, y is the user prototype, ρ x,y is the Pearson correlation coefficient, Cov(x,y) is the covariance of x and y, σ x is the variance of x, σ yis the variance of y, E(x) is the expectation of x, E(y) is the expectation of y, and n is the number of dimensions of the features.

[0065] For example, suppose there are three users, and their online and offline times are (10:00, 10:30), (10:05, 10:35), and (10:10, 10:40) respectively. Converted to numerical values in minutes, they are (600, 630), (605, 635), and (610, 640) respectively. Summing the corresponding dimensions and dividing by the number of users 3, the user prototype is ((600 + 605 + 610) / 3, (630 + 635 + 640) / 3) = (605, 635). This is just a simple example here. In reality, there may be more dimensions of online and offline features, including online time, offline time, online duration, online and offline frequency, etc.

[0066] The Pearson correlation coefficient is a statistical indicator used to measure the linear correlation between two variables, and its value range is between -1 and 1. By calculating the Pearson correlation coefficient between the online and offline features of each user and the user prototype obtained above, it is judged whether the online and offline features of the user are similar to the prototype.

[0067] S5.2. Set a threshold α, and the specific value of the threshold α is 0.8;

[0068] If the Pearson correlation coefficients between the online and offline features of all users and the prototype are higher than the threshold α, it is considered that all users under the superior device have similar online and offline rhythms;

[0069] If only the Pearson correlation coefficients between the online and offline features of some users and the prototype are higher than the threshold α, it is considered that some users have similar online and offline rhythms.

[0070] S5.3. Record the above all users or some users as the fitting user group.

[0071] S6. When there are all users or some users with similar online and offline rhythms, set the length of the sliding window to 1 minute, detect the users within each sliding window, form a user group for the abnormal users within one sliding window, and then count the occurrence times of each user group.

[0072] The user groups formed in this step can be expressed as: [[a,b,c,d],[a,b,c],[a,b,c,d,e]……], where [[a,b,c,d] means that users a, b, c, and d are abnormal within a certain sliding window.

[0073] S7. Compare the occurrence times of the user groups with the dynamic threshold to obtain the sliding user group, and compare the sliding user group with the fitted user group to determine whether the frequent online and offline of users is caused by the failure of the superior device.

[0074] The specific implementation process of step S7 is as follows:

[0075] S7.1. Compare the occurrence times of each user group with the dynamic threshold, and record the user group with the number of users higher than the dynamic threshold and the largest number of users as the sliding user group; where the dynamic threshold = the number of abnormal users * β, and β ranges from 0.8 to 0.95, and the optimal value is 0.9.

[0076] S7.2. Take the intersection of the fitted user group and the sliding user group to obtain the fitted quality difference users:

[0077] a) If the number of fitted quality difference users is equal to the total number of users, it means that the frequent online and offline of the users under the superior device is caused by the failure of the superior device. At this time, record the name, time and delimitation conclusion of the superior device as the delimitation result;

[0078] b) If the number of fitted quality difference users is less than the total number of users, it means that the superior device has not failed, but only some users under the superior device have frequent online and offline. At this time, record the user name, time and delimitation conclusion as the delimitation result;

[0079] S7.3. Output all delimitation results.

[0080] In summary, by using the method for detecting frequent online and offline anomalies of users and superior devices based on feature waveform fitting and sliding window of the present invention, it can quickly determine whether the frequent online and offline of users is caused by the failure of the superior device, changing the situation in the past where only single-user anomalies could be detected and the failure of the superior device could not be determined.

[0081] The above uses specific examples to elaborate in detail the principle and implementation manner of the present invention. These embodiments are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, those skilled in the art of the present technology, without departing from the principle of the present invention, any improvements and modifications made to the present invention shall fall within the scope of patent protection of the present invention.

Claims

1. A method for detecting frequent online and offline anomalies of users and superior devices based on feature waveform fitting and sliding window, characterized in that, It includes the following steps: S1. Obtain the user's online and offline record data; S2. Clean and convert the obtained data; S3. Use general rules to identify abnormal users with frequent online and offline activities; S4. Conduct feature modeling on the online and offline data of abnormal users, accumulate and statistically analyze the number of online and offline times of users according to the time dimension, and construct the online and offline features of abnormal users; S5. Perform waveform fitting between users based on the online and offline features of abnormal users, judge which users have similar online and offline rhythms, and obtain the fitting user groups; S6. When all or some users have similar online and offline rhythms, use a sliding window to statistically analyze the occurrence times and frequencies of each user group; S7. Compare the occurrence times of the user group with the dynamic threshold to obtain the sliding user group, compare the sliding user group with the fitting user group, and judge whether the frequent online and offline of users is caused by the failure of the superior device.

2. The method for detecting frequent online / offline anomalies of users and superior devices based on feature waveform fitting and sliding window according to claim 1, wherein, Execute step S1 to obtain the user's online and offline record data through the 3A server; The 3A server refers to a system that provides authentication, authorization, and accounting services in a computer network. The 3A server determines the network resources and services that a user has permission to access by verifying the user's identity, and at the same time records the usage of network resources by the user, including login and logout times, data transfer volume, and session duration; The user's online and offline record data obtained through the 3A server includes the following fields: "date", "username", "user online time", "user offline time", "BRAS device name", "OLT device name", "PON port device name", and "secondary splitter device name".

3. The method for detecting frequent online and offline anomalies of users and superior devices based on feature waveform fitting and sliding window according to claim 2, wherein The specific steps of S2 include: S2.

1. Clean the user's online and offline data: If there is a missing value in at least one of the fields of "date", "username", "user online time", "user offline time", "BRAS device name", "OLT device name", "PON port device name", and "secondary splitter device name" for a certain user at a certain time point, use the data of other time points of this user for backfilling. If all time points of this user have a missing value in the above-mentioned field, the data of this user is directly discarded and does not participate in the subsequent delimiter process; S2.

2. Convert the format of the user's online and offline data: Set the data formats of "date", "user online time", and "user offline time" to the time type, and set the data formats of "username", "BRAS device name", "OLT device name", "PON port device name", and "secondary splitter device name" to the string type.

4. The method for detecting frequent online and offline anomalies of users and superior devices based on feature waveform fitting and sliding window according to claim 3, characterized in that, The general rule is: There are 5 or more online and offline behaviors within one minute, and the time interval between online and offline is less than or equal to 1 minute; If a certain user has a behavior that conforms to the above general rule at a certain time point, it is considered that this user has an abnormal situation of frequent online and offline.

5. The method for detecting frequent online and offline anomalies of users and superior devices based on feature waveform fitting and sliding window according to claim 4, wherein, Accumulatively count the frequent online and offline times of the abnormal users identified in step S3 according to the time dimension, and construct the online and offline characteristics of the abnormal users. Among them, for a certain user, its characteristics are a 1×n vector, and n depends on the length of time and the granularity of time.

6. The method for detecting abnormal frequent online and offline of users and superior devices based on feature waveform fitting and sliding window according to claim 5, characterized in that, The specific steps of step S5 are as follows: S5.

1. For the users under the same superior device, if the superior device fails, the users may exhibit abnormal behaviors of frequent online and offline. At this time, introduce formula (1), sum the corresponding dimensions of the online and offline characteristics of all users under the faulty superior device and divide by the number of users to obtain the user prototype of this superior device. Then use the Pearson correlation coefficient to determine whether the online and offline characteristics of each user are similar to the prototype: where x is the online / offline feature of the user, y is the user prototype, and ρ x,y is the Pearson correlation coefficient, Cov(x, y) is the covariance of x and y, and σ x is the variance of x, and σ y is the variance of y, E(x) is the expectation of x, E(y) is the expectation of y, and n is the number of dimensions of the features; S5.

2. Set a threshold α. If the Pearson correlation coefficients between the online and offline characteristics of all users and the prototype are higher than the threshold α, it is considered that all users under this superior device have similar online and offline rhythms; if only the Pearson correlation coefficients between the online and offline characteristics of some users and the prototype are higher than the threshold α, it is considered that some users have similar online and offline rhythms; S5.

3. Record the above all users or some users as the fitting user group.

7. The method for detecting frequent online and offline anomalies of users and superior devices based on feature waveform fitting and sliding window according to claim 6, characterized in that, Execute step S6, set the length of the sliding window to 1 minute, detect the users in each sliding window, form a user group for the abnormal users in one sliding window, and then count the occurrence times of each user group.

8. The method for detecting frequent online and offline anomalies of users and superior devices based on feature waveform fitting and sliding window according to claim 7, characterized in that The specific steps of step S7 are as follows: S7.

1. Compare the occurrence times of each user group with the dynamic threshold, and record the user group with the occurrence times higher than the dynamic threshold and the largest number of users as the sliding user group; among them, the dynamic threshold = the number of abnormal users * β, and β ranges from 0.8 to 0.95; S7.

2. Take the intersection of the fitting user group and the sliding user group to obtain the fitting quality-difference users: a) If the number of fitting quality-difference users is equal to the total number of users, it indicates that the failure of the superior device has caused the frequent online and offline of the users under it. At this time, record the name, time, and demarcation conclusion of the superior device as the demarcation result; b) If the number of fitting quality-difference users is less than the total number of users, it indicates that the superior device has not failed, but only some users under this superior device have frequent online and offline. At this time, record the user name, time, and demarcation conclusion as the demarcation result; S7.

3. Output all demarcation results.

Images