Method, apparatus, device, and storage medium for determining object abnormality degree

Abstract

The present disclosure provides a method, apparatus, device, and storage medium for determining the abnormality degree of an object, relating to the field of computer technologies, especially to technical fields such as artificial intelligence, big data, and machine learning. The specific implementation solution is as follows: determine detailed features according to the web access data of the target object; determine longitudinal comparison features according to the web access data of the target object and the web access data of other objects; determine horizontal comparison features according to the web access data of the target object in the current time period and the web access data of the target object in the previous time period; and perform anomaly detection on the longitudinal comparison features, horizontal comparison features, and detailed features to obtain the target abnormality degree of the target object.

Description

Technical Field

[0001] This disclosure relates to the field of computer technologies, and in particular, to fields such as artificial intelligence, big data, and machine learning. Background Art

[0002] With the increasing expansion of the network scale and the complexity of the network structure, information security issues have become increasingly serious. How to protect the security of sensitive information has become an important problem. In recent years, network attacks have gradually shifted from external attacks to internal attacks. Internal attacks have characteristics such as generality, weak specificity, and strong concealment. They can easily bypass the monitoring of firewalls and intrusion detection systems, are more difficult to prevent than external attacks such as external network viruses and hacker attacks, and cause greater losses. Firewalls and IDSs (intrusion detection systems) can no longer meet these security requirements. Internal attacks are generally caused by internal employees, such as commercial spies, carrying information when changing positions, and retaliation after leaving the company. Therefore, how to detect abnormal risk users from a large number of user access behaviors has become an urgent problem to be solved. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, storage medium, and program product for determining the abnormality degree of an object.

[0004] According to one aspect of this disclosure, a method for determining the abnormality degree of an object is provided, including: determining detailed features according to the web access data of a target object; determining longitudinal comparison features according to the web access data of the target object and the web access data of other objects; determining transverse comparison features according to the web access data of the target object in the current time period and the web access data of the target object in the previous time period; and performing anomaly detection on the longitudinal comparison features, transverse comparison features, and detailed features to obtain the target abnormality degree of the target object.

[0005] According to another aspect of this disclosure, an apparatus for determining the abnormality degree of an object is provided, including: a first determination module for determining detailed features according to the web access data of a target object; a second determination module for determining longitudinal comparison features according to the web access data of the target object and the web access data of other objects; a third determination module for determining transverse comparison features according to the web access data of the target object in the current time period and the web access data of the target object in the previous time period; and a detection module for performing anomaly detection on the longitudinal comparison features, transverse comparison features, and detailed features to obtain the target abnormality degree of the target object.

[0006] Another aspect of the present disclosure provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method shown in the embodiments of the present disclosure.

[0007] According to another aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method shown in the embodiments of the present disclosure.

[0008] According to another aspect of the embodiments of the present disclosure, there is provided a computer program product, including computer programs / instructions, characterized in that when the computer programs / instructions are executed by a processor, the steps of the method shown in the embodiments of the present disclosure are implemented.

[0009] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understandable through the following description. Description of the Drawings

[0010] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0011] Figure 1 is a schematic diagram of an application scenario of a method, apparatus, electronic device, and storage medium for determining the abnormality degree of an object according to an embodiment of the present disclosure;

[0012] Figure 2 Schematically shows a flowchart of a method for determining the abnormality degree of an object according to an embodiment of the present disclosure;

[0013] Figure 3 Schematically shows a flowchart of a method for detecting abnormalities in longitudinal comparison features, lateral comparison features, and detailed features according to an embodiment of the present disclosure;

[0014] Figure 4A Schematically shows a schematic diagram of a method for determining a first detection result according to an embodiment of the present disclosure;

[0015] Figure 4B Schematically shows a schematic diagram of a method for determining a second detection result according to an embodiment of the present disclosure;

[0016] Figure 4C Schematically shows a schematic diagram of a method for determining a third detection result according to an embodiment of the present disclosure;

[0017] Figure 4D Schematically shows a schematic diagram of a method for determining a target abnormality degree according to an embodiment of the present disclosure;

[0018] Figure 5 Schematically shows a schematic diagram of a method for determining the abnormality degree of an object according to an embodiment of the present disclosure;

[0019] Figure 6 Schematically shows a block diagram of a device for determining the abnormality degree of an object according to an embodiment of the present disclosure;

[0020] Figure 7 Schematically shows a block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure. Detailed implementation manners

[0021] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to assist understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, descriptions of well-known functions and structures are omitted below for clarity and conciseness.

[0022] The following will be combined with Figure 1 Describe the application scenarios of the method, device, electronic device, and storage medium for determining the abnormality degree of an object provided by the present disclosure.

[0023] Figure 1 Is a schematic diagram of the application scenarios of the method, device, electronic device, and storage medium for determining the abnormality degree of an object according to an embodiment of the present disclosure. It should be noted that Figure 1 The shown is only an example of the application scenarios to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments, or scenarios.

[0024] As Figure 1 Shown, the application scenario 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0025] Users can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0026] The terminal devices 101, 102, and 103 can be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and so on.

[0027] The server 105 can be a server that provides various services, such as a background management server that supports the websites browsed by users using the terminal devices 101, 102, and 103. The background management server can analyze and process data such as user requests received, and feedback the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0028] Exemplarily, in this embodiment, the terminal devices 101, 102, and 103 can be in an intranet environment. Users can browse intranet websites using the terminal devices 101, 102, and 103. The server 105 can be, for example, a background management server that supports the intranet websites browsed by users using the terminal devices 101, 102, and 103.

[0029] When users access websites using the terminal devices 101, 102, and 103, the terminal devices 101, 102, and 103 can generate Internet usage logs. The Internet usage logs record which web pages the user accessed at a specific time and whether the access was successful. Additionally, cookie information, user-agent information, etc. can also be recorded when accessing web pages. Among them, the user-agent can enable the server to identify the operating system and version used by the user, the type of CPU (Central Processing Unit), the browser and version, the browser rendering engine, the browser language, browser plugins, etc.

[0030] It should be noted that the method for determining the abnormality degree of an object provided in the embodiments of the present disclosure can generally be executed by the server 105. Correspondingly, the device for determining the abnormality degree of an object provided in the embodiments of the present disclosure can generally be set in the server 105. The method for determining the abnormality degree of an object provided in the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Correspondingly, the device for determining the abnormality degree of an object provided in the embodiments of the present disclosure can also be set in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105.

[0031] It should be understood that Figure 1 the numbers of terminal devices, networks, and servers in

[0032] The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services (″Virtual Private Server″, or simply ″VPS″). The server can also be a server of a distributed system or a server combined with blockchain.

[0033] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure, and application, etc., of the user's personal information involved all comply with the provisions of relevant laws and regulations, adopt necessary confidentiality measures, and do not violate public order and good customs.

[0034] In the technical solution of the present disclosure, before obtaining or collecting the user's personal information, the authorization or consent of the user is obtained.

[0035] The following will be combined with Figure 2 to describe the method for determining the abnormality degree of an object provided by the present disclosure.

[0036] Figure 2 A flowchart of the method for determining the abnormality degree of an object according to an embodiment of the present disclosure is schematically shown.

[0037] As Figure 2 shown, the method 200 for determining the abnormality degree of an object includes, in operation S210, determining detailed features according to the web access data of the target object.

[0038] According to an embodiment of the present disclosure, the target object may include, for example, a user to be detected, and the user can access the web page through a terminal device.

[0039] According to an embodiment of the present disclosure, the web access data may include, for example, the data recorded by the terminal device and / or the server when the user accesses the web page, and may include, for example, at least one of the following: the number of user agent information used by the target object per unit time, the web pages accessed by the target object per unit time, the number of accesses to each web page, the total number of failures to access each web page by the target object per unit time, and the difference between the number of user agent information during non-working hours and the number of user agent information during working hours of the target object per unit time. The user agent information may include, for example, user-agent. Among them, the unit time can be set according to actual needs, for example, it can be one day.

[0040] According to an embodiment of the present disclosure, the detailed features can be used to represent the fine-grained online behavior characteristics of the user to be detected.

[0041] Then, in operation S220, longitudinal comparison features are determined according to the web access data of the target object and the web access data of other objects.

[0042] According to an embodiment of the present disclosure, other objects may include other users in addition to the user to be detected.

[0043] According to an embodiment of the present disclosure, the longitudinal comparison feature can be used to represent the Internet access behavior characteristics of the user to be detected and other users.

[0044] In operation S230, according to the web access data of the target object in the current time period and the web access data of the target object in the previous time period, a horizontal comparison feature is determined.

[0045] According to an embodiment of the present disclosure, the time period can be set according to actual needs. For example, the time period can be set to 1 week.

[0046] According to an embodiment of the present disclosure, the horizontal comparison feature can be used to represent the Internet access behavior characteristics of the user to be detected in the current time period and the previous time period.

[0047] In operation S240, anomaly detection is performed on the longitudinal comparison feature, the horizontal comparison feature, and the detailed feature to obtain the target anomaly degree of the target object.

[0048] According to an embodiment of the present disclosure, the target anomaly degree can be used to represent the anomaly degree of the target object. It can be determined whether the target object is an abnormal user according to the target anomaly degree. For example, it can be determined whether the target anomaly degree of the target user is greater than the anomaly degree threshold. If it is greater, it is determined that the target user is an abnormal user. Among them, the anomaly degree threshold can be set according to actual needs.

[0049] According to an embodiment of the present disclosure, for example, it can be detected whether there is a large deviation between the Internet access behavior characteristics of the user to be detected and other users according to the longitudinal comparison feature. According to the longitudinal comparison feature, it is detected whether there is a large deviation between the Internet access behavior characteristics of the user to be detected in the current time period and the previous time period. According to the detailed feature, it is detected whether there is an anomaly in the Internet access behavior characteristics of the user to be detected. Then, according to the results of the above three detections, the target anomaly degree of the user to be detected is determined.

[0050] The method for determining the object anomaly degree according to the embodiment of the present disclosure adopts multi-dimensional anomaly detection. It not only compares the user with himself in different periods, but also compares the user with other users, that is, anomaly detection is performed by combining multiple angles such as horizontal and vertical. Horizontal comparison is used to capture the changes in the relatively long-term behavior habits of users imperceptibly. Longitudinal comparison is used to capture the behavior gap of users compared with other users per unit time. Characterizing the user anomaly degree from both longitudinal and horizontal comparisons can better and more comprehensively detect abnormal users.

[0051] According to another embodiment of the present disclosure, longitudinal comparison features may include, for example, the number of user agent information used by an intranet user per unit time, the number of times each web page is accessed per unit time, the number of failed accesses to each web page per unit time, and the difference between the number of user agent information during non-working hours and the number of user agent information during working hours of the user per unit time, etc. Transverse comparison features may include, for example, the difference between the average number of user agent information used by an intranet user in the current time period and the average number of user agent information used in the previous time period, the difference between the average number of accesses to each web page by the user in the current time period and the average number of accesses to the corresponding web page in the previous time period, the difference between the average number of failed accesses to each web page by the user in the current time period and the average number of failed accesses to the corresponding web page in the previous time period, the change amount of the ratio of the difference between the number of user agent information during non-working hours and the number of user agent information during working hours of the user in the current time period compared to the previous time period, etc. Detail features may include, for example, the number of times each web page is accessed per unit time by the user in the current time period.

[0052] By using various types of features such as user agent information, the number of failed accesses, and the number of accesses during non-working hours, the dimension of the features becomes more comprehensive. In addition, by simultaneously using longitudinal comparison features, transverse comparison features, and detail features, where the longitudinal comparison features and transverse comparison features include summary, aggregation, and average features, and the detail features include time series features, the perspective of the features becomes more comprehensive. Thus, the user anomaly degree can be better characterized, and the accuracy of anomaly detection can be improved.

[0053] According to an embodiment of the present disclosure, the Internet access behavior logs of intranet users can be collected. The Internet access behavior logs record what intranet web pages all intranet users accessed at what specific time, whether the access was successful, cookie information, user-agent information, etc. when accessing the intranet web pages. Then, web page access data can be extracted from the Internet access behavior logs. Among them, the web page access data may include the web pages accessed by the user, whether each access was successful, user-agent information for each access, etc.

[0054] According to an embodiment of the present disclosure, the user-agent can be directly used as the user agent information. For example, the user-agent can also be encoded according to the page view volume corresponding to each user-agent to obtain the encoded information of the user-agent as the user agent information. Among them, the larger the page view volume corresponding to the user-agent field, the smaller the corresponding user-agent code. In this embodiment, the encoding can be stored in the form of a string. The smaller the code, the fewer the number of digits and the less storage space is occupied. By encoding the user-agent, the occupation of storage space can be reduced.

[0055] According to the disclosed embodiments, when detecting the user abnormality degree, the behavioral characteristics of the user during non-working hours are considered. Based on this, for example, if the number of users within a certain reference unit time is less than a predetermined number of users, it can be determined that the reference unit time belongs to a non-working time period. Among them, the reference unit time can be set according to actual needs. For example, it can be 1 hour. The predetermined number of users can be set according to actual needs. For example, the average number of online users in the reference unit time within a predetermined time can be counted, and then the predetermined number of users can be determined according to the average number of online users. The predetermined time can be set according to actual needs. For example, it can be the most recent 14 days.

[0056] According to the disclosed embodiments, for example, web pages with the number of users less than the user quantity threshold or the number of accesses less than the access count threshold can be merged and counted as one web page. Thereby, the dimension of the features can be reduced, avoiding feature sparsity. In addition, it can also make the detection of the abnormality degree more comprehensive.

[0057] Figure 3 A flowchart of a method for detecting abnormalities in longitudinal comparison features, transverse comparison features, and detailed features according to an embodiment of the present disclosure is schematically shown.

[0058] As Figure 3 shown, the method 340 for detecting abnormalities in longitudinal comparison features, transverse comparison features, and detailed features includes, in operation S341, detecting abnormalities in the longitudinal comparison features to obtain a first detection result.

[0059] According to the embodiments of the present disclosure, for example, a first machine learning model can be used to detect abnormalities in the transverse comparison features, thereby obtaining a first detection result. Among them, the first machine learning model can include at least one of an isolation forest model, an hbos (Histogram-based Outlier Score) model, and a copod (Copula-based Outlier Detection) model.

[0060] In operation S342, detect abnormalities in the transverse comparison features to obtain a second detection result.

[0061] According to the embodiments of the present disclosure, for example, a second machine learning model can be used to detect abnormalities in the transverse comparison features, thereby obtaining a second detection result. Among them, the second machine learning model can include at least one of an isolation forest model, an hbos model, and a copod model. It should be noted that the first machine learning model and the second machine learning model can be the same or different.

[0062] In operation S343, perform anomaly detection on the detailed features to obtain a third detection result.

[0063] According to an embodiment of the present disclosure, for example, a deep learning model can be used to perform anomaly detection on the detailed features to obtain a third detection result. Among them, the deep learning model can include at least one of an autoencoder model and a clustering model. Among them, the autoencoder model can include, for example, an LSTM (Long Short-Term Memory artificial neural network) model. The clustering model can include, for example, a kmeans (k-means clustering) model.

[0064] In operation S344, determine the target anomaly degree of the target object according to the first detection result, the second detection result, and the third detection result.

[0065] According to an embodiment of the present disclosure, for example, the first detection result, the second detection result, and the third detection result can be comprehensively considered to determine the target anomaly degree of the target object.

[0066] It should be noted that any execution order can be adopted among operations S341 to S343, and the present disclosure does not make specific limitations on this.

[0067] According to an embodiment of the present disclosure, for example, the Internet behavior characteristics of multiple users can be collected in advance as training data, and the training data is input into a network model, trained using the gradient descent method, and the parameters of the model are updated through multiple rounds of iteration until convergence.

[0068] The following refers to Figures 4A to 4D , and in combination with specific embodiments, the method for determining the target anomaly degree of the target object shown above is further described. Those skilled in the art can understand that the following exemplary embodiments are only for understanding the present disclosure, and the present disclosure is not limited thereto.

[0069] Exemplarily, in this embodiment, an isolation forest model, an hbos model, and a copod model can be used to perform anomaly detection on the longitudinal comparison features and the horizontal comparison features respectively, and an autoencoder model and a clustering model can be used to perform anomaly detection on the detailed features.

[0070] Among them, the isolation forest model applies the isolation forest algorithm. The isolation forest algorithm is an anomaly detection algorithm that can process large-scale multi-dimensional data. The isolation forest algorithm believes that abnormal samples are few and very different, so during the process of constructing a binary tree, abnormal samples are very easy to be isolated and thus closest to the root node. However, the isolation forest algorithm also has some disadvantages. For example, there are some users who are very abnormal only in a few individual indicators, but normal in other indicators. Then according to the logic of the isolation forest algorithm, if there are many users with relatively abnormal indicators, users who are abnormal in individual indicators are likely to be judged as normal users. Moreover, there is a certain randomness in the isolation forest algorithm.

[0071] Therefore, in order to perform anomaly assessment more comprehensively, HBOs and Copod models are also introduced in this embodiment for anomaly detection. HBOs is a combination of univariate methods and cannot model the dependencies between features, but it has a faster calculation speed and is friendly to large data sets. The basic assumption of HBOs is that each dimension of the data set is independent of each other, and then each dimension is divided into bins. The higher the density of the bins, the lower the corresponding anomaly. The advantages of HBOs are simplicity, low overhead, parallel calculation, application to large amounts of data, and good actual results in actual use. However, the disadvantage of HBOs is that it cannot consider the relationship between different features.

[0072] Therefore, the copod model is also used in this embodiment, which jointly models all features through the copula probability function, fully considering the relationship between different features. This model has a good detection effect when performing anomaly detection on more data sets.

[0073] In addition, in this embodiment, an LSTM-based autoencoder model can be used to detect anomalies in detailed features. On the other hand, the intermediate features of the autoencoder model, that is, the feature representation after dimensionality reduction, can be determined. Based on this intermediate feature, a kmeans clustering model can be constructed. The intermediate features are input into the kmeans clustering model to obtain the distance from each user to the cluster center as the basis for the user abnormality ranking. The advantage of this is that the kmeans clustering model is relatively poor in multi-dimensional feature scenarios. The intermediate features of the autoencoder model can reduce the dimensionality of the kmeans clustering model input and improve the detection effect. In addition, the powerful temporal feature learning ability of the LSTM-based autoencoder model can be fully utilized.

[0074] Based on this, Figure 4A A schematic diagram of a method for determining a first detection result according to an embodiment of the present disclosure is schematically shown.

[0075] like Figure 4A As shown, according to an embodiment of the present disclosure, for example, the longitudinal contrast feature 410 can be input into the isolation forest model 421 to determine the first abnormality 431 of the target object. The longitudinal contrast feature 410 is input into the hbos model 422 to determine the second abnormality 432 of the target object. The longitudinal contrast feature 410 is input into the copod model 423 to determine the third abnormality 433 of the target object. Then, the first abnormality 431, the second abnormality 432, and the third abnormality 433 can be determined as the first detection result.

[0076] Figure 4B A schematic diagram of a method for determining a second detection result according to an embodiment of the present disclosure is schematically shown.

[0077] As Figure 4B shown, according to an embodiment of the present disclosure, for example, the horizontal comparison feature 410' can be input into the isolation forest model 421' to determine the fourth anomaly degree 431' of the target object. The horizontal comparison feature 410' is input into the hbos model 422' to determine the fifth anomaly degree 432' of the target object. The horizontal comparison feature 410' is input into the copod model 423' to determine the sixth anomaly degree 433' of the target object. Then, the fourth anomaly degree 431', the fifth anomaly degree 432', and the sixth anomaly degree 433' are determined as the second detection result.

[0078] Figure 4C Schematically shows a schematic diagram of a method for determining a third detection result according to an embodiment of the present disclosure.

[0079] As Figure 4C shown, according to an embodiment of the present disclosure, for example, the detailed feature 440 can be input into the autoencoder model 451 to determine the intermediate feature and the seventh anomaly degree 461 of the target object. The intermediate feature is input into the clustering model 452 to determine the eighth anomaly degree 462 of the target object. Then, the seventh anomaly degree 461 and the eighth anomaly degree 462 are determined as the third detection result.

[0080] Figure 4D Schematically shows a schematic diagram of a method for determining the target anomaly degree according to an embodiment of the present disclosure.

[0081] As Figure 4D shown, according to an embodiment of the present disclosure, for example, the first anomaly degree 431, the second anomaly degree 432, the third anomaly degree 433, the fourth anomaly degree 431', the fifth anomaly degree 432', the sixth anomaly degree 433', the seventh anomaly degree 461, and the eighth anomaly degree 462 can be standardized and normalized respectively. Then, the standardized and normalized first anomaly degree 431, second anomaly degree 432, third anomaly degree 433, fourth anomaly degree 431', fifth anomaly degree 432', sixth anomaly degree 433', seventh anomaly degree 461, and eighth anomaly degree 462 are added together to obtain the target anomaly degree 470.

[0082] According to an embodiment of the present disclosure, the standardization can include, for example, Z-score processing. The normalization can be used, for example, to transform the anomaly degrees output by each model to between 0 and 1.

[0083] According to embodiments of the present disclosure, on the one hand, each different type of feature needs to cooperate with a specific type of model to achieve the best effect. For example, if high-dimensional features are implemented using a clustering model, the effect is poor. For example, detailed time-series features will obtain better results when combined with an autoencoder model because it can better learn time-series information. On the other hand, each model has its own advantages. Combining the detection results of multiple models means integrating the advantages of multiple models, which can achieve comprehensive detection. Finally, the detection results of multiple models can be mutually verified to enhance the credibility of user anomalies. For example, if multiple models all consider a user to be an abnormal user, then the probability that people consider this user to be abnormal will increase significantly.

[0084] The following refers to Figure 5 , and further illustrates the method for determining the anomaly degree of an object shown above in combination with specific embodiments. Those skilled in the art can understand that the following exemplary embodiments are only for understanding the present disclosure, and the present disclosure is not limited thereto.

[0085] Exemplarily, in this embodiment, the object is an intranet user, hereinafter simply referred to as a user.

[0086] Figure 5 Schematically shows a schematic diagram of a method for determining the anomaly degree of an object according to an embodiment of the present disclosure.

[0087] In Figure 5 , it is shown that data preparation 510 can be performed. According to embodiments of the present disclosure, the Internet access behavior logs of intranet users can be collected. The Internet access behavior logs record what intranet web pages all intranet users accessed at what specific time, whether the access was successful, cookie information, user-agent information, etc. when accessing the intranet web pages. Then, web access data can be extracted from the Internet access behavior logs. Among them, the web access data can include the web pages accessed by the user, whether each access was successful, the user-agent information of each access, etc. Exemplarily, in this embodiment, the web access data of intranet users in the most recent 14 days can be extracted from the Internet access behavior logs.

[0088] Then, the web access data can be preprocessed at 520. For some large companies, the data volume may be very large. In order to achieve feature extraction faster and better, in this embodiment, the original data can be simply and efficiently encoded. For example, the string length of a specific user-agent is 500, which occupies a large amount of database space and seriously affects the feature extraction efficiency. Therefore, the number of users and the page views (PVs) corresponding to all user-agents can be determined. Each user-agent is encoded in descending order of the PV volume. The larger the PV volume, the smaller the encoding. The encoding can be stored in the form of a string. The smaller the encoding, the fewer the number of digits and the less storage space it occupies. For example, the encoding can start from 1 and increase sequentially. Based on this, for a user-agent with the most occurrences and a string length of 500, it can be encoded as 1, with a length of 1. It can be seen that the occupied space size after encoding has changed from the original 500 characters to 1 character. In an actual application scenario, the number of user-agents in a company is at most in the tens of thousands level. Therefore, the occupied space size after encoding all user-agents is at most 5 characters.

[0089] In addition, in order to better evaluate the user anomaly degree, in this embodiment, various behavioral characteristics of users during non-working hours can be extracted. In an actual application scenario, the working hours of each company are not exactly the same. In order to better adapt to the different situations of each company, the unique non-working hours of each company can be formulated according to the actual data situation of each company. Exemplarily, in this embodiment, if the number of users in a certain hour is less than 30% of the average number of users per hour in the last 14 days, it is considered a non-working hour.

[0090] Generally, the number of intranet web pages in small and medium-sized companies is about 100 or less, but the number of intranet web pages in some large companies may exceed 100. In this embodiment, in order to handle the situation of a large number of intranet web pages, web pages with a user volume less than the user volume threshold or an access count less than the access count threshold can be determined as long-tail web pages, and these long-tail web pages are merged. Among them, the user volume threshold and the access count threshold can be set according to actual needs respectively. For example, web pages with a user volume less than 5 or an access count less than 200 can be determined as long-tail web pages, and these long-tail web pages are merged into one type of web page. Long-tail web pages may be web pages that the company has abandoned, but they may also be some abnormal web pages, which can be used as a breakthrough point for detecting abnormal behaviors. By merging long-tail web pages instead of directly deleting them, it not only prevents the poor model effect caused by high-dimensional feature explosion and feature sparsity, but also can more comprehensively evaluate the anomaly degree.

[0091] Next, feature construction 530 can be performed. According to an embodiment of the present disclosure, for example, multi-dimensional features can be used, which include various types of features. It can not only include detailed and time-series features, such as detailed feature 541, but also include aggregated summary features, such as horizontal comparison feature 542 and vertical comparison feature 543. Additionally, from the perspective of comparison, in this embodiment, horizontal, vertical and other dimensions can be used simultaneously to evaluate user abnormality. Specifically, vertical comparison can include, for example, whether there is a large deviation in the behavioral characteristics between a user every day and other users. If there is, the user is determined to be an abnormal user. Horizontal comparison can include, for example, whether there is a large deviation in the behavioral characteristics of a user in the most recent week compared to those in the previous week. If there is, the user is determined to be an abnormal user. The specific judgment logic can be learned by the model according to the data distribution, and the abnormality of the user can be detected based on the learned and trained model to determine whether the user is an abnormal user.

[0092] According to an embodiment of the present disclosure, the vertical comparison feature 543 can include, for example, the number of user-agents used by an intranet user every day, the number of times each web page is accessed every day, the number of failed accesses to each web page every day, and the difference between the number of user-agents during non-working hours and the number of user-agents during working hours of the user every day. The horizontal comparison feature 542 can include, for example, the difference between the average number of user-agents used by an intranet user in the most recent week and the average number of user-agents used in the previous week, the difference between the average number of accesses to each web page by the user in the most recent week and the average number of accesses to the corresponding web page in the previous week, the difference between the average number of failed accesses to each web page by the user in the most recent week and the average number of failed accesses to the corresponding web page in the previous week, and the change amount of the difference between the number of user-agents during non-working hours and the number of user-agents during working hours of the user in the most recent week compared to the difference between the number of user-agents during non-working hours and the number of user-agents during working hours in the previous week. The detailed feature 541 can include, for example, the number of times each web page is accessed by the user every day in the most recent 7 days.

[0093] Next, the features obtained after data preprocessing and feature construction can be input into the network model. Exemplarily, in this embodiment, the network model may include an Isolation Forest model 551, an HBOS model 552, a Copod model 553, an autoencoder model 554, and a Kmeans clustering model 555. Specifically, the trained Isolation Forest model 551, HBOS model 552, and Copod model 553 can be used to perform anomaly detection on the longitudinal comparison features 543 and the horizontal comparison features 542 respectively, and output the anomaly degree. At the same time, the trained autoencoder model 554 and Kmeans clustering model 555 are used to perform anomaly detection on the detailed features 541 and output the anomaly degree.

[0094] Then, the anomaly degrees output by each model are first standardized and then normalized. Among them, the standardization process may include, for example, Z-score processing. After normalization, the anomaly degrees output by all models for all users for anomaly detection are between 0 and 1. Specifically, each user has 8 anomaly degrees between 0 and 1, which are the anomaly degrees output by three models, namely the Isolation Forest, HBOS, and Copod models based on the horizontal comparison feature 542, the anomaly degrees of the Isolation Forest, HBOS, and Copod models based on the longitudinal comparison feature 543, and the anomaly degrees of two models, namely the autoencoder and Kmeans clustering models based on the detailed time series features 541.

[0095] Then, the 8 anomaly degrees output by the models and normalized can be added up to obtain the final anomaly degree of the user. Next, the users can be sorted according to the anomaly degree from large to small, and the top n users can be determined as abnormal users. Among them, n can be set according to actual needs. For example, the top 5% of users can be output as abnormal users.

[0096] The following will be combined with Figure 6 to describe the apparatus for determining the anomaly degree of an object provided by the present disclosure.

[0097] Figure 6 A block diagram of an apparatus for determining the anomaly degree of an object according to an embodiment of the present disclosure is schematically shown.

[0098] As Figure 6 shown, the apparatus 600 for determining the anomaly degree of an object includes a first determination module 610, a second determination module 620, a third determination module 630, and a detection module 640.

[0099] The first determination module 610 is configured to determine detailed features according to the web access data of the target object.

[0100] The second determination module 620 is configured to determine longitudinal comparison features according to the web access data of the target object and the web access data of other objects.

[0101] A third determination module 630, configured to determine a horizontal comparison feature according to the web access data of the target object in the current time period and the web access data of the target object in the previous time period.

[0102] A detection module 640, configured to perform anomaly detection on the vertical comparison feature, the horizontal comparison feature, and the detail feature to obtain the target anomaly degree of the target object.

[0103] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0104] Figure 7 The block diagram of an example electronic device 700 that can be used to implement the embodiments of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0105] As Figure 7 shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 702 or the computer program loaded from the storage unit 708 into the random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. The input / output (I / O) interface 705 is also connected to the bus 704.

[0106] A plurality of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disc, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0107] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 executes the various methods and processes described above, such as the method for determining the object abnormality degree. For example, in some embodiments, the method for determining the object abnormality degree can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the method for determining the object abnormality degree described above can be executed. Alternatively, in other embodiments, the computing unit 701 can be configured to execute the method for determining the object abnormality degree in any other suitable manner (e.g., by means of firmware).

[0108] Various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), application-specific standard products (ASSP), system-on-a-chip systems (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs, which can be executed and / or interpreted on a programmable system including at least one programmable processor, and the programmable processor can be a special or general programmable processor, receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to the processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing devices, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program codes can be executed entirely on the machine, partially on the machine, executed partially on the machine and partially on a remote machine as an independent software package, or executed entirely on a remote machine or server.

[0110] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic, speech, or tactile input).

[0112] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of a communication network include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0113] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0114] It should be understood that the various forms of processes shown above can be used, with steps reordered, added or deleted. For example, the steps described in the present disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and no limitation is imposed herein.

[0115] The above specific embodiments do not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub - combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A method for determining the abnormality degree of an object, including: determining detailed features according to the web access data of the target object; determining longitudinal comparison features according to the web access data of the target object and the web access data of other objects; determining horizontal comparison features according to the web access data of the target object in the current time period and the web access data of the target object in the previous time period; and performing anomaly detection on the longitudinal comparison features, the horizontal comparison features and the detailed features to obtain the target abnormality degree of the target object; wherein, the web access data includes user agent information, long-tail web pages, and the difference between the number of user agent information during non-working hours and the number of user agent information during working hours per unit time; the method further includes: determining the page view volume corresponding to each of multiple user agent volumes, and encoding the multiple user agent information in descending order of the page view volume to obtain the encoded user agent information; responding to the number of users in the reference unit time being less than the predetermined number of users, determining that the reference unit time is a non-working time period; determining at least one web page with the number of users less than the user volume threshold or the number of accesses less than the access count threshold as a long-tail web page, and performing a merging process on at least one of the long-tail web pages to obtain the merged long-tail web page.

2. The method according to claim 1, wherein, the performing anomaly detection on the longitudinal comparison features, the horizontal comparison features and the detailed features to obtain the abnormality degree of the target object includes: performing anomaly detection on the longitudinal comparison features to obtain a first detection result; performing anomaly detection on the horizontal comparison features to obtain a second detection result; performing anomaly detection on the detailed features to obtain a third detection result; and determining the target abnormality degree of the target object according to the first detection result, the second detection result and the third detection result.

3. The method according to claim 2, wherein, the performing anomaly detection on the longitudinal comparison features to obtain a first detection result includes: inputting the longitudinal comparison features into an isolation forest model to determine the first abnormality degree of the target object; inputting the longitudinal comparison features into an HBOS model to determine the second abnormality degree of the target object; inputting the longitudinal comparison features into a Copod model to determine the third abnormality degree of the target object; and determining the first abnormality degree, the second abnormality degree and the third abnormality degree as the first detection result.

4. The method according to claim 3, wherein, the performing anomaly detection on the horizontal comparison features to obtain a second detection result includes: inputting the horizontal comparison features into an isolation forest model to determine the fourth abnormality degree of the target object; inputting the horizontal comparison features into an HBOS model to determine the fifth abnormality degree of the target object; inputting the horizontal comparison features into a Copod model to determine the sixth abnormality degree of the target object; and determining the fourth abnormality degree, the fifth abnormality degree and the sixth abnormality degree as the second detection result.

5. The method according to claim 4, wherein, Performing anomaly detection on the detailed features to obtain a third detection result includes: Inputting the detailed features into an autoencoder model to determine intermediate features and a seventh anomaly degree of the target object; Inputting the intermediate features into a clustering model to determine an eighth anomaly degree of the target object; and Determining the seventh anomaly degree and the eighth anomaly degree as the third detection result.

6. The method according to claim 5, wherein, Determining the target anomaly degree of the target object according to the first detection result, the second detection result, and the third detection result includes: Normalizing and standardizing the first anomaly degree, the second anomaly degree, the third anomaly degree, the fourth anomaly degree, the fifth anomaly degree, the sixth anomaly degree, the seventh anomaly degree, and the eighth anomaly degree respectively; and Adding the normalized and standardized first anomaly degree, second anomaly degree, third anomaly degree, fourth anomaly degree, fifth anomaly degree, sixth anomaly degree, seventh anomaly degree, and eighth anomaly degree to obtain the target anomaly degree.

7. The method according to claim 1, wherein, The web access data further includes at least one of the following: the number of user agent information used by the target object per unit time, the web pages accessed by the target object per unit time, the number of accesses to each web page, and the total number of failures to access each web page by the target object per unit time.

8. An apparatus for determining the anomaly degree of an object, including: A first determination module for determining detailed features according to the web access data of the target object; A second determination module for determining longitudinal comparison features according to the web access data of the target object and the web access data of other objects; A third determination module for determining horizontal comparison features according to the web access data of the target object in the current time period and the web access data of the target object in the previous time period; and A detection module for performing anomaly detection on the longitudinal comparison features, the horizontal comparison features, and the detailed features to obtain the target anomaly degree of the target object; wherein the web access data includes user agent information, long-tail web pages, and the difference between the number of user agent information during non-working hours and the number of user agent information during working hours per unit time; the apparatus further includes: A user agent information processing module for determining the page view volume corresponding to each of a plurality of user agent quantities, encoding the plurality of user agent information in descending order of the page view volume to obtain encoded user agent information; A non-working time period determination module for determining the reference unit time as a non-working time period in response to the number of users in the reference unit time being less than a predetermined number of users; A long-tail web page processing module for determining at least one web page with the number of users less than the user quantity threshold or the number of accesses less than the access quantity threshold as a long-tail web page, and performing a merging process on the at least one long-tail web page to obtain a merged long-tail web page.

9. An electronic device, including: At least one processor; and A memory communicatively connected to at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, the computer instructions are for causing a computer to execute the method of any one of claims 1-7.

11. A computer program product, comprising a computer program / instructions, wherein, when the computer program / instructions are executed by a processor, the steps of the method of any one of claims 1-7 are implemented.

Images