An apparatus and method for efficiently discovering user abnormal access behavior

Abstract

The application discloses a device and method for efficiently discovering abnormal access behavior of a user, wherein the device comprises an abnormal behavior control, which comprises a control identification module, a basic data reading module, a database module and a basic data cache module; the control identification module is unidirectionally connected with the basic data reading module; the basic data reading module is unidirectionally connected with the database module; and the basic data reading module is unidirectionally connected with the basic data cache module 104. The basic data can be updated in real time; based on control extension, the newly-added control does not need to modify the program and is easy to extend; the developer only needs to focus on the rules and implementation of the newly-added abnormal behavior and does not need to focus on the existing control; the newly-added control tests the function of the existing control; after the active resource flow data passes through various preset abnormal behavior controls, the type of all abnormal access behaviors and other information concerned are obtained, and the efficiency of discovering abnormal access behaviors is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of Internet data streams, and in particular to an apparatus and method for efficiently detecting abnormal user access behavior. Background Technology

[0002] The current standard procedure for detecting abnormal user access behavior is as follows:

[0003] 1. Data collection, mainly including:

[0004] The system collects data via protocols such as FTP / SFTP, WebService, and JDBC / ODBC. The collected data includes access logs, mining application IP address pools, internet user access number information, a database of fraudulent domain names, and basic data from data centers / ISPs. Access logs are written to the big data platform's distributed file system, HDFS. Other data is written to a relational database.

[0005] IDC's Chinese name: Internet Data Center

[0006] ISP (Internet Service Provider) in Chinese: Internet Access Service

[0007] 2. Data analysis and processing, mainly including:

[0008] (1) Group the full access logs by multiple dimensions such as domain name, destination IP, and destination port. Analyze the active resource data of each data center, such as the number of times a source IP accesses a domain name and the number of times a destination IP is accessed within a day. This data serves as the source data for various user behavior analyses.

[0009] (2) Mining behavior analysis: By matching the active resource destination IP with the mining IP address pool IP, the user's mining behavior is obtained.

[0010] (3) Analysis of the domain name database involved in fraud: By matching active resource domain names with the domain name database involved in fraud and IDCISP basic data, the behavior of users visiting websites involved in fraud can be obtained.

[0011] In summary: Data is read from HDFS, and related calculations are performed. These calculations are correlated with underlying data. For example, by correlating with the IP address pool data for mining, abnormal mining behavior logs can be obtained. Similarly, by correlating with the database of fraudulent domain names, abnormal fraudulent behavior logs can be retrieved. This underlying data is stored in a relational database. To improve performance during data calculations, the underlying data is typically queried in batches, then cached, and the data is correlated to avoid querying the database for each record, which would put significant performance pressure on the database.

[0012] Whenever a new abnormal behavior needs to be identified, the program code is modified to identify the new abnormal behavior.

[0013] Common solution problem:

[0014] 1. The basic data is all static, making it impossible to detect real-time updates. Because the basic data is periodically queried from the database and cached, it's impossible to predict when the basic data will be updated; therefore, data can only be loaded from the database periodically.

[0015] 2. If any new abnormal behaviors are found, the code implementation needs to be modified, and the corresponding program logic needs to be changed. Modifying the code is prone to errors. After the modification is completed, regression testing is usually required to verify all functionalities of the system.

[0016] 3. It places high demands on developers, requiring them to understand both the business logic and the code of the entire process. Summary of the Invention

[0017] To overcome the shortcomings of existing technologies, this invention provides an efficient device and method for detecting abnormal user access behavior. Based on active resource data, it identifies abnormal behaviors such as illegal mining and fraudulent domain names through user behavior controls. The basic data of the corresponding behavior controls can be updated in real time, enabling real-time interception of abnormal behaviors. New abnormal behavior data can be dynamically expanded through templates in the behavior controls.

[0018] To achieve the above objectives, the present invention adopts the following technical solution:

[0019] In one embodiment of the present invention, an apparatus for efficiently detecting abnormal user access behavior is proposed, comprising: a control identification module, a basic data reading module, a database module, and a basic data caching module. The control identification module is unidirectionally connected to the basic data reading module, the basic data reading module is unidirectionally connected to the database module, and the basic data reading module is unidirectionally connected to the basic data caching module.

[0020] Furthermore, the control and identification module controls the database module to update its status. If the database module needs to be updated, it initiates the basic data reading module and updates the basic data cache module.

[0021] Furthermore, the method by which the control and identification module controls the identification database module to update its status includes:

[0022] S1011. Specify the flag statement to determine whether the database module is valid, for example: select age from t_table where id=3;

[0023] S1012, The basic data cache module reads the results of the query statement;

[0024] S1013. The control and identification module queries the tag statement every 1 second and compares the result of the tag statement query with the result in the basic data cache module.

[0025] If the results of S1014 and S1013 are the same, then there is no need to update the basic data cache module;

[0026] If the results of S1015 and S1013 are inconsistent, it is necessary to initiate an operation to update the basic data cache module.

[0027] The configuration file structure is as follows:

[0028]

[0029]

[0030]

[0031] Furthermore, the basic data reading module reads data from the database module based on configuration information, and the method of reading data needs to be dynamically expanded.

[0032] Furthermore, the configuration information is the configuration information for the abnormal behavior control, including:

[0033] Define the control name; define the control's key and value fields; define the connection information, database, and table for the underlying data; specify whether the underlying database table has been updated.

[0034] The main contents include:

[0035] name: Defines the name of the control.

[0036] Structure: Defines the control's key and value fields, using the key field to associate with fields in the active resource log. The value field returns the result of the corresponding field value in the underlying data.

[0037] source: Defines the connection information, database, tables, etc. of the underlying data.

[0038] lifetime: Specifies whether the underlying database table has been updated.

[0039] Example as follows:

[0040]

[0041]

[0042]

[0043] Furthermore, the basic data caching module updates the basic data cache based on the read data.

[0044] In one embodiment of the present invention, a method for efficiently detecting abnormal user access behavior is also proposed, the method comprising:

[0045] S01. Use Spark to read data from HDFS, and load the read data into a distributed dataset;

[0046] S02. The read data is used as input to the abnormal behavior control 1, and the abnormal behavior control 1 processes the data.

[0047] S03. Abnormal behavior control 2 takes the distributed dataset processed by abnormal behavior control 1 as input, processes it, generates a new distributed dataset 2, and so on, until the last abnormal behavior control is processed.

[0048] S04. After the last abnormal behavior control is processed, the generated distributed dataset is written to the target storage.

[0049] Furthermore, each of the abnormal behavior controls has its own implementation class, which integrates the data source parent class to implement initialization configuration, configuration checking, preprocessing, and generation of distributed data.

[0050] Define control configuration

[0051] The configuration information includes: the name of the control to be used, which must be a control that has already been defined.

[0052] The source key field that needs to be associated is used to associate with the key field in the control definition, or the value field in the control definition.

[0053] Example as follows:

[0054]

[0055] Furthermore, the initialization configuration initConfig reads the configuration file and adds the configuration properties to the control.

[0056] Furthermore, the checkConfig function checks whether the configuration in the configuration file is valid, and returns a centralized error message for all abnormal configurations.

[0057] Furthermore, the preprocessing refers to the preprocessing steps used in data processing.

[0058] Furthermore, the generation of the distributed dataset getDataSet is the most important method. Based on the previous methods, it converts the data source into a Spark distributed dataset containing schema information according to the configuration properties in the configuration file for subsequent data processing.

[0059] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned method for efficiently detecting abnormal user access behavior.

[0060] In one embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that performs a method for efficiently detecting abnormal user access behavior.

[0061] Beneficial effects:

[0062] 1. The basic data of this invention can be updated in real time;

[0063] 2. This invention is based on control extension; adding new controls does not require modifying the program, making it easy to extend.

[0064] 3. Developers only need to focus on the rules and implementation of the newly added abnormal behaviors, without needing to worry about existing controls. New controls are used to test the functionality of existing controls.

[0065] 4. After passing through various preset abnormal behavior controls, the active resource traffic data yields the types of all abnormal access behaviors and other relevant information, significantly improving the efficiency of abnormal access behavior detection. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of a device for efficiently detecting abnormal user access behavior according to an embodiment of the present invention;

[0067] Figure 2 This is a schematic flowchart of a method for efficiently detecting abnormal user access behavior according to an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of a computer device structure according to an embodiment of the present invention. Detailed Implementation

[0069] The principles and spirit of the present invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of the disclosure to those skilled in the art.

[0070] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0071] According to embodiments of the present invention, an apparatus and method for efficiently detecting abnormal user access behavior are proposed. Based on active resource data, abnormal behaviors such as illegal mining and fraudulent domain names are identified through user behavior controls. The basic data of the corresponding behavior controls can be updated in real time, enabling real-time interception of abnormal behaviors. New abnormal behavior data can be dynamically expanded through templates of the behavior controls.

[0072] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0073] like Figure 1 As shown, the device for efficiently detecting abnormal user access behavior proposed in this invention includes:

[0074] It includes: a control identification module 101, a basic data reading module 102, a database module 103, and a basic data caching module 104. The control identification module 101 is unidirectionally connected to the basic data reading module 102, the basic data reading module 102 is unidirectionally connected to the database module 103, and the basic data reading module 102 is unidirectionally connected to the basic data caching module 104.

[0075] The control and identification module 101 controls the database module 103 to update its status. If the database module 103 needs to be updated, it initiates the basic data reading module 102 and updates the basic data cache module 104.

[0076] The method for updating the status of the control identification module 101 and the control identification database module 103 includes:

[0077] S1011. Specify the flag statement to determine whether database module 103 is valid, for example: select age from t_table where id=3;

[0078] S1012, Basic data cache module 104 reads the results of the query statement;

[0079] S1013. Every second, the control and identification module 101 queries the tag statement and compares the result of the tag statement query with the result in the basic data cache module 104.

[0080] If the results of S1014 and S1013 are the same, then there is no need to update the basic data cache module 104.

[0081] If the results of S1015 and S1013 are inconsistent, it is necessary to initiate an operation to update the basic data cache module 104.

[0082] The basic data cache module 104 updates the basic data cache based on the read data.

[0083] The configuration file structure is as follows:

[0084]

[0085]

[0086]

[0087] The basic data reading module 102 reads data from the database module 103 based on configuration information. The method of reading data needs to be dynamically expanded.

[0088] The configuration information refers to the configuration information of the abnormal behavior control, including:

[0089] Define the control name; define the control's key and value fields; define the connection information, database, and table for the underlying data; specify whether the underlying database table has been updated.

[0090] The main contents include:

[0091] name: Defines the name of the control.

[0092] Structure: Defines the control's key and value fields, using the key field to associate with fields in the active resource log. The value field returns the result of the corresponding field value in the underlying data.

[0093] source: Defines the connection information, database, tables, etc. of the underlying data.

[0094] lifetime: Specifies whether the underlying database table has been updated.

[0095] To provide a clearer explanation of the above-described method for efficiently detecting abnormal user access behavior, a specific embodiment is described below. However, it is important to note that this embodiment is merely for illustrative purposes and does not constitute an undue limitation of the invention. The example is as follows:

[0096]

[0097]

[0098] It should be noted that although several modules of a device for efficiently detecting abnormal user access behavior have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in a single module. Conversely, the features and functions of a single module described above can be further divided and embodied by multiple modules.

[0099] Based on the same inventive concept, this invention also proposes a method for efficiently detecting abnormal user access behavior. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0100] like Figure 2 As shown, the method includes:

[0101] S01. Use Spark to read data from HDFS, and load the read data into a distributed dataset;

[0102] S02. The read data is used as input to the abnormal behavior control 1, and the abnormal behavior control 1 processes the data.

[0103] S03. Abnormal behavior control 2 takes the distributed dataset processed by abnormal behavior control 1 as input, processes it, generates a new distributed dataset 2, and so on, until the last abnormal behavior control is processed.

[0104] S04. After the last abnormal behavior control is processed, the generated distributed dataset is written to the target storage.

[0105] Furthermore, each of the abnormal behavior controls has its own implementation class, which integrates the data source parent class to implement initialization configuration, configuration checking, preprocessing, and generation of distributed data.

[0106] Define control configuration

[0107] The configuration information includes: the name of the control to be used, which must be a control that has already been defined.

[0108] The source key field that needs to be associated is used to associate with the key field in the control definition, or the value field in the control definition.

[0109] Initialization configuration initConfig: Reads the configuration file and adds the configuration properties to the control.

[0110] Furthermore, the checkConfig function checks whether the configuration in the configuration file is valid, and returns a centralized error message for all abnormal configurations.

[0111] Furthermore, the preprocessing refers to the preprocessing steps used in data processing.

[0112] Furthermore, the generation of the distributed dataset getDataSet is the most important method. Based on the previous methods, it converts the data source into a Spark distributed dataset containing schema information according to the configuration properties in the configuration file for subsequent data processing.

[0113] Examples of fraudulent domain name analysis and user cryptocurrency mining behavior:

[0114] The configuration is as follows:

[0115]

[0116]

[0117]

[0118]

[0119]

[0120] illustrate:

[0121] (1) Assume the active resource log data contains five fields: destip, houseid, domain, count, and eventtime. An example of this data is shown in Table 1 below:

[0122]

[0123] Table 1

[0124] (2) Assuming the basic data for analyzing fraudulent domain names includes three fields, the data example is as follows:

[0125] Table 2:

[0126] domain level illegal_domain_flag a.com Whole Domain Black Financial fraud b.com Link Black Gambling scam z.com Link Black Online fraud

[0127] Table 2

[0128] (3) Assume that the basic data of mining behavior includes three fields, and the data example is shown in Table 3 below:

[0129] IP Countries or regions illegal_mining_flag 10.0.0.2 China (Hong Kong) Storage mining 10.0.0.3 USA Mining with computing power 20.0.0.1 U.K. Mining with computing power

[0130] Table 3

[0131] (4) The above configuration defines two controls: illegal_domain (fraudulent domain analysis) and illegal_mining (mining behavior analysis).

[0132] (5) In the data flow module, the data first flows into illegal_domain (which can also be configured as illegal_mining), and then flows into illegal_mining.

[0133] 1) When calling the illegal_domain control, the source key field to be associated is specified as domain. That is, the domain field in the active resource is associated with the key field domain defined in illegal_domain to obtain data. The output results add two columns: level and illegal_domain_flag. Data that is not associated is set to empty.

[0134] The data processed by the illegal_domain control is shown in Table 4 below:

[0135]

[0136] Table 4

[0137] 2) When calling the illegal_mining control, the source key field to be associated is specified as destip. This means the destip field from the active resources is associated with the ip key field defined in illegal_mining to retrieve data. The output results will include two new columns: country and illegal_mining_flag. Data that is not associated will be left blank.

[0138] The data processed by the illegal_mining control is shown in Table 5 below:

[0139]

[0140] Table 5

[0141] (3) Based on the newly added violation markers and related fields, the log content was marked. During the data processing, the control definition module will determine whether the data has been updated at specified time intervals. If the data has been updated, the basic data will be updated immediately.

[0142] (4) When new exception behaviors need to be extended, only a new control needs to be added. No changes are required to the original code; only new code is added. It will not affect the existing controls.

[0143] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0144] Based on the aforementioned inventive concept, such as Figure 3 As shown, the present invention also proposes a computer device 300, including a memory 310, a processor 320, and a computer program 330 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 330, it implements the aforementioned method for efficiently detecting abnormal user access behavior.

[0145] Based on the aforementioned inventive concept, the present invention also proposes a computer-readable storage medium storing a computer program that executes the aforementioned method for efficiently detecting abnormal user access behavior.

[0146] The device and method proposed in this invention for efficiently detecting abnormal user access behavior can update basic data in real time; based on control extension, adding new controls does not require modifying the program, making it easy to extend; developers only need to focus on the rules and implementation of the newly added abnormal behavior, without needing to pay attention to existing controls; adding new controls tests the functionality of existing controls; active resource traffic data, after passing through various preset abnormal behavior controls, yields all types of abnormal access behavior and other information of interest, significantly improving the efficiency of abnormal access behavior detection.

[0147] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0148] Regarding the limitation of the scope of protection of this invention, those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solution of this invention are still within the scope of protection of this invention.

Claims

1. A device for efficiently detecting abnormal user access behavior, characterized in that, The device is an abnormal behavior control, including: a control identification module, a basic data reading module, a database module and a basic data caching module. The control identification module is unidirectionally connected to the basic data reading module, the basic data reading module is unidirectionally connected to the database module, and the basic data reading module is unidirectionally connected to the basic data caching module. The control and identification module controls the database module to update its status. If the database module needs to be updated, it initiates the basic data reading module and updates the basic data cache module. The method for the control and identification module to control the update status of the identification database module includes: S1011, Specify the flag statement to determine whether the database module is valid; S1012, The basic data cache module reads the results of the query statement; S1013. The control and identification module queries the tag statement every 1 second and compares the result of the tag statement query with the result in the basic data cache module. If the results of S1014 and S1013 are the same, then there is no need to update the basic data cache module; If the results of S1015 and S1013 are inconsistent, it is necessary to initiate an operation to update the basic data cache module; The basic data reading module reads data from the database module based on configuration information. The method of reading data needs to be dynamically expanded. The configuration information refers to the configuration information of the abnormal behavior control, including: Define the control name; Define the key and value fields of the control; Define the connection information, database, and tables for the basic data; Specify whether the underlying database table has been updated; The basic data caching module updates the basic data cache based on the read data.

2. A method implemented using the apparatus for efficiently detecting abnormal user access behavior as described in claim 1, characterized in that, The method includes: S01. Use Spark to read data from HDFS, and load the read data into a distributed dataset; S02. The read data is used as input to the abnormal behavior control 1, and the abnormal behavior control 1 processes the data. S03. Abnormal behavior control 2 takes the distributed dataset processed by abnormal behavior control 1 as input, processes it, generates a new distributed dataset 2, and so on, until the last abnormal behavior control is processed. S04. After the last abnormal behavior control is processed, the generated distributed dataset is written to the target storage.

3. The method according to claim 2, characterized in that, Each of the abnormal behavior controls has its own implementation class, which integrates the data source parent class to implement initialization configuration, configuration checking, preprocessing, and generation of distributed data.

4. The method according to claim 3, characterized in that, The initialization configuration involves reading the configuration file and adding the configuration properties to the control.

5. The method according to claim 3, characterized in that, The configuration check involves verifying the validity of the configuration file and returning error messages for any abnormal configurations.

6. The method according to claim 3, characterized in that, The preprocessing refers to the preprocessing of data.

7. The method according to claim 3, characterized in that, The process of generating a distributed dataset involves converting the data source into a distributed data collection for Spark, including schema information, based on the configuration attributes in the configuration file.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 2-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the method according to any one of claims 2-7.

Images