Real-time data warehouse construction, anomaly detection methods, devices, equipment and products

Abstract

The present invention provides a real-time data warehouse construction, anomaly detection method, device, equipment and product, the method includes: pre-acquiring data information of a target user, and performing hierarchical processing on the data information, and storing the hierarchical processing results in a corresponding storage medium; the anomaly detection method includes: acquiring log data and event configuration data generated by the target user within a preset time period, determining behavior data according to the log data, determining the anomaly detection type according to the event configuration data, inputting the log data into a real-time data warehouse to obtain anomaly detection benchmark data corresponding to the anomaly detection type updated according to the log data, and determining whether the target user has an abnormal event according to the behavior data of the target user and the updated anomaly detection benchmark data. The anomaly detection method based on a real-time data warehouse provided by the present invention can ensure the real-time and accuracy of anomaly detection, improve the speed of anomaly detection, and enhance user experience.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a real-time data warehouse construction, anomaly detection method, device, equipment and product. Background Art

[0002] With the continuous development of big data technology, it is becoming increasingly important to achieve accurate and timely analysis of big data.

[0003] At present, there are certain security issues in many big data applications. Many hackers will steal user information to steal big data, which brings certain security issues. Therefore, it is necessary to set up an anomaly detection system to detect and process user operation behaviors to ensure the security of big data. In the prior art, the anomaly detection system identifies the existing risks by combining machine learning and database analysis. It is necessary to compare and analyze the acquired log data with the data stored in the database, but the database only stores historical data, which makes this processing method unable to meet the real-time requirements and the user experience is poor. Summary of the invention

[0004] The present invention provides a real-time data warehouse construction, anomaly detection method, device, equipment and product to solve the technical problems of low real-time anomaly detection and poor user experience in the prior art, so as to achieve the purpose of improving the real-time performance of anomaly detection and enhancing user experience.

[0005] In a first aspect, the present invention provides a real-time data warehouse construction method, comprising:

[0006] Obtain target user data information in advance;

[0007] The data information of the target user is processed in layers, and the results of the layer processing are stored in a corresponding storage medium.

[0008] Furthermore, according to the real-time data warehouse construction method provided by the present invention, the real-time data warehouse includes a detailed data layer, a light summary layer and a public summary layer.

[0009] Accordingly, the layered processing of the target user's data information includes:

[0010] Counting the log data corresponding to different subject types of the target user, and storing the log data under different subject types in the corresponding user subject list in the detailed data layer;

[0011] Counting the behavioral baseline data of target users in multiple dimensions, and storing different behavioral baseline data into the corresponding user behavior list in the light summary layer according to different preset time periods;

[0012] The indicator data of the target user are counted, and the obtained different indicator data are stored in the corresponding user indicator wide table.

[0013] Further, according to the real-time data warehouse construction method provided by the present invention, the real-time data warehouse at least includes the following storage media: a message queue, a third-party database and a non-relational database,

[0014] Accordingly, storing the hierarchical processing result in a corresponding storage medium includes:

[0015] storing the detailed data obtained by processing the detailed data layer into the non-relational database;

[0016] Storing the user behavior baseline data obtained by the light aggregation layer processing into the message queue;

[0017] The indicator data processed by the public aggregation layer is stored in the third-party database.

[0018] In a second aspect, the present invention further provides an anomaly detection method, comprising:

[0019] Obtain the log data and event configuration data generated by the target user within a preset time period;

[0020] Determining the behavior data of the target user according to the log data;

[0021] determining an anomaly detection type according to the event configuration data;

[0022] Inputting the log data into a real-time data warehouse to obtain anomaly detection benchmark data corresponding to the anomaly detection type updated according to the log data;

[0023] It is determined whether there is an abnormal event for the target user according to the behavior data of the target user and the updated abnormality detection benchmark data corresponding to the abnormality detection type.

[0024] Further, according to the anomaly detection method provided by the present invention, determining the behavior data of the target user according to the log data includes:

[0025] Storing the log data in a message queue;

[0026] In the message queue, the log data is distributed as a message to obtain the behavior data of the target user.

[0027] Further, according to the anomaly detection method provided by the present invention, the updating of anomaly detection benchmark data corresponding to the anomaly detection type according to the log data includes:

[0028] Transmitting the log data to a real-time data warehouse, so that the real-time data warehouse calculates anomaly detection benchmark data corresponding to the anomaly detection type according to the log data and the historical log data of the target user, and obtains updated anomaly detection benchmark data corresponding to the anomaly detection type;

[0029] The updated anomaly detection benchmark data corresponding to the anomaly detection type is received from the real-time data warehouse.

[0030] Further, according to the anomaly detection method provided by the present invention, the updated anomaly detection benchmark data corresponding to the anomaly detection type includes: updated target user behavior baseline data corresponding to the anomaly detection type, and an anomaly probability threshold;

[0031] Accordingly, determining whether there is an abnormal event for the target user according to the behavior data of the target user and the updated abnormality detection benchmark data corresponding to the abnormality detection type includes:

[0032] Determining whether the target user has updated target user behavior baseline data corresponding to the anomaly detection type;

[0033] In a case where the target user does not have updated target user behavior baseline data corresponding to the abnormality detection type, it is determined that the target user does not have an abnormal access event.

[0034] Further, according to the anomaly detection method provided by the present invention, determining whether the target user has an abnormal event based on the behavior data of the target user and the updated anomaly detection benchmark data corresponding to the anomaly detection type further includes:

[0035] If the target user has updated target user behavior baseline data corresponding to the anomaly detection type, calculate the abnormal probability value of the target user according to the target user's behavior data and the updated anomaly detection benchmark data corresponding to the anomaly detection type;

[0036] When the abnormal probability value is less than or equal to a preset abnormal probability threshold, determining that the target user has an abnormal event related to the abnormal detection type;

[0037] When the abnormal probability value is greater than a preset abnormal probability threshold, it is determined that the target user does not have an abnormal event related to the abnormal detection type.

[0038] Further, according to the anomaly detection method provided by the present invention, the updated anomaly detection benchmark data corresponding to the anomaly detection type includes: updated target user behavior baseline data corresponding to the anomaly detection type;

[0039] Accordingly, the transmitting the log data to the real-time data warehouse so that the real-time data warehouse calculates the anomaly detection benchmark data corresponding to the anomaly detection type according to the log data and the historical log data of the target user, and obtains the updated anomaly detection benchmark data corresponding to the anomaly detection type, including:

[0040] Transmitting the log data to a real-time data warehouse;

[0041] In the real-time data warehouse, based on the historical log data of the target user and the received log data, the behavior baseline data of the target user in multiple dimensions is calculated;

[0042] According to the target user's behavior baseline data under the multiple dimensions, the target user's behavior baseline data corresponding to the anomaly detection type is determined, and updated target user's behavior baseline data corresponding to the anomaly detection type is obtained.

[0043] In a third aspect, the present invention further provides a real-time data warehouse construction device, comprising:

[0044] An acquisition module is used to obtain data information of target users in advance;

[0045] The processing module is used to perform hierarchical processing on the data information of the target user and store the hierarchical processing results in a corresponding storage medium.

[0046] In a fourth aspect, the present invention further provides an abnormality detection device, comprising:

[0047] The acquisition module is used to obtain the log data and event configuration data generated by the target user within a preset time period;

[0048] A first determination module, configured to determine the behavior data of the target user according to the log data;

[0049] A second determination module, configured to determine an anomaly detection type according to the event configuration data;

[0050] An updating module, used for inputting the log data into a real-time data warehouse to obtain anomaly detection benchmark data corresponding to the anomaly detection type updated according to the log data;

[0051] The third determination module is used to determine whether there is an abnormal event for the target user according to the behavior data of the target user and the updated abnormality detection benchmark data corresponding to the abnormality detection type.

[0052] Further, the first determining module is further used for:

[0053] Storing the log data in a message queue;

[0054] In the message queue, the log data is distributed as a message to obtain the behavior data of the target user.

[0055] In a fifth aspect, the present invention further provides an electronic device, comprising:

[0056] processor, memory and bus, wherein,

[0057] The processor and the memory communicate with each other via the bus;

[0058] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the steps of the real-time data warehouse construction method or the anomaly detection method as described in any of the above items.

[0059] In a sixth aspect, the present invention also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions enable a computer to execute the steps of the real-time data warehouse construction method or the anomaly detection method as described above.

[0060] In a seventh aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the real-time data warehouse construction method or the anomaly detection method as described in any one of the above items.

[0061] The present invention provides a real-time data warehouse construction, anomaly detection method, device, equipment and product. The real-time data warehouse construction method includes: obtaining the data information of the target user in advance, performing layered processing on the data information of the target user, and storing the layered processing results in the corresponding storage medium; the anomaly detection method includes: obtaining the log data and event configuration data generated by the target user within a preset time period, determining the behavior data of the target user according to the log data; determining the anomaly detection type according to the event configuration data; inputting the log data into the real-time data warehouse to obtain the anomaly detection benchmark data corresponding to the anomaly detection type updated according to the log data, and determining whether the target user has an abnormal event according to the behavior data of the target user and the updated anomaly detection benchmark data corresponding to the anomaly detection type. The anomaly detection method based on the real-time data warehouse provided by the present invention can ensure the real-time and accuracy of anomaly detection, improve the speed of anomaly detection, and enhance user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 It is a flow chart of a method for building a real-time data warehouse provided by the present invention;

[0064] Figure 2 It is a schematic diagram of the hierarchical structure of a real-time data warehouse provided by the present invention;

[0065] Figure 3 It is a structural diagram of a real-time data warehouse storage provided by the present invention;

[0066] Figure 4 It is a flowchart of an abnormality detection method provided by the present invention;

[0067] Figure 5 It is a schematic diagram of the overall process of an anomaly detection method provided by the present invention;

[0068] Figure 6 It is a structural schematic diagram of a real-time data warehouse construction device provided by the present invention;

[0069] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] Figure 1 A flow chart of the real-time data warehouse construction method provided by the present invention is shown in FIG. Figure 1 As shown, the real-time data warehouse construction method provided by the present invention comprises the following steps:

[0072] Step 101: Pre-acquire data information of target users.

[0073] In this embodiment, it is necessary to obtain in advance relevant data information of the target user, such as the number of times the target user logs in within a certain period of time, or the IP address through which the target user logs in, or the locations where the target user usually logs in, and other data information.

[0074] It should be noted that the different types of data information obtained need to be classified and processed, such as which ones belong to the detailed type of data and which ones belong to the indicator type of data. By classifying the different types of data, the obtained data information can be subsequently processed in layers. For details, see the following embodiments, which will not be introduced in detail here.

[0075] Step 102: performing layered processing on the target user's data information, and storing the layered processing result in a corresponding storage medium.

[0076] In this embodiment, the data information of the target user obtained in step 101 needs to be hierarchically processed, and then the hierarchical processing results are stored in the corresponding storage medium to form a real-time data warehouse, and the constructed real-time data warehouse is used for anomaly detection.

[0077] It should be noted that the hierarchical processing may specifically be statistical summary processing of the detailed data, or processing according to the classification of each indicator data. The details can be seen in the following embodiments, which will not be described in detail here.

[0078] It should be noted that the storage medium refers to a carrier for storing data. In this embodiment, the storage medium can be Hive, HBase, a third-party database, a message queue, etc., among which Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation, and loading. This is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. The Hive data warehouse tool can map structured data files to a database table, and provide SQL query functions, which can convert SQL statements into MapReduce tasks for execution.

[0079] According to the real-time data warehouse construction method provided by the present invention, the data information of the target user is acquired in advance, and then the data information is layered and processed, and each layered processing result is stored in a corresponding storage medium to generate a real-time data warehouse. This can provide data information to the user in real time and provide data support for subsequent anomaly detection, thereby indirectly improving the processing efficiency of anomaly detection.

[0080] In another embodiment of the present invention, as shown in 2, the real-time data warehouse includes a detailed data layer, a light summary layer and a public summary layer;

[0081] Accordingly, the layered processing of the target user's data information includes:

[0082] Counting the log data corresponding to different subject types of the target user, and storing the log data under different subject types in the corresponding user subject list in the detailed data layer;

[0083] Counting the behavioral baseline data of target users in multiple dimensions, and storing different behavioral baseline data into the corresponding user behavior list in the light summary layer according to different preset time periods;

[0084] The indicator data of the target user are counted, and the obtained different indicator data are stored in the corresponding user indicator wide table.

[0085] In this embodiment, the constructed real-time data warehouse belongs to a multi-dimensional layered mode, which is specifically divided into three layers, namely, a detailed data layer, a light summary layer, and a public summary layer. Among them, the detailed data layer is used to count the corresponding log data under different topic types of the target user, and store the log data in the corresponding user topic list. The topic list can be a user login topic list or a user access topic list. The log data to be counted can be log data of the target user's most recent N access records, or log data of the target user's hourly access traffic, etc., which can be set according to the actual needs of the user and are not specifically limited here.

[0086] In this embodiment, the light aggregation layer is used to count the behavioral baseline data of target users in multiple dimensions, and store different behavioral baseline data in corresponding user behavior lists according to different preset times. For example, in the device dimension, the behavioral baseline data of the target user of the current device is counted, and the behavioral baseline data includes the person, location and time of login, and the obtained behavioral baseline data is stored in the corresponding user behavior list.

[0087] In this embodiment, the public aggregation layer is used to count the target user's indicator data, and store the obtained different indicator data in the corresponding user indicator wide table. The indicator data can be the number of operation behaviors of each target user in each time period, the number of operations of each account in all cities, and the average number of operations of each target user in a certain time period, and then stored in the user indicator wide table.

[0088] It should be noted that, in this embodiment, through effective data stratification processing, the data originally coupled in the data analysis module task is pre-processed, split out and stored in the real-time data warehouse for stream processing. For example, the target user's visits are counted every hour, day or week at a scheduled time, and then stream processing SQL is used to process according to the time window, and the processing results are output to the database for use. If second-level data is to be counted, the output results can be stored in Redis, which can remove the scheduled submission of data analysis module tasks in the crontab part and reduce the number of pending scheduling resources.

[0089] According to the real-time data warehouse construction method provided by the present invention, the constructed real-time data warehouse has a hierarchical structure, and the acquired data is processed in a hierarchical manner through the set multiple layers, which can improve the efficiency of data processing and ensure the real-time performance of anomaly detection.

[0090] In another embodiment of the present invention, Figure 3 As shown, the real-time data warehouse includes at least the following storage media: a message queue, a third-party database, and a non-relational database.

[0091] Accordingly, storing the hierarchical processing result in a corresponding storage medium includes:

[0092] storing the detailed data obtained by processing the detailed data layer into the non-relational database;

[0093] Storing the user behavior baseline data obtained by the light aggregation layer processing into the message queue;

[0094] The indicator data processed by the public aggregation layer is stored in the third-party database.

[0095] In this embodiment, the real-time data warehouse also includes a data operation layer, a data application layer, and a data warehouse layer, which are used to store different types of data. Different storage media are selected for different data layers. In this embodiment, the non-relational database (Redis) in the data operation layer is used to store detailed data, the message queue in the data operation layer is used to store the user behavior baseline data processed by the light summary layer, and the data warehouse layer is used to store the detailed data processed by the detailed data layer. The third-party database in the data warehouse layer is used to store the indicator summary data processed by the public summary layer.

[0096] It should be noted that real-time data is stored in message queues, and historical data is stored in offline queues, such as Hive. In addition, common indicator summary data is directly placed in third-party databases. For example, data required for model training is stored in file management systems, while data with more dimensions and larger write and update times are placed in HBase. If the query requirements are relatively high, such as the current device main body management data, it is directly stored in Redis.

[0097] It should be noted that there may be a large amount of data query when displaying data on the page. It is necessary to select index file ES to store data to improve query efficiency. Redis should be selected for trust level data and basic information of target users that need to be called frequently.

[0098] In this embodiment, the real-time data warehouse is mainly based on data collection tools. For example, the data collection tool writes the original data into a data channel of the message queue type, then aggregates the data through stream processing, and finally writes the collected and processed data into a storage system similar to Mysql, HBase, and Redis, which can provide minute-level or even second-level query solutions to the outside world. In addition, the target user behavior baseline data can be calculated as a scheduled task, and after the calculation is completed, the updated target user behavior baseline data can be pulled for caching, which can provide resource utilization.

[0099] According to the real-time data warehouse construction method provided by the present invention, by storing data after different layer processing in corresponding storage media, the data acquisition rate can be guaranteed and the efficiency of anomaly detection can be improved.

[0100] In another embodiment of the present invention, Figure 4 As shown, the anomaly detection method provided by the present invention specifically includes the following steps:

[0101] Step 401: Obtain log data and event configuration data generated by a target user within a preset time period.

[0102] In this embodiment, it is necessary to obtain the log data and event configuration data generated by the target user within a preset time period, wherein the log data refers to the data stored in the log file, usually ending with *.log, and the log file is a record file or file collection used to record system operation events.

[0103] It should be noted that event configuration data refers to parameters used to determine whether an abnormal event exists, including trigger time interval, abnormal probability type, and analysis time interval, wherein the abnormal probability type includes an abnormal probability threshold and an abnormal number threshold. The specific contents can be set according to the actual needs of the user.

[0104] It should be noted that the method for obtaining log data and event configuration data adopts a more mature acquisition method in the prior art, such as collecting log data based on the HTTP protocol RESTful method, which can be specifically set according to the actual needs of the user and is not specifically limited here.

[0105] Step 402: Determine the behavior data of the target user according to the log data.

[0106] In this embodiment, it is necessary to determine the behavior data of the target user based on the obtained log data of the target user. The specific implementation method is to send the obtained log data to the socket interface through the TCP / UDP protocol, and then collect the data part in the valid Syslog log Json body, and store it in different message queues according to the data type, and determine the behavior information of the target user based on the messages in the message queue.

[0107] For example, when the message queue is determined to be kafka, the message obtained from the message queue kafka is written into the corresponding Topic, such as ida_log_recv, and then the message is further distributed to the user authentication type Topic (user_auth) and the application access Topic (app_access), and then the behavior information of the target user is analyzed and processed. It should be noted that Topic is a transmission medium between the message publisher (Pub) and the subscriber (Sub), which is used for sending and receiving messages.

[0108] Step 403: Determine the anomaly detection type according to the event configuration data.

[0109] In this embodiment, it is necessary to determine the anomaly detection type based on the event configuration data. The latest event configuration data of the target user obtained is compared and analyzed with the event detection benchmark data pre-stored in the event library to determine the anomaly detection type. In this embodiment, the anomaly detection type can be any one of the following detection types, such as time detection type, device type detection type, IP address detection type, geographic location detection type, etc.

[0110] Step 404: input the log data into a real-time data warehouse to obtain anomaly detection benchmark data corresponding to the anomaly detection type that is updated according to the log data.

[0111] In this embodiment, the obtained log data needs to be input into the real-time data warehouse to obtain the anomaly detection benchmark data corresponding to the anomaly detection type after being updated according to the log data. It is also necessary to store the log data in the message queue in the file system as source data for real-time updating of the target user's anomaly detection benchmark data. Among them, the anomaly detection benchmark data at least includes the target user's behavior baseline data and the anomaly probability threshold, and the real-time update of the anomaly detection benchmark data is achieved through the target user's latest log data.

[0112] It should be noted that in this embodiment, it is necessary to monitor the message queue through the broadcast stream. On the one hand, the real-time messages in the message queue are consumed, and then the log data is stored in the file system as the source data to update the target user behavior baseline data, and the latest event configuration data is also updated in real time to the global variables for global effectiveness; on the other hand, after the real-time messages in the message queue are consumed, the detection threads of each event are started according to the latest event configuration data, and the event rules are matched according to the latest event configuration data, and the abnormal events triggered by the detection and discovery are output in real time. Among them, the source data refers to the log data initially sent to the message queue; the consumption data refers to the data in the two topics user_auth and app_access in the message queue.

[0113] Step 405: Determine whether there is an abnormal event for the target user based on the behavior data of the target user and the updated abnormality detection benchmark data corresponding to the abnormality detection type.

[0114] In this embodiment, it is necessary to determine whether the target user has an abnormal event based on the target user's behavior data and the updated abnormal detection benchmark data corresponding to the abnormal detection type. Among them, the abnormal detection benchmark data can be a trigger time interval, and can also be an abnormal probability type, an analysis time interval or user behavior baseline data, wherein the abnormal probability type includes an abnormal probability threshold and an abnormal number threshold. The specific content included can be set according to the actual needs of the user.

[0115] For example, if the abnormal detection data of the access event is the abnormal number threshold, the event configuration data of the target user is determined to be X, and the target user has target user behavior baseline data within the preset time period, and the target user behavior baseline data is Y. When the event configuration data X is less than the target user behavior baseline data Y, that is, the number of target user behavior baselines is greater than the set event configuration data, it indicates that the target user has a large number of visits and often visits within the preset time period, and the access event of the target user is determined to be a normal event.

[0116] When the event configuration data X is greater than the target user behavior baseline data Y, that is, the number of target user behavior baselines is less than the set event configuration data, it indicates that the target user's access times are small and do not reach the set number of accesses within the preset time period, and the target user's access event is determined to be an abnormal event.

[0117] According to an anomaly detection method provided by the present invention, log data and event configuration data generated by a target user within a preset time period are obtained, the behavior data of the target user is determined based on the log data, the anomaly detection type is determined based on the event configuration data, and the log data is input into a real-time data warehouse to obtain anomaly detection benchmark data corresponding to the anomaly detection type updated according to the log data, and based on the behavior data of the target user and the updated anomaly detection benchmark data corresponding to the anomaly detection type, it is determined whether the target user has an abnormal event. The anomaly detection method provided by the present invention can ensure the real-time and accuracy of anomaly detection, improve the speed of anomaly detection, and enhance user experience.

[0118] In one embodiment of the present invention, determining the behavior data of the target user according to the log data includes:

[0119] Storing the log data in a message queue;

[0120] In the message queue, the log data is distributed as a message to obtain the behavior data of the target user.

[0121] In this embodiment, the log data needs to be stored in a corresponding message queue according to its type, and the log data needs to be distributed in the message queue, and the behavior data of the target user needs to be obtained according to the distribution result.

[0122] It should be noted that, in this embodiment, when it is determined that the message queue is kafka, the message obtained from the message queue kafka is written into the corresponding Topic, such as ida_log_recv, and then the message is further distributed to the user authentication type Topic (user_auth) and the application access Topic (app_access), and then analyzed and processed to obtain the behavior information of the target user.

[0123] According to the anomaly detection method provided by the present invention, the real-time acquisition of log data is stored in a message queue, and then the log data is distributed in the message queue to obtain the behavior data of the target user, thereby improving the real-time and accuracy of anomaly detection.

[0124] In another embodiment of the present invention, updating the anomaly detection benchmark data corresponding to the anomaly detection type according to the log data includes:

[0125] Transmitting the log data to a real-time data warehouse, so that the real-time data warehouse calculates anomaly detection benchmark data corresponding to the anomaly detection type according to the log data and the historical log data of the target user, and obtains updated anomaly detection benchmark data corresponding to the anomaly detection type;

[0126] The updated anomaly detection benchmark data corresponding to the anomaly detection type is received from the real-time data warehouse.

[0127] In this embodiment, the log data of the target user needs to be transmitted to a pre-constructed real-time data warehouse, and the anomaly detection benchmark data corresponding to the anomaly detection type is calculated in the real-time data warehouse based on the real-time log data and historical log data of the target user to obtain updated anomaly detection benchmark data, and the updated anomaly detection benchmark data corresponding to the anomaly detection type is transmitted from the real-time data warehouse for use in anomaly detection. Among them, the real-time data warehouse is pre-constructed based on the historical log data, and is used to update and generate the anomaly detection benchmark data of the target user in real time. It should be noted that in this embodiment, the calculation method used by the real-time data warehouse to calculate the anomaly detection benchmark data corresponding to the anomaly detection type is a more mature calculation method in the prior art, and is not specifically limited here.

[0128] According to the anomaly detection method provided by the present invention, the log data of the target user is input into the real-time data warehouse, the anomaly detection benchmark data corresponding to the anomaly detection type is calculated in the real-time data warehouse according to the latest log data and historical log data of the target user, and is output from the real-time data warehouse for use in subsequent anomaly detection, thereby improving the real-time performance of anomaly detection and enhancing the user experience.

[0129] In another embodiment of the present invention, the updated anomaly detection benchmark data corresponding to the anomaly detection type includes: updated target user behavior baseline data corresponding to the anomaly detection type, an anomaly probability threshold;

[0130] Accordingly, determining whether there is an abnormal event for the target user according to the behavior data of the target user and the updated abnormality detection benchmark data corresponding to the abnormality detection type includes:

[0131] Determining whether the target user has updated target user behavior baseline data corresponding to the anomaly detection type;

[0132] In a case where the target user does not have updated target user behavior baseline data corresponding to the abnormality detection type, it is determined that the target user does not have an abnormal event.

[0133] In this embodiment, it is necessary to determine whether the target user has updated target user behavior baseline data corresponding to the anomaly detection type. When the target user has updated target user behavior baseline data corresponding to the anomaly detection type, it is determined that the target user's abnormal probability value is 0, and the target user has no abnormal event. It should be noted that the target user behavior baseline data is obtained through multi-dimensional analysis in the real-time data warehouse, which can be seen in the following embodiment, and will not be introduced in detail here.

[0134] According to the anomaly detection method provided by the present invention, by judging whether there is target user behavior baseline data for the target user, when it is confirmed that there is target user behavior baseline data, it is directly determined that there is no abnormal event for the target user, thereby improving the real-time performance of anomaly detection and enhancing user experience.

[0135] In another embodiment of the present invention, determining whether the target user has an abnormal event based on the behavior data of the target user and the updated abnormality detection benchmark data corresponding to the abnormality detection type further includes:

[0136] If the target user has updated target user behavior baseline data corresponding to the anomaly detection type, calculate the abnormal probability value of the target user according to the target user's behavior data and the updated anomaly detection benchmark data corresponding to the anomaly detection type;

[0137] When the abnormal probability value is less than or equal to a preset abnormal probability threshold, determining that the target user has an abnormal event related to the abnormal detection type;

[0138] When the abnormal probability value is greater than a preset abnormal probability threshold, it is determined that the target user does not have an abnormal event related to the abnormal detection type.

[0139] In this embodiment, when the target user has updated target user behavior baseline data corresponding to the anomaly detection type, the target user's abnormal probability value is calculated based on the determined target user's behavior data and the updated anomaly detection baseline data corresponding to the anomaly detection type. The target user's behavior data is confirmed based on the generated log data.

[0140] In this embodiment, it is necessary to calculate the target user's behavior data and the target user's behavior baseline data in the updated anomaly detection benchmark data corresponding to the anomaly detection type to obtain the target user's abnormal probability value. When the anomaly detection type is the time detection type, the specific calculation formula is as follows:

[0141]

[0142] Among them, nij Indicates the behavior data of the target user within a preset time period. It represents the sum of the behavior data of the target user i in all time periods of the day.

[0143] In this embodiment, when the anomaly detection type is the time detection type, the target user behavior baseline is divided into 7 time periods according to the whole day, and the target user's operation behavior (login / access) data in each time period is counted, and the behavior data is the number of operation behaviors. The details are shown in the following Table 1.

[0144] Table 1

[0145]

[0146]

[0147] When the calculated abnormal probability value of the target user within the preset time period is less than or equal to the preset abnormal probability threshold, it is determined that the target user has an abnormal event related to the abnormal detection type. When the abnormal probability value is greater than the preset abnormal probability threshold, it is determined that the target user does not have an abnormal event related to the abnormal detection type. It should be noted that the judgment method in this embodiment is carried out when the target user has the target user behavior baseline data. If the target user does not have the target user behavior baseline data, the processing process in this embodiment will not be carried out.

[0148] For example, if the abnormal probability threshold of the target user is set to 0.5, the abnormal probability value of the target user obtained by calculation is 0.2, that is, the number of operations performed by the target user in the preset time period is less than the set number of operations, and the target user's operation behavior is abnormal, that is, there is an abnormal event corresponding to the abnormal detection type.

[0149] It should be noted that the anomaly detection types in this embodiment also include IP address detection type, device type detection type and geographic location detection type, and are not limited to time detection type. They can be set according to the actual needs of the user and are not specifically limited here.

[0150] According to the anomaly detection method provided by the present invention, when there is target user behavior baseline data for the target user, by comparing the calculated anomaly probability value with the preset anomaly probability threshold, it is determined whether the target user has an abnormal event. This can accurately detect whether the target user has an abnormal event, improve the processing efficiency of anomaly detection, and enhance the user experience.

[0151] In another embodiment of the present invention, the updated anomaly detection benchmark data corresponding to the anomaly detection type includes: updated target user behavior baseline data corresponding to the anomaly detection type;

[0152] Accordingly, the transmitting the log data to the real-time data warehouse so that the real-time data warehouse calculates the anomaly detection benchmark data corresponding to the anomaly detection type according to the log data and the historical log data of the target user, and obtains the updated anomaly detection benchmark data corresponding to the anomaly detection type, including:

[0153] Transmitting the log data to a real-time data warehouse;

[0154] In the real-time data warehouse, based on the historical log data of the target user and the received log data, the behavior baseline data of the target user in multiple dimensions is calculated;

[0155] According to the target user's behavior baseline data under the multiple dimensions, the target user's behavior baseline data corresponding to the anomaly detection type is determined, and updated target user's behavior baseline data corresponding to the anomaly detection type is obtained.

[0156] In this embodiment, it is necessary to calculate the target user's behavior baseline data in multiple dimensions in the real-time data warehouse, and then determine the target user's behavior baseline data corresponding to the anomaly detection type based on the target user's behavior baseline data in multiple dimensions, and obtain the updated target user's behavior baseline data corresponding to the anomaly detection type. It should be noted that the target user's behavior baseline data is obtained by analyzing the target user's behavior baseline data in multiple dimensions, and the multiple dimensions refer to the time dimension, geographic location dimension, IP address dimension, and device type dimension, and then the target user's behavior baseline data is obtained based on the target user's behavior baseline data in the dimension corresponding to the anomaly detection type.

[0157] It should be noted that when the target user behavior baseline data is applied, the target user behavior baseline data can be obtained based on the behavior baseline data on the dimension corresponding to the anomaly detection type. For example, if the anomaly detection type is the time detection type, the target user's behavior baseline data on the time dimension is determined as the target user behavior baseline data based on the determined anomaly detection type, and is used in anomaly detection.

[0158] According to the anomaly detection method provided by the present invention, log data is transmitted to a real-time data warehouse, and the target user's behavior baseline data in multiple dimensions is calculated based on the target user's historical log data and the received log data. According to the target user's behavior baseline data in multiple dimensions, the target user's behavior baseline data corresponding to the anomaly detection type is determined, and the updated target user's behavior baseline data corresponding to the anomaly detection type is obtained. The target user's behavior baseline data corresponding to the anomaly detection type can be accurately determined to ensure the accuracy of anomaly detection.

[0159] In another embodiment of the present invention, Figure 5 As shown, in this embodiment, the acquired log data and event configuration data of the target user need to be input into the corresponding message queue, and then the log data in the message queue is processed accordingly, such as generating Topic data, while the event configuration data is only stored in the message queue and does not require further processing.

[0160] In this embodiment, it is necessary to determine the behavior data of the target user based on the log data of the target user obtained. The specific implementation method is to send the obtained log data to the socket interface through the TCP / UDP protocol, and then the data acquisition module collects the data part in the valid Syslog log Json body, and stores it in different message queues according to the data type. When the message queue is determined to be kafka, the message obtained from the message queue kafka is written to the corresponding Topic, such as ida_log_recv, and then the message is further distributed to the user authentication type Topic (user_auth) and the application access Topic (app_access), and then the behavior information of the target user is obtained by analysis and processing.

[0161] In this embodiment, the source data in the log data message queue also needs to be stored in the file system. The real-time data warehouse can subsequently read the source data from the file system to generate a target user behavior baseline, where the source data refers to the log data initially sent to the message queue.

[0162] In this embodiment, the event configuration parameters in the event configuration message queue can be updated to global variables for global effectiveness. It should be noted that by establishing a broadcast stream to monitor the message queue, on the one hand, the real-time messages in the message queue are consumed, and then the log data is stored in the file system as source data to update the target user behavior baseline data, and the latest event configuration data is also updated in real time to the global variables for global effectiveness; on the other hand, after consuming the real-time messages in the message queue, the detection threads of each event are started according to the latest event configuration data, and event rules are matched according to the latest event configuration data, and the abnormal events triggered by detection and discovery are output in real time.

[0163] It should be noted that the event library is a device used to store abnormal event detection results. In addition, in the process of detecting abnormal events, it is necessary to obtain the target user behavior baseline data of the corresponding abnormal detection type from the real-time data warehouse according to the event configuration data. At the same time, the target user behavior baseline data provided by the real-time data warehouse is constantly updated.

[0164] Figure 6 A real-time data warehouse construction device provided by an embodiment of the present invention, such as Figure 6 As shown, the real-time data warehouse construction device provided by the embodiment of the present invention includes:

[0165] The acquisition module 601 is used to pre-acquire the data information of the target user;

[0166] The processing module 602 is used to perform hierarchical processing on the data information of the target user and store the hierarchical processing results in a corresponding storage medium.

[0167] According to the real-time data warehouse construction method provided by the present invention, the data information of the target user is acquired in advance, and then the data information is layered and processed, and each layered processing result is stored in a corresponding storage medium to generate a real-time data warehouse. This can provide data information to the user in real time and provide data support for subsequent anomaly detection, thereby indirectly improving the processing efficiency of anomaly detection.

[0168] Furthermore, the real-time data warehouse includes a detailed data layer, a light summary layer and a public summary layer;

[0169] Accordingly, the processing module 602 is further used for:

[0170] Counting the log data corresponding to different subject types of the target user, and storing the log data under different subject types in the corresponding user subject list in the detailed data layer;

[0171] Counting the behavioral baseline data of target users in multiple dimensions, and storing different behavioral baseline data into the corresponding user behavior list in the light summary layer according to different preset time periods;

[0172] The indicator data of the target user are counted, and the obtained different indicator data are stored in the corresponding user indicator wide table.

[0173] According to the real-time data warehouse construction device provided by the present invention, the constructed real-time data warehouse has a hierarchical structure, and the acquired data is processed in layers by setting multiple layers, which can improve the efficiency of data processing and ensure the real-time performance of anomaly detection.

[0174] Furthermore, the real-time data warehouse includes at least the following storage media: a message queue, a third-party database and a non-relational database,

[0175] Accordingly, the processing module 602 is further used for:

[0176] storing the detailed data obtained by processing the detailed data layer into the non-relational database;

[0177] Storing the user behavior baseline data obtained by the light aggregation layer processing into the message queue;

[0178] The indicator data processed by the public aggregation layer is stored in the third-party database.

[0179] According to the real-time data warehouse construction method provided by the present invention, by storing data after different layer processing in corresponding storage media, the data acquisition rate can be guaranteed and the efficiency of anomaly detection can be improved.

[0180] Furthermore, the present invention also provides an abnormality detection device, comprising:

[0181] A first determination module, configured to determine the behavior data of the target user according to the log data;

[0182] A second determination module, configured to determine an anomaly detection type according to the event configuration data;

[0183] An updating module, used for inputting the log data into a real-time data warehouse to obtain anomaly detection benchmark data corresponding to the anomaly detection type updated according to the log data;

[0184] The third determination module is used to determine whether there is an abnormal event for the target user according to the behavior data of the target user and the updated abnormality detection benchmark data corresponding to the abnormality detection type.

[0185] The present invention provides an anomaly detection device, which obtains log data and event configuration data generated by a target user within a preset time period, determines the behavior data of the target user according to the log data, determines the anomaly detection type according to the event configuration data, inputs the log data into a real-time data warehouse to obtain anomaly detection benchmark data corresponding to the anomaly detection type updated according to the log data, and determines whether the target user has an abnormal event according to the behavior data of the target user and the updated anomaly detection benchmark data corresponding to the anomaly detection type. The anomaly detection method provided by the present invention can ensure the real-time and accuracy of anomaly detection, improve the speed of anomaly detection, and enhance user experience.

[0186] Further, the first determining module is further used for:

[0187] Storing the log data in a message queue;

[0188] In the message queue, the log data is distributed as a message to obtain the behavior data of the target user.

[0189] According to the anomaly detection device provided by the present invention, by storing the real-time acquired log data in a message queue, and then distributing the log data in the message queue, the behavior data of the target user is obtained, thereby improving the real-time and accuracy of anomaly detection.

[0190] Furthermore, the update module is also used for:

[0191] Transmitting the log data to a real-time data warehouse, so that the real-time data warehouse calculates anomaly detection benchmark data corresponding to the anomaly detection type according to the log data and the historical log data of the target user, and obtains updated anomaly detection benchmark data corresponding to the anomaly detection type;

[0192] The updated anomaly detection benchmark data corresponding to the anomaly detection type is received from the real-time data warehouse.

[0193] According to the anomaly detection device provided by the present invention, the log data of the target user is input into the real-time data warehouse, the anomaly detection benchmark data corresponding to the anomaly detection type is calculated in the real-time data warehouse according to the latest log data and historical log data of the target user, and is output from the real-time data warehouse for use in subsequent anomaly detection, thereby improving the real-time performance of anomaly detection and enhancing the user experience.

[0194] Further, the updated anomaly detection benchmark data corresponding to the anomaly detection type includes: updated target user behavior baseline data and anomaly probability threshold corresponding to the anomaly detection type;

[0195] Accordingly, the third determination module is also used for:

[0196] Determining whether the target user has updated target user behavior baseline data corresponding to the anomaly detection type;

[0197] In a case where the target user does not have updated target user behavior baseline data corresponding to the abnormality detection type, it is determined that the target user does not have an abnormal access event.

[0198] According to the anomaly detection device provided by the present invention, by judging whether there is target user behavior baseline data for the target user, when it is confirmed that there is target user behavior baseline data, it is directly determined that there is no abnormal event for the target user, thereby improving the real-time performance of anomaly detection and enhancing user experience.

[0199] Further, the third determination module is also used for:

[0200] If the target user has updated target user behavior baseline data corresponding to the anomaly detection type, calculate the abnormal probability value of the target user according to the target user's behavior data and the updated anomaly detection benchmark data corresponding to the anomaly detection type;

[0201] When the abnormal probability value is less than or equal to a preset abnormal probability threshold, determining that the target user has an abnormal event related to the abnormal detection type;

[0202] When the abnormal probability value is greater than a preset abnormal probability threshold, it is determined that the target user does not have an abnormal event related to the abnormal detection type.

[0203] According to the anomaly detection device provided by the present invention, when there is target user behavior baseline data for the target user, by comparing the calculated anomaly probability value with the preset anomaly probability threshold, it is determined whether the target user has an abnormal event. This can accurately detect whether the target user has an abnormal event, improve the processing efficiency of anomaly detection, and enhance the user experience.

[0204] Furthermore, the update module is also used to:

[0205] Transmitting the log data to a real-time data warehouse;

[0206] In the real-time data warehouse, based on the historical log data of the target user and the received log data, the behavior baseline data of the target user in multiple dimensions is calculated;

[0207] According to the target user's behavior baseline data under the multiple dimensions, the target user's behavior baseline data corresponding to the anomaly detection type is determined, and updated target user's behavior baseline data corresponding to the anomaly detection type is obtained.

[0208] According to the anomaly detection device provided by the present invention, log data is transmitted to a real-time data warehouse, and the target user's behavior baseline data in multiple dimensions is calculated based on the target user's historical log data and the received log data. According to the target user's behavior baseline data in multiple dimensions, the target user's behavior baseline data corresponding to the anomaly detection type is determined, and the updated target user's behavior baseline data corresponding to the anomaly detection type is obtained. The target user's behavior baseline data corresponding to the anomaly detection type can be accurately determined to ensure the accuracy of anomaly detection.

[0209] Since the principle of the device described in the embodiment of the present invention is the same as that of the method described in the above embodiment, a more detailed explanation will not be repeated here.

[0210] Figure 7 Schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Figure 7 As shown, the present invention provides an electronic device, including: a processor (processor) 701, a memory (memory) 702 and a bus 703;

[0211] The processor 701 and the memory 702 communicate with each other via the bus 703;

[0212] The processor 701 is used to call the program instructions in the memory 702 to execute the real-time data warehouse construction method provided in the above-mentioned method embodiments, for example, including: pre-acquiring the data information of the target user; performing layered processing on the data information of the target user, and storing the layered processing results in the corresponding storage medium.

[0213] Another example includes an anomaly detection method: obtaining log data and event configuration data generated by a target user within a preset time period; determining the behavior data of the target user based on the log data; determining the type of anomaly detection based on the event configuration data; inputting the log data into a real-time data warehouse to obtain anomaly detection benchmark data corresponding to the anomaly detection type that is updated according to the log data; and determining whether the target user has an abnormal event based on the behavior data of the target user and the updated anomaly detection benchmark data corresponding to the anomaly detection type.

[0214] In an embodiment of the present invention, a non-transitory computer-readable storage medium is provided, and the non-transitory computer-readable storage medium stores computer instructions. The computer instructions enable the computer to execute the real-time data warehouse construction method provided in the above-mentioned method embodiments, for example, including: pre-acquiring data information of target users; performing layered processing on the data information of the target users, and storing the layered processing results in the corresponding storage medium.

[0215] Another example includes an anomaly detection method: obtaining log data and event configuration data generated by a target user within a preset time period; determining the behavior data of the target user based on the log data; determining the type of anomaly detection based on the event configuration data; inputting the log data into a real-time data warehouse to obtain anomaly detection benchmark data corresponding to the anomaly detection type that is updated according to the log data; and determining whether the target user has an abnormal event based on the behavior data of the target user and the updated anomaly detection benchmark data corresponding to the anomaly detection type.

[0216] The present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the real-time data warehouse construction method provided by the above-mentioned embodiments, and the method includes: pre-acquiring data information of target users; performing layered processing on the data information of the target users, and storing the layered processing results in a corresponding storage medium.

[0217] Another example includes an anomaly detection method: obtaining log data and event configuration data generated by a target user within a preset time period; determining the behavior data of the target user based on the log data; determining the type of anomaly detection based on the event configuration data; inputting the log data into a real-time data warehouse to obtain anomaly detection benchmark data corresponding to the anomaly detection type that is updated based on the log data; and determining whether the target user has an abnormal event based on the behavior data of the target user and the updated anomaly detection benchmark data corresponding to the anomaly detection type.

[0218] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0219] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a real-time data warehouse, characterized in that: include: Obtain target user data information in advance; Performing hierarchical processing on the data information of the target user, and storing the hierarchical processing results in a corresponding storage medium; The real-time data warehouse includes a detailed data layer, a light summary layer and a public summary layer. Accordingly, the layered processing of the target user's data information includes: Counting the log data corresponding to different subject types of the target user, and storing the log data under different subject types in the corresponding user subject list in the detailed data layer; Counting the behavioral baseline data of target users in multiple dimensions, storing different behavioral baseline data in the corresponding user behavior list in the light summary layer according to different preset time periods; the behavioral baseline data is used to form anomaly detection benchmark data corresponding to the anomaly detection type, and the anomaly detection benchmark data is updated in real time through the latest log data of the target user; The indicator data of the target user is counted, and the obtained different indicator data are stored in the corresponding user indicator wide table in the public aggregation layer.

2. The real-time data warehouse construction method according to claim 1, characterized in that: The real-time data warehouse includes at least the following storage media: a message queue, a third-party database and a non-relational database, Accordingly, storing the hierarchical processing result in a corresponding storage medium includes: storing the detailed data obtained by processing the detailed data layer into the non-relational database; Storing the user behavior baseline data obtained by the light aggregation layer processing into the message queue; The indicator data processed by the public aggregation layer is stored in the third-party database.

3. An anomaly detection method, based on the real-time data warehouse obtained by the real-time data warehouse construction method according to any one of claims 1-2, characterized in that: include: Obtain the log data and event configuration data generated by the target user within a preset time period; Determining the behavior data of the target user according to the log data; determining an anomaly detection type according to the event configuration data; Inputting the log data into a real-time data warehouse to obtain anomaly detection benchmark data corresponding to the anomaly detection type updated according to the log data, wherein the anomaly detection benchmark data at least includes target user behavior baseline data and an anomaly probability threshold, and the anomaly detection benchmark data is updated in real time according to the latest log data of the target user; It is determined whether there is an abnormal event for the target user according to the behavior data of the target user and the updated abnormality detection benchmark data corresponding to the abnormality detection type.

4. The anomaly detection method according to claim 3, characterized in that: The determining the behavior data of the target user according to the log data includes: Storing the log data in a message queue; In the message queue, the log data is distributed as a message to obtain the behavior data of the target user.

5. The abnormality detection method according to claim 3, characterized in that: The updating of the anomaly detection benchmark data corresponding to the anomaly detection type according to the log data includes: Transmitting the log data to a real-time data warehouse, so that the real-time data warehouse calculates anomaly detection benchmark data corresponding to the anomaly detection type according to the log data and the historical log data of the target user, and obtains updated anomaly detection benchmark data corresponding to the anomaly detection type; The updated anomaly detection benchmark data corresponding to the anomaly detection type is received from the real-time data warehouse.

6. The abnormality detection method according to claim 3, characterized in that: The updated anomaly detection benchmark data corresponding to the anomaly detection type includes: updated target user behavior baseline data and anomaly probability threshold corresponding to the anomaly detection type; Accordingly, determining whether the target user has an abnormal event according to the behavior data of the target user and the updated abnormality detection benchmark data corresponding to the abnormality detection type includes: Determining whether the target user has updated target user behavior baseline data corresponding to the anomaly detection type; In a case where the target user does not have updated target user behavior baseline data corresponding to the abnormality detection type, it is determined that the target user does not have an abnormal access event.

7. The abnormality detection method according to claim 6, characterized in that: The determining whether the target user has an abnormal event according to the behavior data of the target user and the updated abnormality detection benchmark data corresponding to the abnormality detection type further includes: If the target user has updated target user behavior baseline data corresponding to the anomaly detection type, calculate the abnormal probability value of the target user according to the target user's behavior data and the updated anomaly detection benchmark data corresponding to the anomaly detection type; When the abnormal probability value is less than or equal to a preset abnormal probability threshold, determining that the target user has an abnormal event related to the abnormal detection type; When the abnormal probability value is greater than a preset abnormal probability threshold, it is determined that the target user does not have an abnormal event related to the abnormal detection type.

8. The abnormality detection method according to claim 5, characterized in that: The updated anomaly detection benchmark data corresponding to the anomaly detection type includes: updated target user behavior baseline data corresponding to the anomaly detection type; Accordingly, the transmitting the log data to the real-time data warehouse so that the real-time data warehouse calculates the anomaly detection benchmark data corresponding to the anomaly detection type according to the log data and the historical log data of the target user, and obtains the updated anomaly detection benchmark data corresponding to the anomaly detection type, including: Transmitting the log data to a real-time data warehouse; In the real-time data warehouse, based on the historical log data of the target user and the received log data, the behavior baseline data of the target user in multiple dimensions is calculated; According to the target user's behavior baseline data under the multiple dimensions, the target user's behavior baseline data corresponding to the anomaly detection type is determined, and updated target user's behavior baseline data corresponding to the anomaly detection type is obtained.

9. A real-time data warehouse construction device, characterized in that: include: An acquisition module is used to obtain data information of target users in advance; A processing module, used for performing hierarchical processing on the data information of the target user and storing the hierarchical processing results in a corresponding storage medium; The real-time data warehouse includes a detailed data layer, a light summary layer and a public summary layer. Accordingly, the layered processing of the target user's data information includes: Counting the log data corresponding to different subject types of the target user, and storing the log data under different subject types in the corresponding user subject list in the detailed data layer; Counting the behavioral baseline data of target users in multiple dimensions, storing different behavioral baseline data in the corresponding user behavior list in the light summary layer according to different preset time periods; the behavioral baseline data is used to form anomaly detection benchmark data corresponding to the anomaly detection type, and the anomaly detection benchmark data is updated in real time through the latest log data of the target user; The indicator data of the target user is counted, and the obtained different indicator data are stored in the corresponding user indicator wide table in the public aggregation layer.

10. An anomaly detection device, based on the real-time data warehouse construction device according to claim 9, characterized in that: include; The acquisition module is used to obtain the log data and event configuration data generated by the target user within a preset time period; A first determination module, configured to determine the behavior data of the target user according to the log data; A second determination module, configured to determine an anomaly detection type according to the event configuration data; An updating module, used for inputting the log data into a real-time data warehouse to obtain anomaly detection benchmark data corresponding to the anomaly detection type updated according to the log data, wherein the anomaly detection benchmark data at least includes target user behavior baseline data and an anomaly probability threshold, and the anomaly detection benchmark data is updated in real time through the latest log data of the target user; The third determination module is used to determine whether there is an abnormal event for the target user according to the behavior data of the target user and the updated abnormality detection benchmark data corresponding to the abnormality detection type.

11. The abnormality detection device according to claim 10, characterized in that: The first determining module is further used for: Storing the log data in a message queue; In the message queue, the log data is distributed as a message to obtain the behavior data of the target user.

12. An electronic device, characterized in that: include: processor, memory and bus, wherein, The processor and the memory communicate with each other via the bus; The memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the steps of the real-time data warehouse construction method according to any one of claims 1 to 2 or the anomaly detection method according to any one of claims 3 to 8.

13. A computer program product comprising computer executable instructions, characterized in that: When the instruction is executed, it is used to implement the steps of the real-time data warehouse construction method according to any one of claims 1 to 2 or the anomaly detection method according to any one of claims 3 to 8.

14. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which enable a computer to execute the steps of the real-time data warehouse construction method according to any one of claims 1 to 2 or the anomaly detection method according to any one of claims 3 to 8.

Images