An AI-based abnormal behavior detection method, device and storage medium

Abstract

The present application relates to an AI-based anomaly detection method, device and storage medium for abnormal behavior detection. It includes: obtaining the log information, network traffic data and user authority management metadata of the Hadoop cluster; extracting the target behavior characteristics of the target user from the log information and network traffic data; extracting the permission characteristics of the target user from the user authority management metadata; inputting the target behavior characteristics and permission characteristics into a preset abnormal behavior detection model to obtain a first abnormal behavior score; determining whether the target behavior characteristics are abnormal based on the first abnormal behavior score. It can be seen that the present application does not need to rely on manual regular audits of Hadoop cluster logs to check for anomalies, but instead integrates the three core data types of Hadoop cluster log information, network traffic data and user authority metadata to conduct a systematic abnormality assessment of the target user's behavior. This mechanism can not only realize the timely identification of abnormal behavior, but also improve the detection accuracy, thereby effectively protecting the data security of the Hadoop cluster.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to an AI-based abnormal behavior detection method, device, and storage medium. Background Art

[0002] With the development of the internet, the amount of user data generated by internet terminals has exploded. Internet companies store massive amounts of user data and mine valuable insights to analyze user preferences, thereby providing more targeted services. Hadoop, an open-source distributed computing and storage framework, is widely used for storing and analyzing user data. Therefore, ensuring data security in Hadoop clusters has become a crucial issue.

[0003] Currently, traditional data security mechanisms mainly use static blacklist and whitelist methods to resist external intrusions and prevent data from being stolen or tampered with; at the same time, they rely on manual regular audits of Hadoop cluster logs to detect abnormal behavior.

[0004] However, as cyberattacks continue to evolve, traditional data security mechanisms are unable to effectively defend against these new attacks, leaving Hadoop clusters vulnerable to intrusion. Furthermore, the lag inherent in regular manual monitoring of Hadoop cluster logs makes it difficult to detect abnormal behavior in a timely manner, ultimately posing a serious threat to Hadoop cluster data security. Summary of the Invention

[0005] The embodiments of the present application provide an AI-based abnormal behavior detection method, device, and storage medium, which aim to solve the technical problem that abnormal behavior of a Hadoop cluster cannot be identified in a timely manner due to the lag in manual periodic detection.

[0006] In a first aspect, an embodiment of the present application provides an AI-based abnormal behavior detection method, which includes:

[0007] Obtaining log information, network traffic data, and user rights management metadata of the Hadoop cluster;

[0008] extracting target behavior features of the target user from the log information and the network traffic data;

[0009] Extracting the target user's permission characteristics from the user permission management metadata;

[0010] Inputting the target behavior feature and the permission feature into a preset abnormal behavior detection model to obtain a first abnormal behavior score, wherein the preset abnormal behavior detection model is used to evaluate whether the user's behavior feature is abnormal;

[0011] Determine whether the target behavior feature is abnormal according to the first abnormal behavior score.

[0012] Optionally, before determining whether the target behavior feature is abnormal according to the first abnormal behavior score, the method further includes:

[0013] extracting at least one behavior feature of the target user from the log information and the network traffic data, wherein the at least one behavior feature includes the target behavior feature;

[0014] Inputting the at least one behavior feature into a preset long short-term memory network model to obtain a second abnormal behavior score, wherein the preset long short-term memory network model is used to assess whether the user's behavior feature is abnormal;

[0015] Determining whether the target behavior feature is abnormal according to the first abnormal behavior score includes:

[0016] Determine whether the target behavior feature is abnormal according to the first abnormal behavior score and the second abnormal behavior score.

[0017] Optionally, determining whether the target behavior feature is abnormal according to the first abnormal behavior score includes:

[0018] If the first abnormal behavior score is greater than a preset abnormal value, the behavior of the target user is abnormal; if the first abnormal behavior score is less than or equal to the preset abnormal value, the behavior of the target user is normal.

[0019] Optionally, determining whether the target behavior feature is abnormal based on the first abnormal behavior score and the second abnormal behavior score includes:

[0020] performing a weighted summation of the first abnormal behavior score and the second abnormal behavior score according to a preset weight ratio to obtain a third abnormal behavior score;

[0021] Determine whether the target behavior feature is abnormal according to the third abnormal behavior score.

[0022] Optionally, after determining whether the abnormal behavior score is greater than a preset abnormal value, the method further includes:

[0023] If the first abnormal behavior score is greater than the preset abnormal value, it is determined whether the target behavior characteristics deviate from the preset user profile of the target user, and the preset user profile records the normal behavior characteristics of the target user; if so, the behavior of the target user is abnormal, and if not, the behavior of the target user is normal.

[0024] Optionally, the preset abnormal behavior detection model is constructed based on the isolation forest algorithm.

[0025] Optionally, the steps of constructing the preset long short-term memory network model include:

[0026] Acquiring training data, wherein the training data includes behavior data of normal users and behavior data of abnormal users;

[0027] Build a basic long short-term memory network model;

[0028] Setting the loss function of the preset long short-term memory network model to a binary cross entropy loss function;

[0029] Setting the activation function of the preset long short-term memory network model to a hyperbolic tangent function;

[0030] Setting evaluation indicators of the preset long short-term memory network model, wherein the evaluation indicators include precision, recall and F1 score;

[0031] Based on the loss function, the activation function and the evaluation index, the basic long short-term memory network model is trained, verified and tested using the training data to obtain the preset long short-term memory network model.

[0032] Optionally, the method further includes:

[0033] If the target user's behavior is abnormal, freeze the target user's permissions.

[0034] In a second aspect, an embodiment of the present application further provides an AI-based abnormal behavior detection device, which includes a unit for executing the above method.

[0035] In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0036] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can implement the above method when executed by a processor.

[0037] The embodiment of the present application provides an AI-based anomaly detection method, device and storage medium for abnormal behavior detection. The method includes: obtaining the log information, network traffic data and user rights management metadata of the Hadoop cluster; extracting the target behavior characteristics of the target user from the log information and the network traffic data; extracting the permission characteristics of the target user from the user rights management metadata; inputting the target behavior characteristics and the permission characteristics into a preset abnormal behavior detection model to obtain a first abnormal behavior score, and the preset abnormal behavior detection model is used to evaluate whether the user's behavior characteristics are abnormal; determining whether the target behavior characteristics are abnormal based on the first abnormal behavior score. It can be seen that the technical solution of the present application obtains the log information, network traffic data and user rights management metadata of the Hadoop cluster; then, extracts the target behavior characteristics of the target user from the log information and the network traffic data, and extracts the permission characteristics of the target user from the user rights management metadata. Furthermore, the target behavior characteristics and the permission characteristics are input into the abnormal behavior detection model to obtain a first abnormal behavior score. Finally, it is determined whether the target behavior characteristics are abnormal based on the first abnormal behavior score. As can be seen, the technical solution of this application does not rely on manual periodic audits of Hadoop cluster logs to detect anomalies. Instead, it integrates three core data types: Hadoop cluster log information, network traffic data, and user permission metadata, to systematically evaluate the anomalies of target users' behavior. This mechanism can not only achieve timely identification of abnormal behavior, but also improve detection accuracy, thereby effectively protecting the data security of the Hadoop cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0040] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0041] Figure 1 A schematic diagram of an AI-based abnormal behavior detection process provided in an embodiment of the present application;

[0042] Figure 2A schematic block diagram of an AI-based abnormal behavior detection method provided in an embodiment of the present application;

[0043] Figure 3 A computer device is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0045] The disclosure below provides many different embodiments or examples for implementing different structures of the present application. In order to simplify the disclosure of the present application, the components and settings of specific examples are described below. Of course, these are merely examples and are not intended to limit the present application. In addition, the present application may repeat reference numbers and / or letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0046] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0047] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0048] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0049] As used in this specification and the appended claims, the term “if” can be interpreted as “when” or “upon” or “in response to determining” or “in response to detecting,” depending on the context. Similarly, the phrase “if it is determined” or “if [described condition or event] is detected” can be interpreted as meaning “upon determination” or “in response to determining” or “upon detection of [described condition or event]” or “in response to detecting [described condition or event],” depending on the context.

[0050] In order to solve the technical problem in the prior art that abnormal behavior of the Hadoop cluster cannot be identified in a timely manner due to the lag in manual periodic detection, the present application provides an AI-based abnormal behavior detection device that can realize timely identification of abnormal behavior of the Hadoop cluster.

[0051] Figure 1 A flowchart of an AI-based abnormal behavior detection method provided in an embodiment of the present application is provided. In one embodiment, the method is applied to a Hadoop cluster and includes:

[0052] S1. Obtain Hadoop cluster log information, network traffic data, and user rights management metadata.

[0053] Log information includes, but is not limited to, HDFS logs, YARN logs, MapReduce logs, and Spark logs. Network traffic data includes, but is not limited to, traffic data transmitted between all nodes in the Hadoop cluster. User rights management metadata includes user group information and user rights information.

[0054] S2. Extract the target behavior features of the target user from the log information and network traffic data.

[0055] Log information and network traffic data both record user behavior information. Therefore, the embodiment of the present application extracts the target behavior features of the target user from the log information and network traffic data. Among them, the target behavior features are the behavioral features of the target behavior of the target user.

[0056] In the embodiments of the present application, regular expressions and natural language technology are used to extract key fields from log information and generate structured data in JSON format. Among them, key fields include but are not limited to user ID, timestamp, file path and operation time. For example, the HDFS access.log log records "USER: alice READ / data / sales.csv", and the extracted JSON format data is "{"user": "alice", "action": "READ", "path": " / data / sales.csv", "time": "2025-04-25 14:30:00"}".

[0057] In the embodiments of this application, network traffic is parsed and key features are extracted based on the Scapy library. The Scapy library is a Python library for parsing network packets. Key features include, but are not limited to, network connection duration, packet size distribution, source IP address entropy, destination IP address entropy, and protocol type ratio.

[0058] It should be noted that the target behavior characteristics in the embodiment of the present application include the above-mentioned key fields and key characteristics.

[0059] S3. Extract the target user's permission features from the user permission management metadata.

[0060] In the embodiment of the present application, the permission characteristics of the target user are extracted from the user group information and the user's permission information. For example, the permission characteristics are {user A, auditor, view permission}.

[0061] S4. Input the target behavior characteristics and the permission characteristics into a preset abnormal behavior detection model to obtain a first abnormal behavior score.

[0062] The preset abnormal behavior detection model is used to evaluate whether the user's behavior characteristics are abnormal. The preset abnormal behavior detection model outputs an abnormal behavior score.

[0063] S5. Determine whether the target behavior feature is abnormal based on the first abnormal behavior score.

[0064] In one embodiment, determining whether the target behavior feature is abnormal based on the first abnormal behavior score includes:

[0065] a. If the first abnormal behavior score is greater than a preset abnormal value, the target user's behavior is abnormal; if the first abnormal behavior score is less than or equal to the preset abnormal value, the target user's behavior is normal.

[0066] The preset abnormality value is set by the applicant based on actual experience. This application does not impose any restrictions on this. The embodiment of the present application determines whether the target behavior characteristics of the target user are abnormal based on the abnormal behavior score. The larger the abnormal behavior score, the more abnormal the target behavior characteristics of the target user are.

[0067] The embodiment of the present application provides an AI-based anomaly detection and abnormal behavior detection method. The method includes: obtaining the log information, network traffic data and user authority management metadata of the Hadoop cluster; extracting the target behavior characteristics of the target user from the log information and the network traffic data; extracting the permission characteristics of the target user from the user authority management metadata; inputting the target behavior characteristics and the permission characteristics into a preset abnormal behavior detection model to obtain a first abnormal behavior score, and the preset abnormal behavior detection model is used to evaluate whether the user's behavior characteristics are abnormal; determining whether the target behavior characteristics are abnormal based on the first abnormal behavior score. It can be seen that the technical solution of the present application obtains the log information, network traffic data and user authority management metadata of the Hadoop cluster; then, extracts the target behavior characteristics of the target user from the log information and network traffic data, and extracts the permission characteristics of the target user from the user authority management metadata. Furthermore, the target behavior characteristics and the permission characteristics are input into the abnormal behavior detection model to obtain a first abnormal behavior score. Finally, it is determined whether the target behavior characteristics are abnormal based on the first abnormal behavior score. As can be seen, the technical solution of this application does not rely on manual periodic audits of Hadoop cluster logs to detect anomalies. Instead, it integrates three core data types: Hadoop cluster log information, network traffic data, and user permission metadata, to systematically evaluate the anomalies of target users' behavior. This mechanism can not only achieve timely identification of abnormal behavior, but also improve detection accuracy, thereby effectively protecting the data security of the Hadoop cluster.

[0068] In one embodiment, before determining whether the target behavior feature is abnormal based on the first abnormal behavior score, the method further includes:

[0069] S6. Extract at least one behavioral feature of the target user from the log information and network traffic data.

[0070] Wherein, the at least one behavior feature includes a target behavior feature.

[0071] S7. Input at least one behavioral feature into a preset long short-term memory network model to obtain a second abnormal behavior score.

[0072] A preset long-short-term memory network model is used to assess whether a user's behavioral characteristics are abnormal. It should be noted that in order to more accurately determine whether a target's behavioral characteristics are abnormal, the embodiments of this application extract multiple behavioral characteristics similar to the target behavioral characteristics to analyze the correlation between the target user's multiple behavioral characteristics, thereby effectively capturing behavioral characteristics hidden in the attack chain. The preset long-short-term memory network model outputs an abnormal behavior score. The higher the abnormal behavior score, the more abnormal the user's behavioral characteristics.

[0073] Determining whether the target behavior feature is abnormal according to the first abnormal behavior score includes:

[0074] S51. Determine whether the target behavior feature is abnormal based on the first abnormal behavior score and the second abnormal behavior score.

[0075] In one embodiment, determining whether the target behavior feature is abnormal based on the first abnormal behavior score and the second abnormal behavior score includes:

[0076] S511 . Perform a weighted summation of the first abnormal behavior score and the second abnormal behavior score according to a preset weight ratio to obtain a third abnormal behavior score.

[0077] The preset weight ratio is set by the applicant based on actual experience, and is not limited to this.

[0078] In the embodiment of the present application, a weighted sum operation is performed on the first abnormal behavior score and the second abnormal behavior score according to a preset weight ratio to obtain a third abnormal behavior score. A larger third abnormal behavior score indicates a more abnormal behavior characteristic.

[0079] S512: Determine whether the target behavior feature is abnormal based on the third abnormal behavior score.

[0080] It should be noted that S512 is similar to S5 and will not be described in detail in this application.

[0081] In one embodiment, the method further comprises:

[0082] A. If the first abnormal behavior score is greater than a preset abnormal value, determine whether the target behavior characteristics deviate from the preset user profile of the target user; if so, the target user's behavior is abnormal; if not, the target user's behavior is normal.

[0083] The preset user profile records the normal behavior characteristics of the target user. It should be noted that in the embodiment of the present application, the normal working behavior characteristics of the target user are extracted from the historical log data and historical network traffic data of the Hadoop cluster, and the preset user profile of the target user is constructed based on the obtained large amount of normal working behavior characteristics of the target user.

[0084] In the embodiment of the present application, when the first anomaly score is greater than the preset anomaly value, the target behavior characteristics are further judged based on the preset user profile of the target user to ensure that the target user's behavior is not misjudged. For example, at the end of each month, financial personnel need to read sensitive sales-related data multiple times to calculate the commission of each salesperson. For this scenario, the embodiment of the present application makes further judgments based on the preset user profile of the target user, effectively improving the detection accuracy.

[0085] In one embodiment, the preset abnormal behavior detection model is constructed based on the isolation forest algorithm.

[0086] The embodiment of the present application constructs a preset abnormal behavior detection model according to the following steps.

[0087] First, prepare the data. In this application example, we collect behavioral information from a large number of users. This behavioral information includes, but is not limited to, user ID, operation type, operation time, and number of path visits. We then perform preprocessing on this collected behavioral information to generate a feature matrix. This preprocessing includes, but is not limited to, data standardization, data dimensionality reduction, and feature extraction.

[0088] Next, we construct an isolation tree. We randomly select a feature and a random split value for that feature. Next, we divide the data space into two subspaces. We repeat this process recursively to construct a binary tree until each leaf node contains only one data point or the set height limit is reached.

[0089] Finally, the anomaly score is calculated based on the path length of the sample point in the isolation tree. It should be noted that the shorter the path, the easier it is for the sample point to be isolated, and the higher the anomaly score.

[0090] In one embodiment, the step of constructing the preset long short-term memory network model includes:

[0091] Ⅰ. Obtain training data.

[0092] The training data includes the behavior data of normal users and the behavior data of abnormal users. It should be noted that in the embodiment of the present application, the behavior data of normal users and the behavior data of abnormal users are collected from the Hadoop cluster.

[0093] Ⅱ. Construct a basic long short-term memory network model.

[0094] Among them, the basic long short-term memory network model consists of an input layer, a multi-layer LSTM layer, a fully connected layer and an output layer.

[0095] III. Set the loss function of the preset long short-term memory network model to the binary cross entropy loss function.

[0096] It should be noted that in this application example, determining whether user behavior is normal or abnormal is treated as a binary classification problem.

[0097] IV. Set the activation function of the preset long short-term memory network model to the hyperbolic tangent function.

[0098] It should be noted that the embodiment of the present application maps the input to the (0, 1) interval, where 0 represents normal and 1 represents abnormal.

[0099] V. Set the evaluation indicators of the preset long short-term memory network model.

[0100] Evaluation metrics include precision, recall, and F1 score. Precision indicates the proportion of samples predicted as anomalies that are actually anomalies. Recall indicates the proportion of samples correctly predicted as anomalies among samples that are actually anomalies. The F1 score is the harmonic mean of precision and recall, comprehensively reflecting the performance of the pre-set LSTM model.

[0101] VI. Based on the loss function, activation function and evaluation index, the basic long short-term memory network model is trained, verified and tested using the training data to obtain the preset long short-term memory network model.

[0102] In this embodiment of the present application, the training data is divided into a training set, a validation set, and a test set according to a weight ratio of 7:2:1. The training set is used to train the basic long short-term memory network model and obtain the model parameters. The validation set is used to adjust the hyperparameters of the basic long short-term memory network model. The test set is used to evaluate whether the basic long short-term memory network model meets the evaluation criteria.

[0103] In one embodiment, the method further comprises:

[0104] If the target user's behavior is abnormal, freeze the target user's permissions.

[0105] It should be noted that in the embodiment of the present application, if abnormal behavior of the target user is detected, corresponding protection operations will be performed, for example, freezing the target user's permissions so that the target cannot further read or tamper with data.

[0106] In one embodiment, the present invention can be linked with an access control list function module to extract abnormal IP addresses and ports and write them into an access control list to block abnormal IP access.

[0107] In another embodiment, the embodiment of the present application can also immediately generate a snapshot of the attacked file and start data integrity verification.

[0108] In one embodiment, the present application associates all logs of the same session through a distributed transaction ID and generates a visual attack path diagram for management personnel to review.

[0109] In one embodiment, the embodiment of the present application regularly collects new log information and network traffic data on the Hadoop cluster, and fine-tunes the preset long short-term memory network model and the preset abnormal behavior detection model so that the preset long short-term memory network model and the preset abnormal behavior detection model can adapt to changes in business models.

[0110] See also Figure 2 , Figure 2 This is a schematic block diagram of an AI-based abnormal behavior detection device provided by an embodiment of the present application. Corresponding to the above AI-based abnormal behavior detection method, the present application also provides an AI-based abnormal behavior detection device. The AI-based abnormal behavior detection device includes a unit for executing the above AI-based abnormal behavior detection method. The AI-based abnormal behavior detection device can be configured in a terminal such as a desktop computer, a tablet computer, or a laptop computer. Specifically, the AI-based abnormal behavior detection device is applied to a Hadoop cluster and includes:

[0111] An acquisition unit 201 is configured to acquire log information, network traffic data, and user rights management metadata of the Hadoop cluster;

[0112] A first extraction unit 202 is configured to extract target behavior features of a target user from the log information and the network traffic data;

[0113] A second extraction unit 203 is configured to extract the target user's permission characteristics from the user permission management metadata;

[0114] An input unit 204 is configured to input the target behavior feature and the permission feature into a preset abnormal behavior detection model to obtain a first abnormal behavior score. The preset abnormal behavior detection model is used to evaluate whether the user's behavior feature is abnormal.

[0115] The determining unit 205 is configured to determine whether the target behavior feature is abnormal according to the first abnormal behavior score.

[0116] In one embodiment, the first extraction unit 202 is further configured to extract at least one behavior feature of the target user from the log information and the network traffic data, wherein the at least one behavior feature includes the target behavior feature;

[0117] The input unit 204 is further configured to input the at least one behavior feature into a preset long short-term memory network model to obtain a second abnormal behavior score, wherein the preset long short-term memory network model is used to evaluate whether the user's behavior feature is abnormal;

[0118] The determining unit 205 is specifically configured to determine whether the target behavior feature is abnormal according to the first abnormal behavior score and the second abnormal behavior score.

[0119] In one embodiment, the determining unit 205 is further specifically configured to determine that if the first abnormal behavior score is greater than a preset abnormal value, the behavior of the target user is abnormal; if the first abnormal behavior score is less than or equal to the preset abnormal value, the behavior of the target user is normal.

[0120] In one embodiment, the determining unit 205 is further configured to perform a weighted summation of the first abnormal behavior score and the second abnormal behavior score according to a preset weight ratio to obtain a third abnormal behavior score;

[0121] Determine whether the target behavior feature is abnormal according to the third abnormal behavior score.

[0122] In one embodiment, the determination unit 205 is further specifically used to determine whether the target behavior characteristics deviate from the preset user portrait of the target user if the first abnormal behavior score is greater than the preset abnormal value, and the preset user portrait records the normal behavior characteristics of the target user; if so, the behavior of the target user is abnormal, and if not, the behavior of the target user is normal.

[0123] In one embodiment, the apparatus further includes a construction unit 206 for constructing a preset abnormal behavior detection model based on an isolation forest algorithm.

[0124] In one embodiment, the construction unit 206 is further configured to obtain training data, wherein the training data includes behavior data of normal users and behavior data of abnormal users;

[0125] Build a basic long short-term memory network model;

[0126] Setting the loss function of the preset long short-term memory network model to a binary cross entropy loss function;

[0127] Setting the activation function of the preset long short-term memory network model to a hyperbolic tangent function;

[0128] Setting evaluation indicators of the preset long short-term memory network model, wherein the evaluation indicators include precision, recall and F1 score;

[0129] Based on the loss function, the activation function and the evaluation index, the basic long short-term memory network model is trained, verified and tested using the training data to obtain the preset long short-term memory network model.

[0130] In one embodiment, the device further includes a freezing unit 207, configured to freeze the rights of the target user if the target user behaves abnormally.

[0131] like Figure 3 As shown, an embodiment of the present application provides a computer device, including a processor 31, a communication interface 32, a memory 33 and a communication bus 34, wherein the processor 31, the communication interface 32, and the memory 33 communicate with each other through the communication bus 34, and the memory 33 is used to store computer programs;

[0132] In one embodiment of the present application, the processor 31 is configured to implement the AI-based abnormal behavior detection control method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 33 .

[0133] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0134] Therefore, an embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the AI-based abnormal behavior detection method provided in any of the aforementioned method embodiments are implemented.

[0135] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other physical storage media capable of storing program code. The computer-readable storage medium may be either non-volatile or volatile.

[0136] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and other division methods may be used in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not implemented.

[0138] The steps in the method of the embodiment of the present application can be adjusted in order, combined, and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided, and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit.

[0139] If this integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application.

[0140] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0141] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, as long as these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

[0142] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An AI-based abnormal behavior detection method, characterized in that: The method is applied to a Hadoop cluster and includes: Obtaining log information, network traffic data, and user rights management metadata of the Hadoop cluster; extracting target behavior features of the target user from the log information and the network traffic data; Extracting the target user's permission characteristics from the user permission management metadata; Inputting the target behavior feature and the permission feature into a preset abnormal behavior detection model to obtain a first abnormal behavior score, wherein the preset abnormal behavior detection model is used to evaluate whether the user's behavior feature is abnormal; extracting at least one behavior feature of the target user from the log information and the network traffic data, wherein the at least one behavior feature includes the target behavior feature; Inputting the at least one behavior feature into a preset long short-term memory network model to obtain a second abnormal behavior score, wherein the preset long short-term memory network model is used to assess whether the user's behavior feature is abnormal; performing a weighted summation of the first abnormal behavior score and the second abnormal behavior score according to a preset weight ratio to obtain a third abnormal behavior score; Determine whether the target behavior feature is abnormal according to the third abnormal behavior score.

2. The method according to claim 1, characterized in that Determining whether the target behavior feature is abnormal according to the first abnormal behavior score includes: If the first abnormal behavior score is greater than a preset abnormal value, the behavior of the target user is abnormal; if the first abnormal behavior score is less than or equal to the preset abnormal value, the behavior of the target user is normal.

3. The method according to claim 2, characterized in that The method further comprises: If the first abnormal behavior score is greater than the preset abnormal value, it is determined whether the target behavior characteristics deviate from the preset user profile of the target user, and the preset user profile records the normal behavior characteristics of the target user; if so, the behavior of the target user is abnormal, and if not, the behavior of the target user is normal.

4. The method according to any one of claims 1 to 3, characterized in that The preset abnormal behavior detection model is constructed based on the isolation forest algorithm.

5. The method according to claim 1, characterized in that The steps of constructing the preset long short-term memory network model include: Acquiring training data, wherein the training data includes behavior data of normal users and behavior data of abnormal users; Build a basic long short-term memory network model; Setting the loss function of the preset long short-term memory network model to a binary cross entropy loss function; Setting the activation function of the preset long short-term memory network model to a hyperbolic tangent function; Setting evaluation indicators of the preset long short-term memory network model, wherein the evaluation indicators include precision, recall and F1 score; Based on the loss function, the activation function and the evaluation index, the basic long short-term memory network model is trained, verified and tested using the training data to obtain the preset long short-term memory network model.

6. The method according to claim 2, characterized in that The method further comprises: If the target user's behavior is abnormal, freeze the target user's permissions.

7. An AI-based abnormal behavior detection device, characterized in that: The device is applied to a Hadoop cluster and includes: An acquisition unit, configured to acquire log information, network traffic data, and user rights management metadata of the Hadoop cluster; A first extraction unit is used to extract target behavior features of a target user from the log information and the network traffic data; A second extraction unit is used to extract the permission characteristics of the target user from the user permission management metadata; An input unit, configured to input the target behavior feature and the permission feature into a preset abnormal behavior detection model to obtain a first abnormal behavior score, wherein the preset abnormal behavior detection model is used to evaluate whether the user's behavior feature is abnormal; The first extraction unit is further configured to extract at least one behavior feature of the target user from the log information and the network traffic data, wherein the at least one behavior feature includes the target behavior feature; The input unit is further configured to input the at least one behavior feature into a preset long short-term memory network model to obtain a second abnormal behavior score, wherein the preset long short-term memory network model is used to evaluate whether the user's behavior feature is abnormal; A determination unit is configured to perform a weighted summation of the first abnormal behavior score and the second abnormal behavior score according to a preset weight ratio to obtain a third abnormal behavior score; and determine whether the target behavior feature is abnormal based on the third abnormal behavior score.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which, when executed by a processor, can implement the method according to any one of claims 1 to 6.

Images