Abnormal access identification method and device for big data
By constructing a global dataset and optimizing outlier detection parameters, combined with feature fusion and parameter optimization methods, the problem of large volume and high redundancy of device log data in large-scale network scenarios is solved, achieving efficient and accurate abnormal access identification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ELECTRIC POWER RES INST CHINA SOUTHERN POWER GRID CO LTD
- Filing Date
- 2026-02-13
- Publication Date
- 2026-05-01
AI Technical Summary
In large-scale network scenarios, existing technologies result in large amounts of data and high redundancy in device operation logs, which increases the time required to retrieve logs from the database, reduces the efficiency of abnormal access identification, and makes it difficult to meet the needs of real-time detection.
By acquiring behavioral data within the target time period from the target device's work logs, a global dataset is constructed. A preset clustering algorithm is used to divide the target clusters, outlier detection parameters are optimized, and features are extracted by combining wavelet transform and depth-first search algorithms. Feature fusion and parameter optimization are then performed to finally identify abnormal behavior.
It improves the efficiency and accuracy of abnormal access identification, reduces noise interference, shortens identification time, enhances the system's adaptability to dynamic changes in user permissions, and provides a more flexible access control mechanism.
Smart Images
Figure CN121967058A_ABST
Abstract
Description
Methods and devices for identifying abnormal access to big data Technical Field
[0001] This application relates to the field of artificial intelligence technology, specifically to a method and apparatus for identifying abnormal access to big data. Background Technology
[0002] With the development of information technology and network communication technology, network security protection has become a key guarantee for the stable operation of information systems. Network anomaly access identification refers to the technical means of identifying and extracting abnormal patterns and suspicious operations in user access behavior through real-time monitoring and in-depth analysis of core network data such as network traffic, data packets, and connection status, providing a basis for decision-making in the network security protection system.
[0003] In related technologies, a typical method for identifying abnormal access objects includes the following steps: acquiring target access data generated during the process of the object to be detected accessing a preset URL, wherein the target access data includes target access behavior data and account device association data; extracting features from the target access behavior data to obtain a first predictive reference feature; extracting features from the account device association data to obtain a second predictive reference feature; and identifying whether the object to be detected is an abnormal access object based on the first predictive reference feature and the second predictive reference feature.
[0004] However, the above-mentioned abnormal access object identification method has significant technical defects: when the amount of access data of the object to be detected is large, the device working log will show problems of large data volume and high redundancy, which will increase the time spent retrieving logs from the database, thereby reducing the speed of effective feature extraction, and ultimately resulting in low efficiency of abnormal access identification, making it difficult to meet the real-time detection needs of large-scale network scenarios. Summary of the Invention
[0005] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, or computer program product for identifying abnormal access to big data, which can improve the efficiency of abnormal access identification, in order to address the above-mentioned technical problems.
[0006] In a first aspect, embodiments of this application provide a method for identifying abnormal access to big data, the method comprising:
[0007] Obtain the first behavioral data of the target user within the target time period from the working log of the target device, and construct a global dataset based on the first behavioral data;
[0008] A preset clustering algorithm is used to perform clustering training on the global dataset to divide the addresses involved in the interaction in the global dataset into multiple target clusters, and the outlier detection parameters are optimized based on the features of the target clusters to obtain the first optimized outlier detection parameters.
[0009] Outlier detection is performed on the global dataset based on the first optimized outlier detection parameters to obtain outlier points. Based on the global dataset, the outlier points, and the preset normal behavior database, the valid data to be detected for the object to be detected is determined.
[0010] Basic features and effective abnormal behavior features are obtained from the effective data to be detected, and the basic features and effective abnormal behavior features are integrated using a preset feature fusion algorithm to obtain a composite feature set. The outlier detection parameters after the first optimization are optimized based on the composite feature set to obtain the outlier detection parameters after the second optimization.
[0011] Based on the second optimized outlier detection parameters, abnormal behavior identification is performed on the valid data to be detected to obtain abnormal access identification results.
[0012] In one embodiment, determining the valid data to be detected for the object to be detected based on the global dataset, the outliers, and the preset normal behavior database includes:
[0013] Based on the global dataset, the outliers, and the preset normal behavior database, the data to be detected for the object to be detected is determined;
[0014] The data to be detected is denoised using a preset wavelet transform algorithm to obtain the valid data to be detected.
[0015] In one embodiment, determining the data to be detected for the object to be detected based on the global dataset, the outliers, and the preset normal behavior database includes:
[0016] The access patterns of the global dataset and the behavioral data in the outliers are compared with the standard access patterns in the preset normal behavior database.
[0017] If the comparison result shows that the access patterns are different, then the user to whom the corresponding data belongs is identified as the object to be detected.
[0018] In one embodiment, obtaining basic features and valid abnormal behavior features from the valid data to be detected includes:
[0019] Basic features are extracted from the valid data to be detected, and a state model is constructed based on the basic features;
[0020] The state model is traversed using a preset depth-first search algorithm to extract the effective abnormal behavior features.
[0021] In one embodiment, the method further includes:
[0022] The system acquires second behavioral data of the target user within a historical time period, constructs a multi-dimensional behavioral vector based on the second behavioral data, and constructs the preset normal behavior database based on the multi-dimensional behavioral vector.
[0023] In one embodiment, the step of identifying abnormal behavior in the valid data to be detected based on the second optimized outlier detection parameters to obtain an abnormal access identification result includes:
[0024] When abnormal behavior is detected, the abnormal behavior is classified according to a preset abnormality category, which includes abnormal login address, abnormal login time, or abnormal operation command.
[0025] Based on the same inventive concept, in a second aspect, embodiments of this application provide an abnormal access identification device for big data, the device comprising:
[0026] The data acquisition module is used to obtain the first behavioral data of the target user within the target time period from the working log of the target device, and to construct a global dataset based on the first behavioral data;
[0027] The cluster determination module is used to perform clustering training on the global dataset using a preset clustering algorithm to divide the addresses involved in the interaction in the global dataset into multiple target clusters, and optimize the outlier detection parameters based on the features of the target clusters to obtain the first optimized outlier detection parameters.
[0028] The detection determination module is used to perform outlier detection on the global dataset based on the first optimized outlier detection parameters, obtain outlier points, and determine the valid data to be detected for the object to be detected based on the global dataset, the outlier points, and the preset normal behavior database.
[0029] The feature fusion module is used to obtain basic features and effective abnormal behavior features from the effective data to be detected, and to integrate the basic features and effective abnormal behavior features using a preset feature fusion algorithm to obtain a composite feature set. Based on the features of the composite feature set, the module optimizes the outlier detection parameters after the first optimization to obtain the outlier detection parameters after the second optimization.
[0030] The anomaly detection module is used to identify abnormal behavior in the valid data to be detected based on the second optimized outlier detection parameters, and obtain the abnormal access identification result.
[0031] Based on the same inventive concept, in a third aspect, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect above.
[0032] Based on the same inventive concept, in a fourth aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect above.
[0033] Based on the same inventive concept, in a fifth aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.
[0034] The aforementioned method, apparatus, computer equipment, computer-readable storage medium, and computer program product for identifying abnormal access to big data obtains the first behavioral data of the target user within a target time period from the target device's work log, and constructs a global dataset based on the first behavioral data; it uses a preset clustering algorithm to perform clustering training on the global dataset to divide the addresses involved in the interaction in the global dataset into multiple target clusters, and optimizes the outlier detection parameters based on the features of the target clusters to obtain the first optimized outlier detection parameters; it performs outlier detection on the global dataset based on the first optimized outlier detection parameters to obtain outlier points, and determines the valid data to be detected for the target object based on the global dataset, outlier points, and a preset normal behavior database; it obtains basic features and valid abnormal behavior features from the valid data to be detected, and integrates the basic features and valid abnormal behavior features using a preset feature fusion algorithm to obtain a composite feature set, and optimizes the first optimized outlier detection parameters based on the features of the composite feature set to obtain the second optimized outlier detection parameters; based on the second optimized outlier detection parameters, it performs abnormal behavior identification on the valid data to be detected to obtain the abnormal access identification result; thus improving the efficiency of abnormal access identification.
[0035] The solution in this application, through feature extraction and state model establishment, can separate noise information in behavioral data, reduce the redundancy of database access to behavioral data, and suppress the interference of noise information on behavioral information, thereby quickly extracting effective features and shortening the recognition time. Through feature synthesis and parameter optimization, the judgment factor for abnormal access recognition can be optimized, thereby improving the recognition accuracy. By using dimensionality reduction and clustering processing, the parameters for outlier detection can be adjusted, the addresses involved in the interaction can be accurately divided, the accuracy of outlier detection can be improved, outlier data can be obtained in a timely manner, and data omissions can be reduced. Thus, the entire system is more adaptable to dynamic changes in user permissions, has low control overhead, and has a more granular and flexible access control mechanism, further improving the recognition of abnormal user access behavior. Attached Figure Description
[0036] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0037] Figure 1 is a flowchart illustrating the abnormal access identification method for big data in an embodiment of this application;
[0038] Figure 2 is a flowchart of step S103 in an embodiment of this application;
[0039] Figure 3 is a flowchart of step S104 in an embodiment of this application;
[0040] Figure 4 is a schematic diagram of the laboratory test results of the abnormal access identification method using big data in the embodiments of this application;
[0041] Figure 5 is one of the structural block diagrams of the abnormal access identification device for big data in the embodiments of this application;
[0042] Figure 6 is a second structural block diagram of the abnormal access identification device for big data in an embodiment of this application;
[0043] Figure 7 is a schematic diagram of the internal structure of the computer device in an embodiment of this application. Detailed Implementation
[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0045] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.
[0046] This application aims to provide a method for identifying abnormal access to large amounts of data, in order to solve the problem mentioned in the background art that when the amount of data accessed is large, the data stored in the device's working log is large and has high redundancy, and the time spent accessing the log file using the database is long, which reduces the speed of extracting effective features and thus reduces the efficiency of identifying abnormal access.
[0047] In an exemplary embodiment, referring to FIG1, a method for identifying abnormal access to big data is provided, which may include the following steps S101 to S105.
[0048] S101: Obtain the first behavior data of the target user within the target time period from the working log of the target device, and construct a global dataset based on the first behavior data.
[0049] Here, the target device is, for example, the current device. The target user is, for example, the current user; as an example, the target user can be all users under the target device. It can be based on traversing the target device's work logs using big data to obtain the target user's behavioral data within the target time period, which serves as the first behavioral data. As an example, it can be based on the behavioral data of all users under the target device to construct a global dataset.
[0050] S102, a preset clustering algorithm is used to perform clustering training on the global dataset to divide the addresses involved in the interaction in the global dataset into multiple target clusters, and the outlier detection parameters are optimized based on the features of the target clusters to obtain the first optimized outlier detection parameters.
[0051] As an example, the preset clustering algorithm can be the K-Means algorithm. As another example, the global dataset can be simplified first using a dimensionality reduction algorithm (such as, but not limited to, Principal Component Analysis, PCA). Then, the addresses involved in the interaction are divided into multiple target clusters. Simultaneously, outlier detection parameters are calculated and adjusted based on the distribution characteristics of the target clusters (such as, but not limited to, cluster centers, data distribution variance), resulting in the first optimized outlier detection parameters. As another example, the number of target clusters can be 3, 4, 5, etc. This embodiment does not specify a particular number of target clusters.
[0052] S103, based on the outlier detection parameters optimized in the first step, outlier detection is performed on the global dataset to obtain outlier points, and based on the global dataset, outlier points, and a preset normal behavior database, the valid data to be detected for the object to be detected is determined.
[0053] One approach is to quickly classify the global dataset based on the outlier detection parameters optimized in the first iteration, marking data that cannot be classified into any of the multiple target clusters as outliers.
[0054] Understandably, outliers are key data points in big data anomaly detection methods, as they can improve detection efficiency. In the process of determining valid data for the target object based on the global dataset, outliers, and a pre-defined normal behavior database, outliers are used to improve comparison efficiency, the global dataset is used as a fallback to avoid missing anomalies due to clustering errors, and the optimized outlier detection parameters provide an initial benchmark for subsequent optimizations.
[0055] Optionally, the big data abnormal access identification method may also include the steps of: obtaining the second behavior data of the target user within a historical period, constructing a multi-dimensional behavior vector based on the second behavior data, and constructing a preset normal behavior database based on the multi-dimensional behavior vector.
[0056] As an example, one approach could be to acquire historical behavioral data of all users on the target device over a given period, serving as secondary behavioral data. Based on this secondary behavioral data, a multi-dimensional behavioral vector (such as a three-dimensional vector of "access frequency - access period - target address") could be constructed. General normal access patterns could then be extracted from these multi-dimensional behavioral vectors, and a normal behavior database could be built using simulation software. This would establish a more accurate historical benchmark for normal behavior, providing a reference for current behavior comparisons.
[0057] In some exemplary embodiments, referring to FIG2, the determination of the valid data to be detected of the object to be detected in step S103 based on the global dataset, outliers and preset normal behavior database may include the following steps S1031 to S1032.
[0058] S1031, Based on the global dataset, outliers, and preset normal behavior database, determine the data to be detected for the object to be detected.
[0059] One approach is to compare the current behavior data with the standard access patterns in a preset normal behavior database. If the access patterns are the same, the data is recorded using the work log without any anomaly detection. If the access patterns are different, the current user is locked and identified as the target of detection. Then, the behavior data of the target of detection is extracted as the data to be detected.
[0060] In step S1031, the detection data of the target object is determined based on the global dataset and outliers. The dual data support logic of "prioritizing key data and providing comprehensive backup" is used for comparison to obtain the detection data. This achieves precise shrinkage from "global data" to "focused data", reducing the computational cost for subsequent feature processing.
[0061] S1032 uses a preset wavelet transform algorithm to reduce noise in the data to be detected, thus obtaining the valid data to be detected.
[0062] In step S1032, a preset wavelet transform algorithm is used to process the data to be detected, separating effective behavioral information from noise (such as invalid data generated by network fluctuations and acquisition errors). Noise can be removed by setting a threshold, suppressing the interference of noise on effective information, improving data purity, and avoiding feature extraction deviation caused by noise.
[0063] S104: Obtain basic features and effective abnormal behavior features from the valid data to be detected, and integrate the basic features and effective abnormal behavior features using a preset feature fusion algorithm to obtain a composite feature set. Then, optimize the outlier detection parameters based on the composite feature set after the first optimization to obtain the outlier detection parameters after the second optimization.
[0064] This involves referencing feature data from a pre-defined feature library to extract basic features from the valid data to be detected, based on the type of feature data. This means initially extracting the data features of the valid data to be detected according to the type of feature data. Through communication with cloud devices, feature library information can be synchronized, enabling regular updates to the feature library data. Furthermore, association rule algorithms can be used to process composite feature sets, select thresholds for identifying abnormal access behavior, and optimize the judgment of abnormal access behavior.
[0065] Step S104 involves fusing basic features and abnormal features, and iteratively optimizing outlier detection parameters based on the fused features to improve the accuracy of anomaly detection and reduce false positives and false negatives. In other words, based on the distribution characteristics of the composite feature set, the outlier detection parameters (such as feature matching threshold, abnormal behavior weight, etc.) after the first optimization are adjusted and optimized to make the parameters more closely match the actual anomaly detection requirements.
[0066] As an example, the expression for feature fusion calculation is as follows:
[0067] ;
[0068] In the above formula, T is the number of features fused, i is the number of iterations, j is the feature value, d is the number of efeat functions, s is the number of rfeat functions, and a and b are the number of backward and forward relations, respectively.
[0069] In some exemplary embodiments, referring to FIG3, the step S104 of obtaining basic features and effective abnormal behavior features from the valid data to be detected may include the following steps S1041~S1042.
[0070] S1041, extract basic features from the valid data to be detected, and build a state model based on the basic features.
[0071] Step S1041 involves the preliminary extraction of data features from the valid data to be detected, and the establishment of a state model. For example, basic features include, but are not limited to, access frequency, login address, and operation command type. As an example, an information set of user access behavior patterns is constructed using a time window function and behavioral information, and a user access behavior state model is established using a random weighted network.
[0072] S1042 uses a preset depth-first search algorithm to traverse the state model in order to extract effective abnormal behavior features.
[0073] The preset depth-first search algorithm is DFS (Depth First Search). This algorithm processes the state model, filtering out redundant features related to normal behavior and retaining features strongly correlated with abnormal access, thereby extracting effective abnormal access behavior information features.
[0074] S105, based on the outlier detection parameters optimized in the second round, performs abnormal behavior identification on the valid data to be detected, and obtains the abnormal access identification result.
[0075] In step S105, as an example, if an abnormal behavior is identified, it is categorized according to preset abnormality types, including abnormal user login address, abnormal login time, and abnormal operation command. This clarifies the abnormality type, supporting subsequent targeted security responses (such as account locking, pop-up alerts, and operation interception), thus improving the efficiency of security protection responses.
[0076] The scheme in this application embodiment features unidirectional data flow and layer-by-layer contraction at each step to ensure computational efficiency; simultaneously, it avoids anomaly omissions through a "global data fallback." The beneficial effects achieved by the scheme in this application embodiment include: 1) a global normal benchmark construction method based on K-Means clustering to achieve preliminary and efficient screening of massive user behavior data; 2) a hierarchical comparison logic of "prioritizing key areas + comprehensive fallback" to balance detection efficiency and comprehensiveness; 3) an anomaly feature deep mining method combining wavelet transform denoising and DFS algorithm to improve feature extraction accuracy; and 4) a parameter optimization mechanism based on feature fusion to achieve dynamic calibration of judgment criteria, reducing false positives and false negatives.
[0077] In the abnormal access identification method for big data in this application embodiment:
[0078] As Example 1, the addresses participating in the interaction were divided into 5 target clusters, and the DSF algorithm was used for feature point extraction. As Example 2, the addresses participating in the interaction were divided into 4 target clusters, and the DSF algorithm was used for feature point extraction. As Example 3, the addresses participating in the interaction were divided into 3 target clusters, and the DSF algorithm was used for feature point extraction. As Comparative Example 1, the only difference from Example 1 is that the BP neural network algorithm was used for feature point extraction. As Comparative Example 2, the only difference from Example 1 is that the random forest algorithm was used for feature point extraction. As Comparative Example 3, the only difference from Example 1 is that the basic features and effective abnormal behavior features were not fused.
[0079] Laboratory test results:
[0080] The recognition data obtained by processing Example 1, Example 2 and Example 3 are recorded as Example Group 1 to 3, and the recognition data obtained by processing Comparative Example 1, Comparative Example 2 and Comparative Example 3 are recorded as Comparison Group 1 to 3. The test duration and accuracy of the recognition data of Example Group 1 to 3 and Comparison Group 1 to 3 are tested, and the relevant experimental data are recorded in Table 1 and Figure 4.
[0081] Table 1
[0082]
[0083] As shown in Table 1, the recognition data obtained by processing through Examples 1 to 3 has a test time of ≤0.003ms and an accuracy of ≥77.6%. The test time is short and the accuracy is high. Therefore, the recognition data obtained by processing through Examples 1, 2 and 3 have high detection efficiency and accuracy.
[0084] By changing the method of extracting access behavior information data features, it can be seen that when using the DSF algorithm, the test time is shorter and the accuracy is higher. When using the BF neural network algorithm, the test time increases and the accuracy decreases. When using the random forest algorithm, the test time continues to increase and the accuracy further decreases. When fusing user data features with abnormal access behavior information features, although the test time is shorter, the accuracy is lower. Therefore, when using the BF neural network and random forest algorithms for feature extraction in the identification of abnormal user access, the overall identification time is longer and the accuracy is lower.
[0085] In this embodiment, by feature extraction and state model establishment, noise information in behavioral data can be separated, reducing the redundancy of database access to behavioral data and suppressing the interference of noise information on behavioral information. This allows for the rapid extraction of effective features and shortens the recognition time. Through feature synthesis and parameter optimization, the judgment factors for abnormal access recognition can be optimized, thereby improving the recognition accuracy. By periodically updating the feature library data, assistance can be provided during feature extraction of the detected object data, reducing the lag and insufficient coverage of the feature library. Dimensionality reduction and clustering processing can be used to adjust the parameters of outlier detection, accurately classify the addresses involved in the interaction, improve the accuracy of outlier detection, obtain outlier data in a timely manner, and reduce data omissions. As a result, the entire system is more adaptable to dynamic changes in user permissions, has low control overhead, and has a more granular and flexible access control mechanism, further improving the recognition of abnormal user access behavior.
[0086] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
[0087] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.
[0088] Based on the same inventive concept, this application also provides an abnormal access identification device for implementing the abnormal access identification method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more abnormal access identification device embodiments provided below can be found in the limitations of the abnormal access identification method described above, and will not be repeated here.
[0089] In an exemplary embodiment, as shown in FIG5, an abnormal access identification device for big data is provided, which may include the following modules:
[0090] The data acquisition module 510 is used to acquire the first behavior data of the target user within the target time period from the working log of the target device, and to construct a global dataset based on the first behavior data;
[0091] The cluster determination module 520 is used to perform cluster training on the global dataset using a preset clustering algorithm to divide the addresses involved in the interaction in the global dataset into multiple target clusters, and optimize the outlier detection parameters based on the features of the target clusters to obtain the first optimized outlier detection parameters.
[0092] The detection determination module 530 is used to perform outlier detection on the global dataset based on the first optimized outlier detection parameters, obtain outlier points, and determine the valid data to be detected for the object to be detected based on the global dataset, the outlier points and the preset normal behavior database.
[0093] The feature fusion module 540 is used to obtain basic features and effective abnormal behavior features from the effective data to be detected, and to integrate the basic features and effective abnormal behavior features using a preset feature fusion algorithm to obtain a composite feature set, and to optimize the outlier detection parameters after the first optimization based on the composite feature set to obtain the outlier detection parameters after the second optimization.
[0094] The anomaly identification module 550 is used to identify abnormal behavior in the valid data to be detected based on the second optimized outlier detection parameters, and obtain the abnormal access identification result.
[0095] In an exemplary embodiment, the detection determination module 530 is further configured to determine the data to be detected of the object to be detected based on the global dataset, the outliers and the preset normal behavior database; and to perform noise reduction on the data to be detected using a preset wavelet transform algorithm to obtain the valid data to be detected.
[0096] In an exemplary embodiment, the detection determination module 530 is further configured to compare the access patterns of the global dataset and the behavioral data in the outliers with the standard access patterns in the preset normal behavior database; if the comparison result shows that the access patterns are different, then the user to which the corresponding data belongs is determined as the object to be detected.
[0097] In an exemplary embodiment, the feature fusion module 540 is further configured to extract basic features from the valid data to be detected, and construct a state model based on the basic features; and traverse the state model using a preset depth-first search algorithm to extract the valid abnormal behavior features.
[0098] In an exemplary embodiment, referring to FIG6, the big data abnormal access identification device may further include a database construction module 560, which is used to acquire the second behavior data of the target user in a historical period, construct a multi-dimensional behavior vector based on the second behavior data, and construct the preset normal behavior database based on the multi-dimensional behavior vector.
[0099] In an exemplary embodiment, the anomaly identification module is further configured to classify the abnormal behavior according to a preset anomaly category when an abnormal behavior is determined, wherein the preset anomaly category includes abnormal login address, abnormal login time, or abnormal operation command.
[0100] The modules in the aforementioned big data anomaly access identification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0101] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram is shown in Figure 7. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device stores relevant data. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for identifying abnormal access to large amounts of data.
[0102] Those skilled in the art will understand that the structure shown in Figure 7 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0103] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.
[0104] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.
[0105] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0106] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0107] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0108] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for identifying abnormal access to big data, characterized in that, The method includes: obtaining first behavioral data of a target user within a target time period from the working log of the target device, and constructing a global dataset based on the first behavioral data; performing clustering training on the global dataset using a preset clustering algorithm to divide the addresses involved in the interaction in the global dataset into multiple target clusters, and optimizing outlier detection parameters based on the features of the target clusters to obtain first-optimized outlier detection parameters; performing outlier detection on the global dataset based on the first-optimized outlier detection parameters to obtain outlier points, and determining the valid data to be detected for the target object based on the global dataset, the outlier points, and a preset normal behavior database; obtaining basic features and valid abnormal behavior features from the valid data to be detected, and integrating the basic features and the valid abnormal behavior features using a preset feature fusion algorithm to obtain a composite feature set, and optimizing the first-optimized outlier detection parameters based on the features of the composite feature set to obtain second-optimized outlier detection parameters; and performing abnormal behavior identification on the valid data to be detected based on the second-optimized outlier detection parameters to obtain abnormal access identification results.
2. The method for identifying abnormal access to big data according to claim 1, characterized in that, The step of determining the valid data to be detected for the target object based on the global dataset, the outliers, and the preset normal behavior database includes: determining the data to be detected for the target object based on the global dataset, the outliers, and the preset normal behavior database; and using a preset wavelet transform algorithm to denoise the data to be detected to obtain the valid data to be detected.
3. The method for identifying abnormal access to big data according to claim 2, characterized in that, The step of determining the data to be detected for the target object based on the global dataset, the outliers, and the preset normal behavior database includes: comparing the access patterns of the behavioral data in the global dataset and the outliers with the standard access patterns in the preset normal behavior database; if the comparison results show that the access patterns are different, then the user to whom the corresponding data belongs is determined as the target object.
4. The method for identifying abnormal access to big data according to claim 1, characterized in that, The step of obtaining basic features and effective abnormal behavior features from the valid data to be detected includes: extracting basic features from the valid data to be detected and constructing a state model based on the basic features; and traversing the state model using a preset depth-first search algorithm to extract the effective abnormal behavior features.
5. The method for identifying abnormal access to big data according to claim 1, characterized in that, The method further includes: acquiring second behavioral data of the target user within a historical time period, constructing a multi-dimensional behavioral vector based on the second behavioral data, and constructing the preset normal behavior database based on the multi-dimensional behavioral vector.
6. The method for identifying abnormal access to big data according to claim 1, characterized in that, The step of identifying abnormal behavior in the valid data to be detected based on the second optimized outlier detection parameters to obtain abnormal access identification results includes: when abnormal behavior is determined, classifying the abnormal behavior according to a preset abnormal category, wherein the preset abnormal category includes abnormal login address, abnormal login time, or abnormal operation command.
7. A device for identifying abnormal access to big data, characterized in that, The device includes: a data acquisition module, configured to acquire first behavioral data of a target user within a target time period from the working log of a target device, and construct a global dataset based on the first behavioral data; a cluster determination module, configured to perform clustering training on the global dataset using a preset clustering algorithm to divide the addresses involved in the interaction in the global dataset into multiple target clusters, and optimize outlier detection parameters based on the features of the target clusters to obtain the first optimized outlier detection parameters; and a detection determination module, configured to perform outlier detection on the global dataset based on the first optimized outlier detection parameters to obtain outlier points, and determine the outlier points based on the global data. The system uses a database of outliers and a preset database of normal behavior to determine the valid data to be detected for the target object. A feature fusion module is used to obtain basic features and valid abnormal behavior features from the valid data to be detected, and to integrate the basic features and valid abnormal behavior features using a preset feature fusion algorithm to obtain a composite feature set. Based on the features of the composite feature set, the system optimizes the first-optimized outlier detection parameters to obtain a second-optimized outlier detection parameter. An anomaly identification module is used to identify abnormal behavior in the valid data to be detected based on the second-optimized outlier detection parameter to obtain an abnormal access identification result.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.