Data processing method and device, equipment and storage medium

Abstract

The application discloses a data processing method and device, equipment and a storage medium. The method comprises the following steps: determining M difference data of an i-th user in M time periods according to log records generated by operation behaviors of N users, wherein the j-th difference data is used for representing a difference between the operation behavior of the i-th user in the j-th time period and the operation behavior of the i-th user in the M time periods, and the M difference data is used for determining an initial first analysis result data of whether the operation behavior of the i-th user in a target time period has an abnormal operation behavior; and correcting the first analysis result data by using reward feedback data generated by a second analysis result data of the operation behavior of the i-th user in the j-th time period and user feedback results of the operation behavior, so as to obtain third analysis result data used for determining whether the operation behavior of the i-th user in the target time period has an abnormal operation behavior. In this way, the accuracy of user abnormal behavior identification is improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular relates to a data identification method, device, equipment and storage medium. Background Art

[0002] With the continuous development of Internet technology and the continuous enhancement of digital capabilities, cyber attacks or data leaks from inside and outside the system that threaten data security have followed. Therefore, how to identify and intercept abnormal operational behaviors in user behavior data and ensure data security has become a focus of attention.

[0003] In related technologies, deep learning network models can be trained using massive amounts of historical user behavior data. Based on this trained model, the model can be used to identify abnormal operations in the current user behavior data. This indicates that the quality of user behavior data in this approach directly impacts the accuracy of the results of identifying abnormal operations. Missing or erroneous user behavior data can lead to biased identification of abnormal operations, increasing the false positive rate. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device, and storage medium, which can solve the problems of low accuracy and high false alarm rate in identifying abnormal user behavior in related technologies.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, which may include:

[0006] Get the log records generated by N users' operation behaviors;

[0007] Based on the log records, determine a difference data set of the i-th user among N users within M time periods. The difference data set includes M difference data. The j-th difference data among the M difference data is used to characterize the difference between the operation behavior of the i-th user in the j-th time period and the operation behavior of the i-th user in the M time periods. The difference data set is used to determine the first analysis result data. The first analysis result data is the initial analysis result for determining whether the operation behavior of the i-th user in the target time period contains abnormal operation behavior, j∈[1,M];

[0008] Generate reward feedback data for the i-th user's operation behavior in M ​​time periods based on the second analysis result data of the i-th user's operation behavior in the j-th time period and the user feedback result of the i-th user on the operation behavior in the j-th time period;

[0009] The first analysis result data is corrected by rewarding feedback data to obtain third analysis result data, and the third analysis result data is used to determine whether there is abnormal operation behavior in the operation behavior of i users within the target time period.

[0010] In a second aspect, an embodiment of the present application provides a data processing device, which may include:

[0011] The acquisition module is used to obtain log records generated by N users' operation behaviors;

[0012] A determination module is configured to determine, based on log records, a difference data set of the i-th user among N users within M time periods, the difference data set including M difference data, wherein the j-th difference data among the M difference data is used to characterize the difference between the operation behavior of the i-th user within the j-th time period and the operation behavior of the i-th user within the M time periods, and the difference data set is used to determine first analysis result data, which is an initial analysis result for determining whether the operation behavior of the i-th user within the target time period contains abnormal operation behavior, j∈[1,M];

[0013] A generation module, configured to generate reward feedback data for the operation behavior of the i-th user in the M time periods based on the second analysis result data of the operation behavior of the i-th user in the j-th time period and the user feedback result of the i-th user on the operation behavior in the j-th time period;

[0014] The correction module is used to correct the first analysis result data through reward feedback data to obtain third analysis result data, and the third analysis result data is used to determine whether there is any abnormal operation behavior in the operation behavior of i users within the target time period.

[0015] In a third aspect, an embodiment of the present application provides a computer device, the computer device comprising: a processor and a memory storing computer program instructions;

[0016] When the processor executes the computer program instructions, the data processing method shown in the first aspect is implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a computer storage medium having computer program instructions stored thereon, which, when executed by a processor, implements the data processing method shown in the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the data processing method shown in the first aspect.

[0019] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the data processing method shown in the first aspect.

[0020] The data processing method, apparatus, device and storage medium of the embodiments of the present application determine, based on the log records generated by the operation behaviors of N users, a difference data set of the i-th user among the N users within M time periods, the difference data set including M difference data, the j-th difference data among the M difference data being used to characterize the difference between the operation behaviors of the i-th user in the j-th time period and the operation behaviors of the i-th user in the M time periods, the difference data set being used to determine first analysis result data, the first analysis result data being the initial analysis result for determining whether the operation behaviors of the i-th user in the target time period have abnormal operation behaviors, in this way, the unstructured log records can be converted into structured difference data, and regular data in the log records can be moved to the target time period. In addition to obtaining difference data between the user's operation behavior in certain time periods and the operation behavior in most time periods, while avoiding interference from regular data on the difference data, more complete and accurate difference data can be obtained through log recording, thereby improving data quality; then, based on the second analysis result data of the i-th user's operation behavior in the j-th time period and the user feedback results of the i-th user on the operation behavior in the j-th time period, reward feedback data of the i-th user's operation behavior in M ​​time periods are generated; the first analysis result data is corrected using the reward feedback data to obtain third analysis result data, which is used to determine whether the i-th user's operation behavior in the target time period has abnormal operation behavior. In this way, the reward feedback data can be determined based on the user feedback results of the operation behavior, and the reward feedback data can be used to optimize the initially determined first analysis result data, further improving the accuracy of determining whether the user's operation behavior in the target time period has abnormal operation behavior, thereby reducing the false alarm rate of whether the operation behavior has abnormal operation behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of log recording of a data processing method provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of structured data of a data processing method provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of determining difference data in a data processing method provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram showing a comparison of false alarms of abnormal operation behaviors of a data processing method provided in an embodiment of the present application;

[0028] Figure 7 is a structural diagram of a data processing device provided by an embodiment of the present application;

[0029] Figure 8 It is a structural diagram of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0030] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0031] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0032] The acquisition, storage, use, and processing of data (including but not limited to the features and information herein) in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0033] In order to solve the above technical problems, the embodiments of the present application provide a data processing system, method, apparatus, computer equipment and storage medium.

[0034] Based on this, the following will be combined with the Figures 1 to 8 , describes in detail the data processing system, method, apparatus, computer equipment and storage medium of the embodiments of the present application. It should be noted that these embodiments are not intended to limit the scope of disclosure of the present application.

[0035] First, combine Figure 1 The data processing system provided in the embodiments of the present application is described in detail.

[0036] Figure 1 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application.

[0037] like Figure 1 As shown, in some embodiments of the present application, the data processing system 10 provided in the embodiments of the present application may include a log access module and an automated audit module.

[0038] The logging module can be used to establish a connection with at least one service system, such as Service 1, Service 2, and Service 3, and to obtain log records generated by the operation behavior of users of the application system that requires data security audits and compliance checks through traffic or log access. Specifically, the logging module is used to obtain log records and traffic records generated by the operation behavior of users in the service system, wherein the log records are structured query language (SQL), syslog, etc.

[0039] The automated audit module can be deployed on service platforms such as security operation platforms and log management platforms to achieve systematic docking and be used to perform structured parsing of log records to obtain the structured data of the i-th user among N users. The structured data includes the feature data of the i-th user's operation behavior divided according to P preset features of manually configured rules, where P is an integer greater than 1; the structured data of the i-th user is aggregated according to the time sequence of the operation behavior to obtain the structured data set of the i-th user in M ​​time periods. The structured data set includes M aggregated data, which is the operation behavior of the i-th user in each of the M time periods. Aggregated feature data of the operation behavior of the i-th user in M ​​time periods; based on the structured data set, determining a difference data set of the i-th user in M ​​time periods, the difference data set including M difference data, the j-th difference data in the M difference data being used to characterize the difference between the operation behavior of the i-th user in the j-th time period and the operation behavior of the i-th user in the M time periods, the difference data set being used to determine first analysis result data, the first analysis result data being the initial analysis result for determining whether the operation behavior of the i-th user in the target time period contains abnormal operation behavior, the start time of the target time period being after the end time of each of the M time periods, j∈[1,M]. In addition, the automated audit module can also be used to adaptively optimize the learning strategy, that is, based on the second analysis result data of the i-th user's operation behavior in the j-th time period and the user feedback result of the i-th user on the operation behavior in the j-th time period, generate reward feedback data of the i-th user's operation behavior in the M time periods; correct the first analysis result data using the reward feedback data to obtain third analysis result data, the third analysis result data being used to determine whether the operation behavior of the i-th user in the target time period contains abnormal operation behavior. In addition, the automated audit module can also be used in the user behavior data processing process, that is, when the analysis result value of the third analysis result data is greater than or equal to the preset analysis result value, it is determined that there are abnormal operation behaviors in the operation behaviors of i users within the target time period; when the analysis result value of the third analysis result data is less than the preset analysis result value, it is determined that there are no abnormal operation behaviors in the operation behaviors of i users within the target time period.

[0040] Based on this, the data processing system 10 can also include a reminder handling module, wherein the reminder handling module can include an application system and a work order reminder handling unit, which can be used to record abnormal operation behaviors of the i users determined by the automated audit module output within the target time period, that is, in the event of an abnormal situation, it can be output through an interface to the application system that needs to carry out data security audits and compliance checks for handling, or connected to the work order system to issue a reminder, and the user's handling situation (whether it is abnormal) can be fed back to the automated audit module to further improve the accuracy of subsequent abnormal reminders. Specifically, the reminder handling module can prompt the application system and the work order reminder handling unit based on the abnormal operation behaviors of the i users determined by the automated audit module within the target time period, so that the application system and the work order reminder handling unit intercept their abnormal operation behaviors and manage the users corresponding to their abnormal operation behaviors.

[0041] It should be noted that, based on the above data processing system 10, the data processing method provided in the embodiment of the present application can be applied to the application scenario of automatically auditing the data generated by the user's operating behavior in the service system.

[0042] Therefore, the data processing system provided in the embodiment of the present application can, through intelligent analysis and identification of individual behavior patterns when using the service system, and the introduction of an adaptive learning optimization mechanism, reduce the false alarm rate while discovering potential abnormal operating behaviors, thereby improving the accuracy of determining whether the user's operating behavior within the target time period contains abnormal operating behaviors, thereby improving the accuracy of discovering potential security threats.

[0043] Based on this, in order to better illustrate the contents of the embodiments of this application, the following are respectively combined Figures 2 to 8 A data processing method provided in an embodiment of the present application is described as follows.

[0044] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application.

[0045] like Figure 2 As shown, the data processing method can be applied to a data processing system, and the data processing method may specifically include the following steps:

[0046] Step 210, obtaining log records generated by the operation behaviors of N users, where N is an integer greater than 1; Step 220, determining a difference data set of the i-th user in the N users within M time periods based on the log records, the difference data set including M difference data, the j-th difference data in the M difference data being used to characterize the difference between the operation behaviors of the i-th user in the j-th time period and the operation behaviors of the i-th user in the M time periods, the difference data set being used to determine first analysis result data, the first analysis result data being an initial analysis result for determining whether the operation behaviors of the i-th user in the target time period have abnormal operation behaviors, j∈[1,M]; Step 230, based on the The second analysis result data of the operation behavior of the i-th user in the j-th time period and the user feedback result of the i-th user on the operation behavior in the j-th time period are used to generate reward feedback data of the operation behavior of the i-th user in the M-th time period. Here, the reward feedback data is used to reflect the consistency between the abnormality detection result of the user behavior of the i-th user in the j-th time period and the user feedback result of the i-th user on the user behavior in the j-th time period; step 240, using the reward feedback data, correct the first analysis result data to obtain third analysis result data, and the third analysis result data is used to determine whether the operation behavior of the i-th user in the target time period has abnormal operation behavior.

[0047] In this way, based on the log records generated by the operation behaviors of N users, the difference data set of the i-th user in the N users within M time periods is determined. The difference data set includes M difference data. The j-th difference data in the M difference data is used to characterize the difference between the operation behavior of the i-th user in the j-th time period and the operation behavior of the i-th user in the M time period. The difference data set is used to determine the first analysis result data. The first analysis result data is the initial analysis result for determining whether there is abnormal operation behavior in the operation behavior of the i-th user in the target time period. In this way, the unstructured log records can be converted into structured difference data, and the regular data in the log records can be removed to obtain the user's operation behavior in certain time periods. The difference data between the operation behavior in the target time period and the operation behavior in most time periods is obtained through log recording, which avoids the interference of regular data on the difference data. At the same time, more complete and accurate difference data can be obtained through log recording, thereby improving data quality. Then, based on the second analysis result data of the operation behavior of the i-th user in the j-th time period and the user feedback result of the i-th user on the operation behavior in the j-th time period, reward feedback data of the operation behavior of the i-th user in the M time periods is generated. The first analysis result data is corrected by the reward feedback data to obtain third analysis result data, which is used to determine whether the operation behavior of the i-th user in the target time period has abnormal operation behavior. In this way, the reward feedback data can be determined by the user feedback result of the operation behavior, and the reward feedback data can be used to optimize the initially determined first analysis result data, further improving the accuracy of determining whether the user's operation behavior in the target time period has abnormal operation behavior, thereby reducing the false alarm rate of whether the operation behavior has abnormal operation behavior.

[0048] The above steps are described in detail below.

[0049] First, regarding step 210, in some embodiments of the present application, log recording may refer to recording and analyzing log data generated during system operation, so as to track and review the use and behavior of the system, help discover abnormal operation behavior, potential security threats, and provide key evidence for subsequent investigations. Therefore, the log records generated by the user's operation behavior can be obtained from the service system through system integration, unified storage, and manual pulling. The service system can be the service system of the entire application or the service system of a certain function under the application. Here, the log record sample is as follows: Figure 3 As shown, it includes logs such as user ID, user name, department, operation module, operation type such as login, task execution, download, operation time, etc.

[0050] Next, step 220 is involved. In some embodiments of the present application, step 220 may specifically include steps 2201 to 2203.

[0051] Step 2201: Perform structured parsing on the log records to obtain structured data for the i-th user among N users. The structured data includes feature data of the i-th user's operational behavior divided according to P preset features, where P is an integer greater than 1. The preset features may be determined by business scenarios and / or data characteristics.

[0052] For example, unstructured log records are structured and parsed to extract feature data. Here, feature data can be extracted using tools that support SPL syntax or scripts such as Python. Feature engineering needs to be customized and optimized according to business scenarios and data characteristics. Figure 4 As shown in the figure, in actual applications, structured data can include characteristic data of various preset features such as user name, department, operation type (query / login / download), task time, etc. Among them, the level and data category of the data operation object will be parsed from the script executing the task, including the specific query field information and the number of affected rows, and then associated with the corresponding data level and category in the hierarchical classification field list table.

[0053] Step 2202: Aggregate the structured data of the i-th user according to the chronological order of the operation behaviors to obtain a structured data set of the i-th user in M ​​time periods. The structured data set includes M aggregated data, which are aggregated feature data of the i-th user's operation behaviors in each of the M time periods.

[0054] For example, if M is 3, the operation types of the i-th user, such as the number of queries, are aggregated according to the chronological order of the operation behaviors to obtain a structured data set of the i-th user in three time periods. The structured data set includes three aggregated data, such as the number of queries of the i-th user from 10:00 a.m. to 11:00 a.m. on April 1, which is 5; the number of queries of the i-th user from 11:00 a.m. to 12:00 p.m. on April 1, which is 8; and the number of queries of the i-th user from 1:00 p.m. to 2:00 p.m. on April 1, which is 10.

[0055] Step 2203: Determine the difference data set of the i-th user in M ​​time periods based on the structured data set.

[0056] Furthermore, step 2203 may specifically include step 22031 and step 22032.

[0057] Step 22031, based on the aggregation results of the characteristic data of the operation behavior of the i-th user in each of the M time periods, generate the periodic data and trend data of the i-th user in the M time periods, the periodic data includes the characteristic data of the operation behavior in the M time periods showing periodic changes, and the trend data includes the characteristic data of the operation behavior in the M time periods showing a changing trend.

[0058] Specifically, the M time periods include a first time period and a second time period. Based on this, step 22031 may specifically include step 220311 and step 220312.

[0059] Step 220311: Generate trend data based on the aggregation results of the feature data of the operation behavior of the i-th user in the first time period and the aggregation results of the feature data of the operation behavior of the i-th user in the second time period.

[0060] For example, still taking the example shown in step 2202, the first time period can be the first time period of M time periods, and the second time period can be the last time period of M time periods. Based on the aggregation result of the characteristic data of the operation behavior in the first time period, such as 5, and the aggregation result of the characteristic data of the operation behavior in the last time period, such as 10, the trend data can be (5+10) / 2=7.5.

[0061] Step 220312: Determine the average of the cumulative difference between the aggregation results of the feature data of the operation behavior of the i-th user in each of the M time periods and the trend data as periodic data.

[0062] Exemplarily, still taking the example shown in step 220311, if M is 2, the periodic data may be |(3-7.5)+(5-7.5)+(10-7.5)| / 3=3.12.

[0063] Step 22032: Remove the periodic data and trend data from the M aggregated data respectively to obtain the difference data between the operation behavior of the i-th user in each time period and the operation behavior of the i-th user in the M time periods.

[0064] For example, Figure 5 As shown, since the aggregated result, i.e., the raw data, is broken down into three parts, including trend data, periodic data, and difference data, the periodic data and trend data in the aggregated result are removed to obtain the difference data, i.e., the residual. This prevents normal trends and periods from affecting the results and reduces the false positive rate. In this embodiment, the raw data is the number of successful user query tasks per hour, grouped by user and time.

[0065] Therefore, the embodiment of the present application provides user behavior data processing rules, namely, unstructured log records are parsed through preset feature matching to form structured data, and then sorted by operation behavior time to form a time series, which is broken down into three parts: trend data, periodic data and difference data. Finally, the trend data and periodic data in the aggregation results within each time period are eliminated, and only the difference data is retained to avoid its interference and reduce the false alarm rate.

[0066] Furthermore, regarding step 230 , in some embodiments of the present application, step 230 may specifically include step 2301 and step 2302 .

[0067] Step 2301: Based on the second analysis result data of the i-th user's operation behavior in the j-th time period and the user feedback result of the i-th user on the operation behavior in the j-th time period, sub-reward feedback data for the i-th user's operation behavior in the j-th time period is obtained. Here, in the embodiment of the present application, the second analysis result data can be obtained by identifying the operation behavior of the i-th user in the j-th time period using an abnormal operation behavior recognition model.

[0068] Furthermore, step 2301 may specifically include step 23011 and step 23012.

[0069] Step 23011: When the second analysis result data matches the user feedback result, the first preset reward feedback value is determined as the sub-reward feedback data of the operation behavior of the i-th user in the j-th time period.

[0070] Step 23012: When the second analysis result data does not match the user feedback result, the second preset reward feedback value is determined as the sub-reward feedback data of the operation behavior of the i-th user in the j-th time period; wherein the second preset reward feedback value is less than the first preset reward feedback value.

[0071] For example, an adaptive learning optimization mechanism is introduced to optimize the criteria for subsequent behavior judgment based on user feedback. The value function is defined as shown in the following formula (1) to calculate the sub-reward feedback data:

[0072]

[0073] Among them, f t (a) represents the second analysis result data obtained for any one of the M time periods, such as the operation behavior a in the j-th time period (in this embodiment, the second analysis result data has two states, namely, the operation behavior a in the j-th time period has abnormal operation behavior and the operation behavior a in the j-th time period does not have abnormal operation behavior), R iA represents the preset reward feedback value for operation behavior a in the jth time period (in this embodiment, if the second analysis result data is consistent with the user feedback result, the first preset reward feedback value is 100, and the second preset reward feedback value is -1000). i represents the operation behavior in the jth time period, and the denominator is the total number of times operation behavior a occurs before the target time period.

[0074] Step 2302: Determine the target sub-reward feedback data that meets the preset conditions in the sub-reward feedback data of the operation behavior of the i-th user in the j-th time period as the reward feedback data of the operation behavior of the i-th user in the M-th time period.

[0075] For example, the target sub-reward feedback data that meets the preset conditions in the sub-reward feedback data of the operation behavior of the i-th user in the j-th time period is screened with reference to the following formula (2):

[0076] Adaptive learning strategy function: A t =argmaxf t (a) (2)

[0077] Among them, A of the operation behavior in the jth time period t , is the sub-reward feedback data obtained before the target time period.

[0078] Therefore, the embodiment of the present application proposes an adaptive optimization mechanism combined with user feedback: a function as shown in formula (1) is defined, where f t (a) represents the preset reward feedback value of the sub-reward feedback data obtained by operation behavior a in the target time period, to avoid a one-sided situation (because the normal state is much higher than the abnormal state in actual conditions). When the predicted situation in M ​​time periods is consistent with the actual situation, a reward is given. If it is inconsistent, a penalty is imposed. In the target time period, such as the M+1th time period, the behavior with the target sub-reward feedback data with the highest reward is selected. Based on the user's feedback, the criteria for subsequent behavior judgment are optimized to improve the accuracy of correcting the first analysis result.

[0079] Then, step 240 is involved. In some embodiments of the present application, before step 140, first analysis result data needs to be generated. Based on this, the data processing method provided in the embodiment of the present application may further include step 250:

[0080] The first analysis result data is generated based on the M difference data through the T-test algorithm.

[0081] For example, H0 can be used to indicate that there is no abnormal point; H1 can be used to indicate that there is an abnormal point. In this way, the T-test algorithm can be used to determine whether the operation behavior in the target time period is abnormal. ′ tThe analysis result data (-1 indicates no abnormality, 1 indicates abnormality) is the first analysis result data.

[0082] Based on this, step 250 may specifically include steps 2501 to 2503.

[0083] Step 2501: Calculate the difference between the first difference data in the M difference data and the mean value of M-1 difference data in the M difference data except the first difference data.

[0084] Step 2502 : Generate abnormality degree data of the first difference data based on the difference degree value and the standard error of the M difference data. The abnormality degree data is used to characterize whether there is abnormal data in the first difference data.

[0085] Step 2503: determine the abnormality degree data as the first analysis result data.

[0086] Based on this, in some embodiments of the present application, step 140 may specifically include steps 1401 to 1403.

[0087] Step 1401: Adjust the weight coefficient of the abnormal operation behavior recognition model through reward feedback data to obtain a correction coefficient. The abnormal operation behavior recognition model in the embodiment of the present application is obtained by training historical abnormal operation behaviors and is a neural network model.

[0088] Step 1402: Generate converted standard analysis result data of the first analysis result data according to the weight coefficient and the first analysis result data.

[0089] Step 1403: Generate third analysis result data based on the correction coefficient and the converted standard analysis result data.

[0090] For example, within the target time period, the third analysis result data provided by the abnormal operation behavior recognition model can be expressed as follows:

[0091] A ′ t ′ =θA ′ t +(1-θ)A t (3)

[0092] Among them, θ is the weight coefficient (θ∈[0,1]) in the abnormal operation behavior recognition model, where θA ′ t is the converted standard analysis result data, (1-θ)A t is the correction coefficient.

[0093] In this way, after step 140, the data processing method provided in the embodiment of the present application may further include steps 1601 and 1602.

[0094] Step 1601: When the analysis result value of the third analysis result data is greater than or equal to the preset analysis result value, it is determined that the operation behaviors of the i users within the target time period have abnormal operation behaviors.

[0095] Step 1602 : If the analysis result value of the third analysis result data is less than the preset analysis result value, it is determined that there is no abnormal operation behavior in the operation behaviors of the i users within the target time period.

[0096] For example, when A ′ t ′ If it is greater than or equal to 0, there is an exception. ′ t ′ If the value is less than 0, there is no abnormality. Based on the analysis of each user's real behavior and taking into account the user's real feedback, we can adapt to the ever-changing user behavior patterns and further reduce the false alarm rate.

[0097] Therefore, the embodiment of the present application combines the status determination method of user behavior and user feedback, and considers the method of real user feedback to adapt to the ever-changing user behavior patterns and further reduce the false alarm rate.

[0098] It should be noted that the data processing method provided in the embodiment of the present application is evaluated by using the dataset AMAD anomaly detection data. The dataset comes from the real traffic log records of the shopping application recommendation system, and abnormal instances are randomly introduced into the dataset through manual perturbation. The dataset includes user name, time, user behavior characteristics (click product id), and behavior label (normal / abnormal). There are 6151 users and 75963 operation behaviors in the data, of which 73649 are normal operation behaviors and 2314 are abnormal operation behaviors. Figure 6 As shown, compared with the total false alarm rate of the threshold judgment method in the prior art and the method of eliminating trend and periodic judgment and adding adaptive learning in the embodiment of the present application, the data method provided by the embodiment of the present application can avoid the subjectivity of manual threshold judgment, reduce the false alarm rate of abnormal operation behavior identification, and discover potential security threats.

[0099] The present application also provides a data processing device, specifically in combination with Figure 7 Provide detailed explanation.

[0100] Figure 7 It is a structural diagram of a data processing device provided by an embodiment of the present application.

[0101] In some embodiments of the present application, Figure 7 The data processing device shown can be set in the computing device provided in the embodiment of the present application.

[0102] like Figure 7 As shown, the data processing device 70 may specifically include:

[0103] An acquisition module 701 is used to acquire log records generated by N users' operation behaviors;

[0104] Determination module 702 is used to determine, based on log records, a difference data set of the i-th user among N users within M time periods, where the difference data set includes M difference data, where the j-th difference data among the M difference data is used to represent the difference between the operation behavior of the i-th user in the j-th time period and the operation behavior of the i-th user in the M time periods, and the difference data set is used to determine first analysis result data, where the first analysis result data is an initial analysis result for determining whether the operation behavior of the i-th user in the target time period contains abnormal operation behavior, where j∈[1,M].

[0105] A generating module 703 is configured to generate reward feedback data for the operation behavior of the i-th user in the M time periods based on the second analysis result data of the operation behavior of the i-th user in the j-th time period and the user feedback result of the i-th user on the operation behavior in the j-th time period;

[0106] The correction module 704 is used to correct the first analysis result data through the reward feedback data to obtain third analysis result data, and the third analysis result data is used to determine whether there is any abnormal operation behavior in the operation behavior of i users within the target time period.

[0107] The data processing device 70 in the embodiment of the present application is described in detail below.

[0108] In some embodiments of the present application, the determination module 702 may be specifically configured to perform structured parsing on the log record to obtain structured data of the i-th user among N users, where the structured data includes feature data of the operation behavior of the i-th user divided according to P preset features, where P is an integer greater than 1.

[0109] Aggregate the structured data of the i-th user according to the chronological order of the operation behaviors to obtain a structured dataset of the i-th user in M ​​time periods. The structured dataset includes M aggregated data, which are the aggregated feature data of the i-th user's operation behaviors in each of the M time periods.

[0110] Based on the structured data set, the difference data set of the i-th user in M ​​time periods is determined.

[0111] In some embodiments of the present application, the determination module 702 may be specifically configured to generate periodic data and trend data of the i-th user in the M time periods based on an aggregation result of feature data of the operation behavior of the i-th user in each of the M time periods, wherein the periodic data includes feature data of periodic changes in the feature data of the operation behavior in the M time periods, and the trend data includes feature data of a changing trend in the feature data of the operation behavior in the M time periods.

[0112] Periodic data and trend data are removed from the M aggregated data respectively to obtain the difference between the operation behavior of the i-th user in each time period and the operation behavior of the i-th user in the M time periods.

[0113] In some embodiments of the present application, the determination module 702 may be specifically configured to, when the M time periods include a first time period and a second time period, generate trend data based on an aggregation result of feature data of the operation behavior of the i-th user in the first time period and an aggregation result of feature data of the operation behavior of the i-th user in the second time period;

[0114] The average amount of the accumulated difference between the aggregation result of the feature data of the operation behavior of the i-th user in each of the M time periods and the trend data is determined as the periodic data.

[0115] In some embodiments of the present application, the generating module 703 may also be configured to generate first analysis result data based on M difference data using a T-test algorithm.

[0116] In some embodiments of the present application, the generating module 703 may be specifically configured to calculate, based on the M difference data, a difference degree value between a first difference data in the M difference data and an average value of M-1 difference data in the M difference data excluding the first difference data;

[0117] Generate abnormality degree data of the first difference data based on the difference degree value and the standard error of the M difference data, where the abnormality degree data is used to indicate whether the first difference data has abnormal data;

[0118] The abnormality degree data is determined as the first analysis result data.

[0119] In some embodiments of the present application, the generation module 703 may be specifically configured to obtain sub-reward feedback data for the operation behavior of the i-th user in the j-th time period based on the second analysis result data of the operation behavior of the i-th user in the j-th time period and the user feedback result of the i-th user on the operation behavior in the j-th time period;

[0120] The target sub-reward feedback data that meets the preset conditions in the sub-reward feedback data of the operation behavior of the i-th user in the j-th time period is determined as the reward feedback data of the operation behavior of the i-th user in the M-th time period.

[0121] In some embodiments of the present application, the generation module 703 may be specifically configured to, when the second analysis result data matches the user feedback result, determine the first preset reward feedback value as the sub-reward feedback data for the operation behavior of the i-th user in the j-th time period;

[0122] If the second analysis result data does not match the user feedback result, the second preset reward feedback value is determined as the sub-reward feedback data of the operation behavior of the i-th user in the j-th time period;

[0123] The second preset reward feedback value is smaller than the first preset reward feedback value.

[0124] In some embodiments of the present application, the correction module 704 may be specifically configured to adjust the weight coefficient of the abnormal operation behavior recognition model using reward feedback data to obtain a correction coefficient;

[0125] Generate converted standard analysis result data of the first analysis result data according to the weight coefficient and the first analysis result data;

[0126] Based on the correction coefficient and the converted standard analysis result data, third analysis result data is generated.

[0127] In some embodiments of the present application, the determining module 702 may be specifically configured to determine that the operation behaviors of the i users within the target time period are abnormal when the analysis result value of the third analysis result data is greater than or equal to a preset analysis result value;

[0128] When the analysis result value of the third analysis result data is less than the preset analysis result value, it is determined that there is no abnormal operation behavior in the operation behaviors of the i users within the target time period.

[0129] Therefore, the data processing device of the embodiment of the present application determines the difference data set of the i-th user in the N users within M time periods based on the log records generated by the operation behaviors of N users. The difference data set includes M difference data. The j-th difference data in the M difference data is used to characterize the difference between the operation behavior of the i-th user in the j-th time period and the operation behavior of the i-th user in the M time periods. The difference data set is used to determine the first analysis result data. The first analysis result data is the initial analysis result for determining whether the operation behavior of the i-th user in the target time period has abnormal operation behavior. In this way, the unstructured log records can be converted into structured difference data, and the regular data in the log records can be removed to obtain the user. The difference data between the user's operation behavior in certain time periods and the operation behavior in most time periods can be obtained through log recording, while avoiding the interference of regular data on the difference data. At the same time, more complete and accurate difference data can be obtained through log recording, thereby improving data quality. Then, based on the second analysis result data of the operation behavior of the i-th user in the j-th time period and the user feedback results of the i-th user on the operation behavior in the j-th time period, reward feedback data of the operation behavior of the i-th user in the M time periods are generated. The first analysis result data is corrected by the reward feedback data to obtain third analysis result data, and the third analysis result data is used to determine whether the operation behavior of the i-th user in the target time period has abnormal operation behavior. In this way, the reward feedback data can be determined based on the user feedback results of the operation behavior, and the reward feedback data can be used to optimize the initially determined first analysis result data, further improving the accuracy of determining whether the user's operation behavior in the target time period has abnormal operation behavior, thereby reducing the false alarm rate of whether the operation behavior has abnormal operation behavior.

[0130] Based on the same inventive concept, the present application also provides a computer device. Figure 8 Provide detailed explanation.

[0131] Figure 8 It is a structural diagram of a computer device provided in one embodiment of the present application.

[0132] like Figure 8 As shown, the computer device may include at least one of the following involved in the embodiments of the present application: an electronic device, a server. The computer device may include a processor 801 and a memory 802 storing computer program instructions.

[0133] Specifically, the processor 801 may include a central processing unit (CPU), or an application specific integrated circuit (ASTC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0134] The memory 802 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 802 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 802 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 802 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 802 is a non-volatile solid-state memory. In a specific embodiment, the memory 802 includes a solid-state memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0135] The processor 801 implements any one of the data processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 802 .

[0136] In one example, the computer device may further include a communication interface 803 and a bus 810. Figure 8 As shown, the processor 801, the memory 802, and the communication interface 803 are connected via a bus 810 and communicate with each other.

[0137] The communication interface 803 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0138] Bus 810 comprises hardware, software or both, and the parts of flow control device are coupled to each other.For example, and not limitation, bus can comprise accelerated graphics port (AGP) or other graphics bus, enhanced industry standard system (ETSA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard system (TSA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel system (MCA) bus, peripheral component interconnection (PCT) bus, PCT-Express (PCT-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more above these combination.In suitable case, bus 810 can comprise one or more buses.Although the present application embodiment describes and shows specific bus, the application considers any suitable bus or interconnection.

[0139] The data processing device can execute the data processing method in the embodiment of the present application, thereby realizing the combination Figures 1 to 8 Described data processing method and device.

[0140] In addition, in conjunction with the data processing methods in the above embodiments, embodiments of the present application may provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the data processing methods in the above embodiments is implemented.

[0141] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0142] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in unit, a function card or the like. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0143] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0144] The above is only a specific implementation method of the present application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited to this. Any technician familiar with this technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the scope of protection of this application.

Claims

1. A data processing method, comprising: Get the log records generated by N users' operation behaviors; Determine, based on the log records, a difference data set of an i-th user among the N users within M time periods, the difference data set including M difference data, the j-th difference data among the M difference data being used to characterize the difference between an operation behavior of the i-th user within the j-th time period and an operation behavior of the i-th user within the M time periods, the difference data set being used to determine first analysis result data, the first analysis result data being an initial analysis result for determining whether the operation behavior of the i-th user within a target time period contains abnormal operation behavior, j∈[1,M]; generating reward feedback data for the operation behavior of the i-th user in the M time periods based on the second analysis result data of the operation behavior of the i-th user in the j-th time period and the user feedback result of the i-th user on the operation behavior in the j-th time period; The first analysis result data is corrected by using the reward feedback data to obtain third analysis result data, and the third analysis result data is used to determine whether there is abnormal operation behavior in the operation behavior of the i users within the target time period.

2. The method according to claim 1, wherein Determining, based on the log records, a difference data set of the i-th user among the N users within M time periods includes: Performing structured parsing on the log record to obtain structured data of an i-th user among the N users, the structured data including feature data of the operation behavior of the i-th user divided according to P preset features, where P is an integer greater than 1; Aggregate the structured data of the i-th user according to the chronological order of the operation behaviors to obtain a structured dataset of the i-th user in M ​​time periods, wherein the structured dataset includes M aggregated data, and the aggregated data is aggregated feature data of the operation behaviors of the i-th user in each of the M time periods; Based on the structured data set, a difference data set of the i-th user in M ​​time periods is determined.

3. The method according to claim 2, wherein: The determining, based on the structured data set, a difference data set of the i-th user in M ​​time periods includes: generating, based on an aggregation result of the feature data of the operation behavior of the i-th user in each of the M time periods, periodic data and trend data of the i-th user in the M time periods, the periodic data including feature data of the operation behavior in the M time periods showing periodic changes in the operation behavior, and the trend data including feature data of the operation behavior in the M time periods showing a changing trend in the operation behavior; The periodic data and the trend data are removed from the M aggregated data respectively to obtain data on the difference between the operation behavior of the i-th user in each time period and the operation behavior of the i-th user in the M time periods.

4. The method according to claim 3, wherein: The M time periods include a first time period and a second time period; The generating of periodic data and trend data of the i-th user in the M time periods based on the aggregation result of the feature data of the operation behavior of the i-th user in each of the M time periods includes: generating the trend data based on an aggregation result of the feature data of the operation behavior of the i-th user in the first time period and an aggregation result of the feature data of the operation behavior of the i-th user in the second time period; The average amount of the accumulated difference between the aggregation result of the feature data of the operation behavior of the i-th user in each of the M time periods and the trend data is determined as the periodic data.

5. The method according to claim 1, wherein Before correcting the first analysis result data using the reward feedback data to obtain the third analysis result data, the method further includes: The first analysis result data is generated based on the M difference data using a T-test algorithm.

6. The method according to claim 5, wherein: The generating of the first analysis result data based on the M difference data by using a T-test algorithm includes: Calculating, based on the M difference data, a difference degree value between a first difference data in the M difference data and an average value of M-1 difference data in the M difference data except the first difference data; generating abnormality degree data of the first difference data based on the difference degree value and the standard error of the M difference data, wherein the abnormality degree data is used to indicate whether the first difference data has abnormal data; The abnormality degree data is determined as the first analysis result data.

7. The method according to claim 1, wherein The generating of reward feedback data for the operation behavior of the i-th user in the M time period based on the second analysis result data of the operation behavior of the i-th user in the j-th time period and the user feedback result of the i-th user on the operation behavior in the j-th time period includes: Obtaining sub-reward feedback data for the i-th user's operation behavior in the j-th time period based on the second analysis result data of the i-th user's operation behavior in the j-th time period and the user feedback result of the i-th user on the operation behavior in the j-th time period; The target sub-reward feedback data that meets the preset conditions in the sub-reward feedback data of the operation behavior of the i-th user in the j-th time period is determined as the reward feedback data of the operation behavior of the i-th user in the M-th time period.

8. The method according to claim 7, wherein: The obtaining, based on the second analysis result data of the operation behavior of the i-th user in the j-th time period and the user feedback result of the i-th user on the operation behavior in the j-th time period, sub-reward feedback data of the operation behavior of the i-th user in the j-th time period includes: If the second analysis result data matches the user feedback result, determining the first preset reward feedback value as the sub-reward feedback data for the operation behavior of the i-th user in the j-th time period; If the second analysis result data does not match the user feedback result, determining a second preset reward feedback value as sub-reward feedback data for the operation behavior of the i-th user in the j-th time period; The second preset reward feedback value is smaller than the first preset reward feedback value.

9. The method according to claim 1, wherein: The first analysis result data is corrected by using the reward feedback data to obtain the third analysis result data, including: Adjusting the weight coefficient of the abnormal operation behavior recognition model using the reward feedback data to obtain a correction coefficient; generating, according to the weight coefficient and the first analysis result data, converted standard analysis result data of the first analysis result data; Based on the correction coefficient and the converted standard analysis result data, third analysis result data is generated.

10. The method according to claim 1, wherein The method further comprises: If the analysis result value of the third analysis result data is greater than or equal to the preset analysis result value, it is determined that the operation behaviors of the i users within the target time period are abnormal; When the analysis result value of the third analysis result data is less than the preset analysis result value, it is determined that there is no abnormal operation behavior in the operation behaviors of the i users within the target time period.

11. A data processing device comprising: The acquisition module is used to obtain log records generated by N users' operation behaviors; a determination module, configured to determine, based on the log records, a difference data set of an i-th user among the N users within M time periods, the difference data set including M difference data, the j-th difference data among the M difference data being used to characterize the difference between the operation behavior of the i-th user within the j-th time period and the operation behavior of the i-th user within the M time periods, the difference data set being used to determine first analysis result data, the first analysis result data being an initial analysis result for determining whether the operation behavior of the i-th user within the target time period contains abnormal operation behavior, j∈[1,M]; a generating module, configured to generate reward feedback data for the operation behavior of the i-th user in the M time period based on the second analysis result data of the operation behavior of the i-th user in the j-th time period and the user feedback result of the i-th user on the operation behavior in the j-th time period; A correction module is used to correct the first analysis result data using the reward feedback data to obtain third analysis result data, and the third analysis result data is used to determine whether the operation behaviors of the i users within the target time period are abnormal.

12. A computer device, comprising: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the data processing method according to any one of claims 1 to 10 is implemented.

13. A storage medium storing computer program instructions, wherein the computer program instructions, when executed by a processor, implement the data processing method according to any one of claims 1 to 10.

Images