Log processing method and device and electronic equipment

Abstract

The invention discloses a log processing method and device and electronic equipment, the method is applied to the field of log detection, and the method comprises the following steps: obtaining a plurality of first logs; for each first log in the plurality of first logs, processing words in the first log according to the first word set and the second word set to obtain a second log corresponding to the first log; and determining the threat probability of each second log through the target detection model, and performing threat detection on the second log of which the threat probability is greater than a preset probability to obtain a threat detection result. Through the log detection method and device, the problem that the detection efficiency of a deep learning model is low and the log detection efficiency is low due to the fact that massive logs are input into the deep learning model to detect the threat information in the logs in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of log detection, and in particular to a log processing method, device and electronic device. Background Art

[0002] Existing threat detection methods often need to process massive log data generated by terminals. Machine learning methods are usually used for log analysis, attempting to automatically identify threat features by training models. This solution collects a large amount of log data, including known threat logs and normal logs, as a training set to train the classification model. By learning from historical data, it attempts to gradually improve the accuracy and efficiency of threat detection. Using machine learning algorithms such as decision trees, support vector machines (SVMs), and random forests, the model learns to distinguish the characteristics of threat logs from normal logs. However, due to the diversity and continuous evolution of network threats, it may be difficult for the model to cover all types of threats, especially the insufficient detection capability for zero-day attacks, and the limited generalization ability of the model. In the face of emerging threat forms, the model needs to be retrained and cannot immediately reflect the latest threat features. At the same time, this method requires a lot of manual data annotation and preprocessing of massive log data, which is time-consuming, labor-intensive and costly.

[0003] Currently, no effective solution has been proposed to the problem that in related technologies, massive logs are input into deep learning models to detect threat information in the logs, which results in low detection efficiency of the deep learning models and low log detection efficiency. Summary of the invention

[0004] The main purpose of the present application is to provide a log processing method, device and electronic device to solve the problem in the related art that massive logs are input into a deep learning model to detect threat information in the logs, which makes the detection efficiency of the deep learning model low, resulting in low log detection efficiency.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a log processing method is provided, the method comprising: obtaining multiple first logs; for each first log in the multiple first logs, processing the words in the first log according to a first word set and a second word set to obtain a second log corresponding to the first log, wherein the word frequency of each word in the first word set is higher than the word frequency of each word in the second word set, and each word in the second word set is set with a risk level higher than a preset level; determining the threat probability of each second log through a target detection model, and performing threat detection on the second log whose threat probability is greater than the preset probability to obtain a threat detection result.

[0006] Furthermore, the words in the first log are processed according to the first word set and the second word set to obtain a second log corresponding to the first log, including: determining the words in the first word set as first words, and retaining the first words in the first log; determining the words in the second word set as second words, and replacing the second words in the first log according to the risk level of each second word in the second word set; determining the words in the first log other than the first word and the second word as third words, and replacing the third words in the first log with a preset character string to obtain a second log corresponding to the first log.

[0007] Furthermore, before processing the words in the first log according to the first word set and the second word set to obtain the second log corresponding to the first log, the method also includes: obtaining multiple historical logs, calculating the word frequency of each word in the multiple historical logs, and obtaining the word frequency of each word; generating the first word set based on the words whose word frequency is greater than a first preset threshold, and generating the second word set based on the words whose word frequency is less than or equal to the first preset threshold and greater than a second preset threshold; calculating the proportion of the words in the second word set in the preset threat log set, and determining the risk level of each word in the second word set based on the proportion value of each word in the second word set.

[0008] Furthermore, before calculating the word frequency of each word in the multiple historical logs to obtain the word frequency of each word, the method also includes: replacing a preset type of character string in the historical log by a regular expression to obtain a replaced historical log, wherein the preset type of character string is a character string that is unrelated to user behavior; and segmenting the replaced historical log according to preset characters to obtain each word in the historical log.

[0009] Further, each second log includes at least one key-value pair, the key in the key-value pair represents the attribute of the log data, the value in the key-value pair includes words in the second log, the values ​​include at least first-class values ​​and second-class values, the first-class values ​​represent words with semantic relationship between contexts, and the second-class values ​​represent words with no semantic relationship between contexts, and the threat probability of each second log is determined by the target detection model, including: for each key-value pair in each second log, using a first conversion model to convert the first-class value into an embedded vector, using a fully connected layer to convert the second-class value and the key into embedded vectors respectively, to obtain a key-value pair vector set of each second log; for each second log, inputting the key-value pair vector set of the second log into the multi-head attention mechanism of the target detection model to obtain the threat probability of each key-value pair in the second log; and determining the threat probability of the second log according to the threat probability of each key-value pair in the second log.

[0010] Furthermore, the target detection model is trained by the following steps: obtaining multiple sample logs, and constructing a key-value pair vector set of the multiple sample logs based on the first word set and the second word set; constructing a training sample set based on the key-value pair vector set of the multiple sample logs; iteratively training a preset detection model based on the training sample set to obtain at least one intermediate model; and determining the target detection model based on the at least one intermediate model.

[0011] Furthermore, after iteratively training a preset detection model based on the training sample set to obtain at least one intermediate model, the method also includes: obtaining a prediction result of each intermediate model in the at least one intermediate model, wherein the prediction result of the intermediate model represents the threat probability of each key-value pair in the multiple sample logs predicted by the intermediate model; determining the key-value pair with the highest threat probability in the prediction result of each intermediate model respectively to obtain a key-value pair set; and updating the second word set based on the key-value pair set.

[0012] Furthermore, after performing threat detection on the second log whose threat probability is greater than a preset probability and obtaining a threat detection result, the method further includes: determining a plurality of target words based on the threat detection result, wherein the plurality of target words are words that are threatening in the second log whose threat probability is greater than a preset probability; determining the number of threats for each of the plurality of target words, wherein the number of threats represents the number of times the target word is a word that is threatening; updating the risk level of each target word based on the number of threats for each target word to obtain an updated risk level for each target word; and updating the second word set based on the plurality of target words and the updated risk level of each target word to obtain an updated second word set.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a log processing device is provided, which includes: a first acquisition unit, used to acquire multiple first logs; a processing unit, used to process the words in the first log according to a first word set and a second word set for each first log in the multiple first logs, to obtain a second log corresponding to the first log, wherein the word frequency of each word in the first word set is higher than the word frequency of each word in the second word set, and each word in the second word set is set with a risk level higher than a preset level; a detection unit, used to determine the threat probability of each second log through a target detection model, and perform threat detection on the second log whose threat probability is greater than the preset probability to obtain a threat detection result.

[0014] Furthermore, the processing unit includes: a first determination subunit, used to determine the words in the first word set as first words, and retain the first words in the first log; a first replacement subunit, used to determine the words in the second word set as second words, and replace the second words in the first log according to the risk level of each second word in the second word set; a second replacement subunit, used to determine the words in the first log other than the first word and the second word as third words, and replace the third words in the first log with a preset character string to obtain a second log corresponding to the first log.

[0015] Furthermore, the device also includes: a first calculation unit, used to obtain multiple historical logs, calculate the word frequency of each word in the multiple historical logs, and obtain the word frequency of each word before processing the words in the first log according to the first word set and the second word set to obtain the second log corresponding to the first log; a generation unit, used to generate the first word set based on the words whose word frequency is greater than a first preset threshold, and generate the second word set based on the words whose word frequency is less than or equal to the first preset threshold and greater than the second preset threshold; a second calculation unit, used to calculate the proportion of the words in the second word set in the preset threat log set, and determine the risk level of each word in the second word set according to the proportion value of each word in the second word set.

[0016] Furthermore, the device also includes: a replacement unit, which is used to replace a preset type of character string in the historical log by a regular expression before calculating the word frequency of each word in the multiple historical logs to obtain the word frequency of each word, so as to obtain a replaced historical log, wherein the preset type of character string is a character string that is unrelated to user behavior; and a segmentation unit, which is used to segment the replaced historical log according to preset characters to obtain each word in the historical log.

[0017] Further, each of the second logs includes at least one key-value pair, the key in the key-value pair represents the attribute of the log data, the value in the key-value pair includes words in the second log, the values ​​include at least first-class values ​​and second-class values, the first-class values ​​represent words with semantic relationship between contexts, and the second-class values ​​represent words with no semantic relationship between contexts, and the detection unit includes: a conversion subunit, for each key-value pair in each second log, using a first conversion model to convert the first-class value into an embedded vector, using a fully connected layer to convert the second-class value and the key into embedded vectors respectively, to obtain a key-value pair vector set of each second log; a calculation subunit, for each second log, inputting the key-value pair vector set of the second log into the multi-head attention mechanism of the target detection model, to obtain the threat probability of each key-value pair in the second log; a second determination subunit, for determining the threat probability of the second log according to the threat probability of each key-value pair in the second log.

[0018] Furthermore, the device also includes: a second acquisition unit, used to acquire multiple sample logs, and construct a key-value pair vector set of the multiple sample logs based on the first word set and the second word set; a construction unit, used to construct a training sample set based on the key-value pair vector set of the multiple sample logs; a training unit, used to iteratively train a preset detection model based on the training sample set to obtain at least one intermediate model; and a first determination unit, used to determine the target detection model based on the at least one intermediate model.

[0019] Furthermore, the device also includes: a third acquisition unit, used to iteratively train a preset detection model based on the training sample set to obtain at least one intermediate model, and then obtain the prediction result of each intermediate model in the at least one intermediate model, wherein the prediction result of the intermediate model represents the threat probability of each key-value pair in the multiple sample logs predicted by the intermediate model; a second determination unit, used to determine the key-value pair with the highest threat probability in the prediction result of each intermediate model respectively to obtain a key-value pair set; and a first updating unit, used to update the second word set based on the key-value pair set.

[0020] Furthermore, the device also includes: a third determination unit, which is used to perform threat detection on the second log whose threat probability is greater than a preset probability, and after obtaining the threat detection result, determine multiple target words according to the threat detection result, wherein the multiple target words are words that exist threats in the second log whose threat probability is greater than the preset probability; a fourth determination unit, which is used to determine the number of threats of each target word in the multiple target words, wherein the number of threats represents the number of times the target word is used as a word that exists threats; a second updating unit, which is used to update the risk level of each target word according to the number of threats of each target word, and obtain the updated risk level of each target word; and a third updating unit, which is used to update the second word set according to the multiple target words and the updated risk level of each target word, and obtain an updated second word set.

[0021] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a computer program product is provided, including a computer program, which, when executed by a processor, implements any one of the above-mentioned log processing methods, and which, when executed by a processor, implements the steps of the log processing method described in each embodiment of the present application.

[0022] In order to achieve the above objective, according to one aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes stored computer instructions, wherein when the computer instructions are executed by a processor, any one of the above-mentioned log processing methods is implemented.

[0023] In order to achieve the above-mentioned purpose, according to one aspect of the present application, an electronic device is provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement any one of the log processing methods described above.

[0024] Through the present application, the following steps are adopted: obtaining multiple first logs; for each first log in the multiple first logs, processing the words in the first log according to the first word set and the second word set to obtain a second log corresponding to the first log, wherein the word frequency of each word in the first word set is higher than the word frequency of each word in the second word set, and each word in the second word set is set with a risk level higher than a preset level; determining the threat probability of each second log through a target detection model, and performing threat detection on the second log whose threat probability is greater than the preset probability to obtain a threat detection result, thereby solving the problem in the related art that the threat information in the log is detected by inputting massive logs into a deep learning model, resulting in low detection efficiency of the deep learning model, leading to low log detection efficiency. By processing the pre-prepared first word set and the second word set for multiple first logs to be detected, the model can better understand the contextual information in the log data. At the same time, the massive log data can be filtered and screened by the target detection model, which significantly reduces the data input amount for the subsequent threat detection model, thereby reducing the additional computing resource consumption caused by processing a large number of unnecessary logs, reducing the company's spending on data processing, storage and computing resources, and achieving the effect of improving the efficiency of log threat detection, so that the threat detection system can respond faster, improving the response speed of the entire security monitoring system, and further achieving the effect of enhancing the company's security defense system, resisting unknown threats and reducing potential risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0026] Figure 1 is a flow chart of a log processing method provided according to Embodiment 1 of the present application;

[0027] Figure 2 This is a schematic diagram of an optional log processing method provided according to the first embodiment of the present application. Figure 1 ;

[0028] Figure 3 This is a schematic diagram of an optional log processing method provided according to the first embodiment of the present application. Figure 2 ;

[0029] Figure 4 is a schematic diagram of a log processing device provided according to Embodiment 2 of the present application;

[0030] Figure 5 It is a schematic diagram of a log processing electronic device provided according to Embodiment 5 of the present application. DETAILED DESCRIPTION

[0031] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, collected data, used data, generated data, processed data, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, collected information, used information, generated information, processed information, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of relevant countries and regions, necessary confidentiality measures are taken, and public order and good morals are not violated, and corresponding operation portals are provided for users to choose to authorize or refuse. For example, an interface is set between this system and relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0033] It should be noted that this application provides users with corresponding operation entrances for them to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] Embodiment 1

[0037] The present invention is described below in conjunction with preferred implementation steps. Figure 1 is a flow chart of a log processing method provided in accordance with the first embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0038] Step S101, obtaining multiple first logs.

[0039] The log processing method provided in the first embodiment of the present application is applied to a target system for threat detection of logs generated by various systems or terminals. In the first embodiment of the present application, the target system collects multiple logs generated by a system (or terminal) to be detected, namely the first logs mentioned above, so as to perform threat detection based on the obtained multiple first logs, identify and prevent potential network security threats. By analyzing the logs in real time or periodically, potential security threats can be discovered in advance, so that preventive measures can be taken to avoid or reduce damage.

[0040] Step S102: for each first log in the plurality of first logs, the words in the first log are processed according to the first word set and the second word set to obtain a second log corresponding to the first log, wherein the word frequency of each word in the first word set is higher than the word frequency of each word in the second word set, and each word in the second word set is set with a risk level higher than a preset level.

[0041] In the first embodiment, in order to detect potential threats in the log, a pre-constructed high-frequency word set (i.e., the first word set mentioned above) and a word set of different risk levels (i.e., the second word set mentioned above) can be used to process each received first log to obtain a second log corresponding to each processed first log. Exemplarily, a preset character string can be used to replace meaningless words appearing in the first log, thereby preventing words without semantic information from affecting the threat probability of the log.

[0042] High-frequency words refer to words that appear frequently in logs. Words usually have certain semantic and grammatical meanings, such as single characters, phrases, or short phrases. Word frequency refers to the number of times a word appears in a text. The risk level of a word refers to the level of security risk that a word has. The higher the risk level corresponding to a word, the higher the degree of threat. Risk levels can be arranged from low to high, for example: level 1 risk level (e.g., low risk), level 2 risk level (e.g., general risk), level 3 risk level (e.g., greater risk), and level 4 risk level (e.g., major risk).

[0043] Step S103, determining the threat probability of each second log through the target detection model, and performing threat detection on the second logs whose threat probability is greater than the preset probability to obtain a threat detection result.

[0044] In the first embodiment, the threat probability of each second log is calculated by the target detection model, and the second log with a threat probability greater than the preset probability is determined as a second log with a threat, and other threat detection methods are used for the second log with a threat. By predicting the threat probability of each second log by the target detection model, the logs that are most likely to have security risks can be screened out from the massive logs to be detected, so that other threat detection methods are used to further perform threat detection on the screened logs with higher threat probabilities to obtain the final threat detection results, avoiding the problem of low detection efficiency and high detection cost caused by detecting massive logs in the prior art, improving the efficiency of log threat detection, and at the same time, being able to exclude log data that is not related to security, ensuring that the subsequent threat detection model can more centrally analyze and process those log data that are potentially risky, thereby helping to improve the accuracy of threat detection and reduce the false alarm rate of threat detection.

[0045] In summary, the log processing method provided in the first embodiment of the present application obtains multiple first logs; for each first log in the multiple first logs, processes the words in the first log according to the first word set and the second word set to obtain a second log corresponding to the first log, wherein the word frequency of each word in the first word set is higher than the word frequency of each word in the second word set, and each word in the second word set is set with a risk level higher than a preset level; determines the threat probability of each second log through a target detection model, and performs threat detection on the second log whose threat probability is greater than the preset probability to obtain a threat detection result, thereby solving the problem in the related art that the threat information in the log is detected by inputting massive logs into a deep learning model, which makes the detection efficiency of the deep learning model low, resulting in low log detection efficiency. By processing the pre-prepared first word set and the second word set for multiple first logs to be detected, the model can better understand the contextual information in the log data. At the same time, the massive log data can be filtered and screened by the target detection model, which significantly reduces the data input amount for the subsequent threat detection model, thereby reducing the additional computing resource consumption caused by processing a large number of unnecessary logs, reducing the company's spending on data processing, storage and computing resources, and achieving the effect of improving the efficiency of log threat detection, so that the threat detection system can respond faster, improving the response speed of the entire security monitoring system, and further achieving the effect of enhancing the company's security defense system, resisting unknown threats and reducing potential risks.

[0046] Optionally, in the log processing method provided in Example 1 of the present application, the words in the first log are processed according to the first word set and the second word set to obtain a second log corresponding to the first log, including: determining the words in the first word set as the first words, and retaining the first words in the first log; determining the words in the second word set as the second words, and replacing the second words in the first log according to the risk level of each second word in the second word set; determining the words in the first log other than the first word and the second word as the third word, and replacing the third word in the first log with a preset character string to obtain a second log corresponding to the first log.

[0047] In the first embodiment of the present invention, in order to effectively reduce noise and improve model processing efficiency, it is necessary to retain each high-frequency word in the first word set in each first log to be detected, that is, to determine the words in the first word set as the first words and retain the first words in the first log.

[0048] Then, in order to more accurately identify and reflect the words and phrases related to security threats in the log, it is necessary to replace each second word that appears in each first log to be detected with the risk level of the second word, that is, the words in the second word set are determined as second words, and the second words in the first log are replaced according to the risk level of each second word in the second word set.

[0049] Finally, for the remaining words that cannot be classified into the first word set or the second word set, they are uniformly replaced with "[Unknown]" (i.e., the preset string mentioned above), which ensures that the model can handle words whose risk levels cannot be directly identified, and also provides the model with the possibility of learning and adapting to new threats.

[0050] In an optional embodiment, if a first log is represented as “2023-0315 12:45:23ERRORGUID: ABCD1234EF56789 file attempts to access the registry key HKEY_local_MACHINE\SOFTWARE\Microsoft\Windows”, the corresponding second log after processing the first log is represented as “[time token][Risklevel1][guid token][Risklevel2][Unknown][Risklevel2][Unknown][Unknown]LSTM

[0051] (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows)".

[0052] Optionally, in the log processing method provided in Example 1 of the present application, before processing the words in the first log according to the first word set and the second word set to obtain the second log corresponding to the first log, the above method also includes: obtaining multiple historical logs, calculating the word frequency of each word in the multiple historical logs, and obtaining the word frequency of each word; generating a first word set based on words whose word frequency is greater than a first preset threshold, and generating a second word set based on words whose word frequency is less than or equal to the first preset threshold and greater than the second preset threshold; calculating the proportion of words in the second word set in the preset threat log set, and determining the risk level of each word in the second word set based on the proportion value of each word in the second word set.

[0053] In an optional embodiment, before processing the words in the first log, it is necessary to calculate the word frequency of each word in the historical log, that is, to calculate the number of times each independent word or phrase appears in multiple historical days (that is, the above-mentioned word frequency) to identify and filter out high-frequency words in the log. These high-frequency words are often of high value for understanding the log content and context, so that the model can more accurately understand the log content and the context in the log. Exemplarily, based on the statistical results of the word frequency, words or phrases with a frequency of occurrence in the top 40% can be selected to construct a first word set (that is, the above-mentioned first word set is generated based on words with a word frequency greater than the first preset threshold) for subsequent vectorization processing. By retaining high-frequency words in the log when processing the log to be detected, noise can be effectively reduced and model processing efficiency can be improved.

[0054] Then, for the words that are not included in the first word set, the words outside the first word set can be classified based on the word frequency of each word and divided into different risk levels, such as the above-mentioned [Risklevel1], [Risklevel2] and [Risklevel3]. Exemplarily, for the remaining 60% of the words outside the above-mentioned first word set, 50% of the high-frequency words are further screened out, and based on the proportion of these words appearing in the marked threat samples (i.e., the above-mentioned preset threat log set) (i.e., the above-mentioned proportion value), they are dynamically classified into different risk levels [Risklevel1], [Risklevel2] and [Risklevel3]. If the proportion of a certain word appearing in the threat log is greater than or equal to 0.6, it is classified as [Risklevel1]; if the proportion of a certain word appearing in the threat log is between 0.2 and 0.6, it is classified as [Risklevel2]; if the proportion of a certain word appearing in the threat log is less than 0.2, it is classified as [Risklevel3].

[0055] It should be noted that the allocation of risk levels [Risklevel1], [Risklevel2], and [Risklevel3] here is assumed to be based on the frequency of occurrence and threat level of each word in the labeled samples, but the actual risk level will be assigned based on the dynamic update of model learning during training and the occurrence of these words in threat logs.

[0056] By calculating the frequency of each word in the historical logs and constructing the first word set and the second word set according to the frequency of each word, the logs to be detected can be pre-filtered based on the vectorized token vocabulary (i.e., the first word set and the second word set mentioned above), making the maintenance and expansion of the model simpler. Therefore, when faced with different types of log formats or the need to adjust the filtering logic, it can be achieved by adjusting the first word set and / or the second word set without significantly modifying the entire model, thereby achieving the effect of improving the efficiency of model maintenance.

[0057] Optionally, in the log processing method provided in Example 1 of the present application, before calculating the word frequency of each word in multiple historical logs to obtain the word frequency of each word, the above method also includes: replacing a preset type of string in the historical log by a regular expression to obtain a replaced historical log, wherein the preset type of string is a string that is unrelated to user behavior; and segmenting the replaced historical log according to preset characters to obtain each word in the historical log.

[0058] In the first embodiment, in order to process log data, it is necessary to perform preprocessing operations on the collected historical logs.

[0059] In an optional embodiment, the strings in the historical log that are not related to user behavior (i.e., the strings of the preset type mentioned above) can be replaced by regular expressions to obtain the replaced historical logs. The strings of the preset type include, but are not limited to: the timestamp of each log, the globally unique identifier of each log (Globally UniqueIdentifier, each log can be assigned a unique GUID to ensure that when analyzing, archiving and processing a large amount of log data, it is easy to distinguish and track specific events or messages, hereinafter referred to as GUID), the timestamp of each log, the process ID of each log and the hash value of each log. For example, in a certain log, the timestamp "2023-03-1512:45:23" is replaced with "[time token]", the GUID "ABCD1234EF56789" is replaced with "[guid token]", the process ID is replaced with "[process id token]", and the hash value is replaced with "[hash token]".

[0060] Then, the text content of each log in the replaced historical log is simply segmented according to the space (i.e., the preset character mentioned above) to obtain the basic word or phrase unit, i.e., each word in the above historical log. For example, "[time token]ERROR[guid token]" will be segmented into "[time token]", "ERROR", and "[guidtoken]".

[0061] By segmenting the logs, the model can distinguish and identify the different components of the log entries (for example, operation instructions, system paths, error codes, etc.), which is conducive to understanding and parsing the log content, thereby achieving the effect of improving the accuracy of log threat detection.

[0062] Optionally, in the log processing method provided in Example 1 of the present application, each second log includes at least one key-value pair, the key in the key-value pair represents the attribute of the log data, the value in the key-value pair includes words in the second log, the value includes at least a first-class value and a second-class value, the first-class value represents words with a semantic relationship between contexts, and the second-class value represents words with no semantic relationship between contexts, and the threat probability of each second log is determined by a target detection model, including: for each key-value pair in each second log, using a first conversion model to convert the first-class value into an embedded vector, using a fully connected layer to convert the second-class value and the key into embedded vectors respectively, to obtain a key-value pair vector set of each second log; for each second log, inputting the key-value pair vector set of the second log into the multi-head attention mechanism of the target detection model to obtain the threat probability of each key-value pair in the second log; and determining the threat probability of the second log based on the threat probability of each key-value pair in the second log.

[0063] In the first embodiment, in order to convert the string data in the second log into a data format that can be processed by the model, it is necessary to convert the words in each second log into a key-value pair and convert each key-value pair into an embedded vector before the model processes each second log.

[0064] In an optional embodiment, at least one word in the second log can be used as a value, and the log attribute corresponding to the at least one word can be used as a key to form a key-value pair for each second log, that is, each of the second logs mentioned above includes at least one key-value pair. The value of each key-value pair in the second log can also be subdivided into sequence information and non-sequence information. Sequence information refers to the existence of a semantic relationship between multiple words contained in the value, or the existence of a semantic relationship between the contexts of a single word contained in the value. Exemplarily, a key-value pair of the second log can be expressed as "["file path": "C:\Users\A\A\A"]".

[0065] Then, each key-value pair in the second log is converted into an embedded vector, wherein the first type of value in each key-value pair can be processed by a long short-term memory model (Long Short-Term Memory, i.e., the first conversion model mentioned above, hereinafter referred to as LSTM model), the second type of value in each key-value pair can be processed by a fully connected layer, and the key in each key-value pair is also processed by a fully connected layer, so as to obtain the key-value pair vector set of each second log mentioned above.

[0066] Finally, the key-value pair vector set of each second log is input into the trained target detection model, and the multi-head attention mechanism of the target detection model focuses on all key-value pairs to obtain the threat probability of each key-value pair in the second log output by the target detection model. The highest threat probability corresponding to the key-value pair contained in each second log is used as the threat probability of each second log.

[0067] Optionally, in the log processing method provided in Example 1 of the present application, the above-mentioned target detection model is trained by the following steps: obtaining multiple sample logs, and constructing a key-value pair vector set of the multiple sample logs based on the first word set and the second word set; constructing a training sample set based on the key-value pair vector set of the multiple sample logs; iteratively training a preset detection model based on the training sample set to obtain at least one intermediate model; and determining the target detection model based on the at least one intermediate model.

[0068] In an optional embodiment, the historical logs of the system can be obtained to obtain the above-mentioned multiple sample logs. The first word set and the second word set that have been constructed are used to perform retention operations or replacement operations on the multiple sample logs, and the key-value pairs in each sample are constructed and converted into embedded vectors to obtain the key-value pair vector set of the above-mentioned multiple sample logs. Then, based on the multiple sample logs and the key-value pairs in each sample log, the above-mentioned training sample set is constructed. The training sample set is used to iteratively train the converter model to obtain an intermediate model obtained in the iterative training process, and the intermediate model with the highest accuracy is determined as the above-mentioned target detection model.

[0069] Optionally, in the log processing method provided in Example 1 of the present application, after iteratively training a preset detection model based on a training sample set to obtain at least one intermediate model, the method further includes: obtaining a prediction result of each intermediate model in at least one intermediate model, wherein the prediction result of the intermediate model represents the threat probability of each key-value pair in multiple sample logs predicted by the intermediate model; determining the key-value pair with the highest threat probability in the prediction result of each intermediate model respectively to obtain a key-value pair set; and updating the second word set based on the key-value pair set.

[0070] In the first embodiment, in order to improve the adaptability of the model, it is necessary to update the risk level of the words according to the result of each model training, so as to improve the accuracy of the detection result of the model.

[0071] In an optional embodiment, after obtaining the intermediate model obtained in the iterative training process of the target detection model, the threat probability of each key-value pair in the log can be calculated according to each intermediate model, and the key-value pair with the highest threat probability is determined in the prediction results of each intermediate model to obtain a key-value pair set. Then, a second word set is obtained based on the words contained in the value of each key-value pair in the key-value pair set, and the risk level of each word in the second word set is updated.

[0072] It should be noted that some judgment methods can be used to judge whether the threat probability calculated by each key-value pair in the key-value pair set is accurate, and the words contained in the key-value pair whose threat probability is accurately calculated are added to the second word set. Exemplarily, the prediction result of each intermediate model can be sent to the target object, and the target object can be used to judge whether the threat probability calculated by the intermediate model is accurate. If the calculation result of the model is accurate, the second word set is updated according to the words with the highest threat probability in the calculation result of the model.

[0073] By using the prediction results of the model to adjust the second word set, the words in the second word set can be updated in real time according to the log data, realizing dynamic learning and adaptation of the model, thereby achieving the effect of improving the accuracy of the detection results of the target detection model.

[0074] Optionally, in the log processing method provided in Example 1 of the present application, after performing threat detection on the second log whose threat probability is greater than a preset probability and obtaining the threat detection result, the method further includes: determining multiple target words based on the threat detection result, wherein the multiple target words are words that exist as threats in the second log whose threat probability is greater than the preset probability; determining the number of threats for each of the multiple target words, wherein the number of threats represents the number of times the target word is used as a word that exists as a threat; updating the risk level of each target word based on the threat number of each target word to obtain an updated risk level for each target word; and updating the second word set based on the multiple target words and the updated risk level of each target word to obtain an updated second word set.

[0075] Optionally, after obtaining the updated second word set, multiple logs to be detected may be processed according to the first word set and the updated second word set.

[0076] In the first embodiment of the present invention, the accuracy of the log threat detection results can be improved by updating the second word set based on the threat words detected in each detection result, so that the log to be detected can be processed according to the continuously optimized second word set, thereby enhancing the model's detection ability for new threats. This dynamic learning ability makes threat detection more flexible and adaptable, thereby achieving the effect of enhancing the accuracy of the model detection results.

[0077] In an optional embodiment, the key-value pair with the highest threat probability in each log can be determined from the threat detection results, and the above-mentioned multiple target words can be determined based on the value of the key-value pair with the highest threat probability. Then, the number of threats for each target word among the multiple target words is determined, so as to redetermine the risk level of each target word based on the number of threats of each target word, and obtain the updated risk level of each target word. Finally, the second word set is updated based on the multiple target words and the updated risk level of each target word to obtain an updated second word set. When multiple logs to be detected are subsequently received, the words in the logs to be detected are retained or replaced based on the first word set and the updated second word set.

[0078] For example, if a log is "Zhang San is a student", "is" and "student" are the first words in the first word set, and "Zhang San" is a word of the "[Unknown]" class. After threat detection is performed on the log, it is determined that the log is a log with threats, and the word "Zhang San" appears in the threat detection result, then the word "Zhang San" is added to the second word set, and the risk level of the word "Zhang San" is set to "[Risklevel1]". If other logs containing the word "Zhang San" are subjected to threat detection, and the word "Zhang San" appears again in the threat detection result, then the risk level of the word "Zhang San" in the second word set is adjusted to "[Risklevel2]", and so on.

[0079] Optionally, in the first embodiment, the process of training the target detection model in this solution can be as follows: Figure 2As shown, multiple sample logs are preprocessed, and the first word set and the second word set are used to retain or replace the words in the multiple sample logs; a key-value pair set of each sample log is constructed, and the LSTM model is used to process the value containing contextual semantic information in each key-value pair (i.e., the second type of value mentioned above), and the fully connected layer is used to process the value that does not contain contextual semantic information in each key-value pair (i.e., the first type of value mentioned above) and the key in each key-value pair to obtain a key-value pair vector set of each sample log; the key-value pair vector set of each sample log is input into the multi-head attention mechanism of the preset detection model for calculation, the threat probability of each key-value pair is output through the activation function, the preset detection model is iteratively trained, and the model with the highest accuracy is selected as the target detection model; the word with the highest threat probability calculated during the iterative training of the model is updated to the second word set.

[0080] Optionally, in the first embodiment, the process of using the target detection model to perform threat detection on multiple first logs to be detected can be as follows: Figure 3 As shown, a first word set and a second word set are loaded, and multiple first logs to be detected are received; the first words appearing in each first log are retained according to the first word set, and the second words appearing in the second word set are used to replace the second words in each first log, so as to obtain multiple second logs, wherein the first words are words in the first word set, and the second words are words in the second word set; for each second log, each key-value pair in the second log is constructed, a fully connected layer is used to convert the key of each key-value pair into an embedded vector, a fully connected layer is used to convert the value of each key-value pair that does not contain semantic information into an embedded vector, an LSTM model is used to convert the value of each key-value pair that contains semantic information into an embedded vector, and the embedded vector of the key of each key-value pair and the embedded vector of the value of each key-value pair are concatenated to obtain the embedded vector of each key-value pair. , forming a key-value pair vector set of each second log; inputting the key-value pair vector set of each second log into the trained target detection model, the multi-head attention mechanism in the target detection model pays attention to all key-value pairs in each second log, obtaining the threat probability of each key-value pair in each second log, and determining the highest threat probability of the key-value pairs in each second log as the threat probability of each second log, and when the threat probability of the second log is greater than a preset value, determining the second log as a second log with a threat, and obtaining the above-mentioned threat detection result; judging whether each second log in the threat detection result has a threat or a security risk by a manual detection method, and updating the second word set according to the words contained in the key-value pair with the highest threat probability of the second log with a threat or a security risk in the judgment result; and using the updated second word set to continue detecting the next log to be detected.

[0081] Optionally, the log processing method provided in Example 1 of the present application can also be applied in the field of network security, especially for enterprise-level users. Faced with the massive log data generated every day, it is necessary to accurately and efficiently identify potential security threats, thereby significantly reducing unnecessary log analysis workload and improving the efficiency and accuracy of threat detection.

[0082] Optionally, the log processing method provided in Example 1 of the present application can also be applied to a cloud platform that provides SaaS, PaaS or IaaS services, providing cloud platform customers with more intelligent and efficient log analysis services, helping customers to screen out important security logs for timely response while reducing the consumption of computing resources.

[0083] Optionally, the log processing method provided in Example 1 of the present application can also be applied to terminal protection software (for example, antivirus software, intrusion detection system, etc.) to perform preliminary screening of logs generated by terminal devices (personal computers, mobile phones, etc.), reduce false alarms and missed alarms, ensure the effective use of system resources, and enhance user experience.

[0084] Optionally, the log processing method provided in Example 1 of the present application can also be applied to organizations that need to comply with specific industry standards and regulatory requirements (such as finance, medical care, authoritative institutions, etc.). By pre-filtering a large number of irrelevant logs, it can focus more on the key points of compliance audits and improve the efficiency and quality of audits.

[0085] Optionally, the log processing method provided in Example 1 of the present application can also be applied to the widespread deployment of smart home devices and Internet of Things devices. By performing threat detection on the large amount of log data generated by these devices, log information that may contain security threats can be effectively screened out, and risk warnings and processing can be performed in advance.

[0086] Optionally, the log processing method provided in Example 1 of the present application can also be applied in data centers and large servers. These devices will generate a large number of logs during operation. By filtering irrelevant logs, it can help managers efficiently screen and analyze log data, quickly identify system anomalies, configuration errors or security vulnerabilities, and improve system stability and security.

[0087] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0088] Embodiment 2

[0089] The second embodiment of the present application also provides a log processing device. It should be noted that the log processing device of the second embodiment of the present application can be used to execute the log processing method provided in the first embodiment of the present application. The log processing device provided in the second embodiment of the present application is introduced below.

[0090] Figure 4 Schematic diagram of a log processing device according to Embodiment 2 of the present application. Figure 4 As shown, the device includes: a first acquisition unit 401, a processing unit 402 and a detection unit 403.

[0091] Specifically, the first obtaining unit 401 is used to obtain multiple first logs.

[0092] The processing unit 402 is used to process the words in each first log of the multiple first logs according to the first word set and the second word set to obtain a second log corresponding to the first log, wherein the word frequency of each word in the first word set is higher than the word frequency of each word in the second word set, and each word in the second word set is set with a risk level higher than a preset level.

[0093] The detection unit 403 is used to determine the threat probability of each second log through the target detection model, and perform threat detection on the second logs whose threat probability is greater than a preset probability to obtain a threat detection result.

[0094] The log processing device provided in the second embodiment of the present application obtains multiple first logs through the first acquisition unit 401; the processing unit 402 processes the words in the first log according to the first word set and the second word set for each first log in the multiple first logs to obtain a second log corresponding to the first log, wherein the word frequency of each word in the first word set is higher than the word frequency of each word in the second word set, and each word in the second word set is set with a risk level higher than a preset level; the detection unit 403 determines the threat probability of each second log through the target detection model, and performs threat detection on the second log whose threat probability is greater than the preset probability to obtain a threat detection result, which solves the problem in the related art that the threat information in the log is detected by inputting massive logs into the deep learning model, resulting in low detection efficiency of the deep learning model, resulting in low log detection efficiency. By processing the pre-prepared first word set and the second word set for multiple first logs to be detected, the model can better understand the contextual information in the log data. At the same time, the massive log data can be filtered and screened by the target detection model, which significantly reduces the data input amount for the subsequent threat detection model, thereby reducing the additional computing resource consumption caused by processing a large number of unnecessary logs, reducing the company's spending on data processing, storage and computing resources, and achieving the effect of improving the efficiency of log threat detection, so that the threat detection system can respond faster, improving the response speed of the entire security monitoring system, and further achieving the effect of enhancing the company's security defense system, resisting unknown threats and reducing potential risks.

[0095] Optionally, in the log processing device provided in Example 2 of the present application, the above-mentioned processing unit 402 includes: a first determination subunit, used to determine a word in the first word set as a first word, and retain the first word in the first log; a first replacement subunit, used to determine a word in the second word set as a second word, and replace the second word in the first log according to the risk level of each second word in the second word set; a second replacement subunit, used to determine a word in the first log other than the first word and the second word as a third word, and replace the third word in the first log with a preset character string to obtain a second log corresponding to the first log.

[0096] Optionally, in the log processing device provided in Example 2 of the present application, the above-mentioned device also includes: a first calculation unit, which is used to obtain multiple historical logs, calculate the word frequency of each word in the multiple historical logs, and obtain the word frequency of each word before processing the words in the first log according to the first word set and the second word set to obtain the second log corresponding to the first log; a generation unit, which is used to generate the first word set based on the words whose word frequency is greater than the first preset threshold, and generate the second word set based on the words whose word frequency is less than or equal to the first preset threshold and greater than the second preset threshold; a second calculation unit, which is used to calculate the proportion of the words in the second word set in the preset threat log set, and determine the risk level of each word in the second word set according to the proportion value of each word in the second word set.

[0097] Optionally, in the log processing device provided in Example 2 of the present application, the above-mentioned device also includes: a replacement unit, which is used to replace a preset type of string in the historical log by a regular expression before calculating the word frequency of each word in multiple historical logs to obtain the word frequency of each word, so as to obtain a replaced historical log, wherein the preset type of string is a string that is independent of user behavior; and a segmentation unit, which is used to segment the replaced historical log according to preset characters to obtain each word in the historical log.

[0098] Optionally, in the log processing device provided in Example 2 of the present application, each of the above-mentioned second logs includes at least one key-value pair, the key in the above-mentioned key-value pair represents the attribute of the log data, the value in the above-mentioned key-value pair includes words in the second log, the above-mentioned values ​​include at least first-class values ​​and second-class values, the above-mentioned first-class values ​​represent words with semantic relationships between contexts, and the above-mentioned second-class values ​​represent words without semantic relationships between contexts, and the above-mentioned detection unit 403 includes: a conversion subunit, which is used to convert the first-class value into an embedded vector using a first conversion model for each key-value pair in each second log, and convert the second-class value and the key into embedded vectors respectively using a fully connected layer to obtain a key-value pair vector set of each second log; a calculation subunit, which is used to input the key-value pair vector set of the second log into the multi-head attention mechanism of the target detection model for each second log, and obtain the threat probability of each key-value pair in the second log; a second determination subunit, which is used to determine the threat probability of the second log based on the threat probability of each key-value pair in the second log.

[0099] Optionally, in the log processing device provided in Example 2 of the present application, the above-mentioned device also includes: a second acquisition unit, used to acquire multiple sample logs, and construct a key-value pair vector set of the multiple sample logs based on the first word set and the second word set; a construction unit, used to construct a training sample set based on the key-value pair vector set of the multiple sample logs; a training unit, used to iteratively train a preset detection model based on the training sample set to obtain at least one intermediate model; and a first determination unit, used to determine the target detection model based on the at least one intermediate model.

[0100] Optionally, in the log processing device provided in Example 2 of the present application, the above-mentioned device also includes: a third acquisition unit, used to iteratively train a preset detection model based on a training sample set to obtain at least one intermediate model, and then obtain a prediction result of each intermediate model in at least one intermediate model, wherein the prediction result of the intermediate model represents the threat probability of each key-value pair in multiple sample logs predicted by the intermediate model; a second determination unit, used to determine the key-value pair with the highest threat probability in the prediction result of each intermediate model respectively, to obtain a key-value pair set; a first updating unit, used to update the second word set based on the key-value pair set.

[0101] Optionally, in the log processing device provided in Example 2 of the present application, the above-mentioned device also includes: a third determination unit, used to perform threat detection on the second log whose threat probability is greater than a preset probability, and after obtaining the threat detection result, determine multiple target words based on the threat detection result, wherein the multiple target words are words that exist threats in the second log whose threat probability is greater than the preset probability; a fourth determination unit, used to determine the number of threats of each target word in the multiple target words, wherein the number of threats represents the number of times the target word is used as a word that exists threats; a second updating unit, used to update the risk level of each target word based on the threat number of each target word, and obtain the updated risk level of each target word; a third updating unit, used to update the second word set based on the multiple target words and the updated risk level of each target word, and obtain an updated second word set.

[0102] The log processing device includes a processor and a memory. The first acquisition unit 401, the processing unit 402 and the detection unit 403 are all stored in the memory as program units. The processor executes the program units stored in the memory to implement corresponding functions.

[0103] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the log detection efficiency can be improved by adjusting the kernel parameters.

[0104] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0105] Embodiment 3 of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a log processing method is implemented.

[0106] A fourth embodiment of the present invention provides a processor, which is used to run a program, wherein the log processing method is executed when the program is running.

[0107] like Figure 5 As shown, embodiment 5 of the present invention provides an electronic device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the following steps when executing the program: obtaining multiple first logs; for each first log in the multiple first logs, processing the words in the first log according to the first word set and the second word set to obtain a second log corresponding to the first log, wherein the word frequency of each word in the first word set is higher than the word frequency of each word in the second word set, and each word in the second word set is set with a risk level higher than a preset level; determining the threat probability of each second log through a target detection model, and performing threat detection on the second log whose threat probability is greater than the preset probability to obtain a threat detection result.

[0108] When the processor executes the program, the following steps are also implemented: words in the first log are processed according to the first word set and the second word set to obtain a second log corresponding to the first log, including: determining the words in the first word set as the first words, and retaining the first words in the first log; determining the words in the second word set as the second words, and replacing the second words in the first log according to the risk level of each second word in the second word set; determining the words in the first log other than the first word and the second word as the third words, and replacing the third words in the first log with a preset character string to obtain the second log corresponding to the first log.

[0109] When the processor executes the program, the following steps are also implemented: before processing the words in the first log according to the first word set and the second word set to obtain the second log corresponding to the first log, the above method also includes: obtaining multiple historical logs, calculating the word frequency of each word in the multiple historical logs, and obtaining the word frequency of each word; generating a first word set based on words with a word frequency greater than a first preset threshold, and generating a second word set based on words with a word frequency less than or equal to the first preset threshold and greater than a second preset threshold; calculating the proportion of words in the second word set in the preset threat log set, and determining the risk level of each word in the second word set according to the proportion value of each word in the second word set.

[0110] When the processor executes the program, the following steps are also implemented: before calculating the word frequency of each word in multiple historical logs to obtain the word frequency of each word, the above method also includes: replacing the preset type of string in the historical log by a regular expression to obtain the replaced historical log, wherein the preset type of string is a string that is unrelated to user behavior; segmenting the replaced historical log according to preset characters to obtain each word in the historical log.

[0111] When the processor executes the program, the following steps are also implemented: each second log includes at least one key-value pair, the key in the key-value pair represents the attribute of the log data, the value in the key-value pair includes words in the second log, the value includes at least a first-class value and a second-class value, the first-class value represents words with a semantic relationship between contexts, and the second-class value represents words with no semantic relationship between contexts, and the threat probability of each second log is determined by the target detection model, including: for each key-value pair in each second log, using a first conversion model to convert the first-class value into an embedded vector, using a fully connected layer to convert the second-class value and the key into the embedded vector respectively, to obtain a key-value pair vector set of each second log; for each second log, inputting the key-value pair vector set of the second log into the multi-head attention mechanism of the target detection model to obtain the threat probability of each key-value pair in the second log; and determining the threat probability of the second log according to the threat probability of each key-value pair in the second log.

[0112] When the processor executes the program, the following steps are also implemented: the above-mentioned target detection model is trained by the following steps: obtaining multiple sample logs, and constructing a key-value pair vector set of the multiple sample logs based on the first word set and the second word set; constructing a training sample set based on the key-value pair vector set of the multiple sample logs; iteratively training a preset detection model based on the training sample set to obtain at least one intermediate model; and determining the target detection model based on at least one intermediate model.

[0113] When the processor executes the program, the following steps are also implemented: after iteratively training the preset detection model based on the training sample set to obtain at least one intermediate model, the above method also includes: obtaining the prediction result of each intermediate model in at least one intermediate model, wherein the prediction result of the intermediate model represents the threat probability of each key-value pair in multiple sample logs predicted by the intermediate model; determining the key-value pair with the highest threat probability in the prediction result of each intermediate model respectively to obtain a key-value pair set; and updating the second word set based on the key-value pair set.

[0114] When the processor executes the program, the following steps are also implemented: after performing threat detection on the second log whose threat probability is greater than the preset probability and obtaining the threat detection result, the above method also includes: determining multiple target words based on the threat detection result, wherein the multiple target words are words that exist threats in the second log whose threat probability is greater than the preset probability; determining the number of threats of each target word in the multiple target words, wherein the number of threats represents the number of times the target word is used as a word that exists threats; updating the risk level of each target word based on the number of threats of each target word, and obtaining the updated risk level of each target word.

[0115] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0116] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: obtaining multiple first logs; for each first log in the multiple first logs, processing the words in the first log according to the first word set and the second word set to obtain a second log corresponding to the first log, wherein the word frequency of each word in the first word set is higher than the word frequency of each word in the second word set, and each word in the second word set is set with a risk level higher than a preset level; determining the threat probability of each second log through a target detection model, and performing threat detection on the second log whose threat probability is greater than the preset probability to obtain a threat detection result.

[0117] When executed on a data processing device, it is also suitable for executing a program initialized with the following method steps: processing words in a first log according to a first word set and a second word set to obtain a second log corresponding to the first log, including: determining a word in the first word set as a first word, and retaining the first word in the first log; determining a word in the second word set as a second word, and replacing the second word in the first log according to the risk level of each second word in the second word set; determining a word in the first log other than the first word and the second word as a third word, and replacing the third word in the first log with a preset character string to obtain a second log corresponding to the first log.

[0118] When executed on a data processing device, it is also suitable for executing a program that is initialized with the following method steps: before processing the words in the first log according to the first word set and the second word set to obtain the second log corresponding to the first log, the above method also includes: obtaining multiple historical logs, calculating the word frequency of each word in the multiple historical logs, and obtaining the word frequency of each word; generating a first word set based on words with a word frequency greater than a first preset threshold, and generating a second word set based on words with a word frequency less than or equal to the first preset threshold and greater than a second preset threshold; calculating the proportion of words in the second word set in the preset threat log set, and determining the risk level of each word in the second word set based on the proportion value of each word in the second word set.

[0119] When executed on a data processing device, it is also suitable for executing a program that is initialized with the following method steps: before calculating the word frequency of each word in multiple historical logs to obtain the word frequency of each word, the above method also includes: replacing a preset type of character string in the historical log by a regular expression to obtain a replaced historical log, wherein the preset type of character string is a character string that is unrelated to user behavior; segmenting the replaced historical log according to preset characters to obtain each word in the historical log.

[0120] When executed on a data processing device, it is also suitable for executing a program initialized with the following method steps: each second log includes at least one key-value pair, the key in the key-value pair represents the attribute of the log data, the value in the key-value pair includes words in the second log, the value includes at least a first-class value and a second-class value, the first-class value represents words with a semantic relationship between contexts, and the second-class value represents words without a semantic relationship between contexts, and the threat probability of each second log is determined by a target detection model, including: for each key-value pair in each second log, using a first conversion model to convert the first-class value into an embedded vector, using a fully connected layer to convert the second-class value and the key into an embedded vector respectively, to obtain a key-value pair vector set of each second log; for each second log, inputting the key-value pair vector set of the second log into the multi-head attention mechanism of the target detection model to obtain the threat probability of each key-value pair in the second log; and determining the threat probability of the second log according to the threat probability of each key-value pair in the second log.

[0121] When executed on a data processing device, it is also suitable for executing a program initialized with the following method steps: the above-mentioned target detection model is trained by the following steps: obtaining multiple sample logs, and constructing a key-value pair vector set of the multiple sample logs based on the first word set and the second word set; constructing a training sample set based on the key-value pair vector set of the multiple sample logs; iteratively training a preset detection model based on the training sample set to obtain at least one intermediate model; and determining the target detection model based on at least one intermediate model.

[0122] When executed on a data processing device, it is also suitable for executing a program initialized with the following method steps: after iteratively training a preset detection model based on a training sample set to obtain at least one intermediate model, the above method also includes: obtaining a prediction result of each intermediate model in at least one intermediate model, wherein the prediction result of the intermediate model represents the threat probability of each key-value pair in multiple sample logs predicted by the intermediate model; determining the key-value pair with the highest threat probability in the prediction result of each intermediate model respectively to obtain a key-value pair set; and updating the second word set based on the key-value pair set.

[0123] When executed on a data processing device, it is also suitable for executing a program initialized with the following method steps: after performing threat detection on a second log whose threat probability is greater than a preset probability and obtaining a threat detection result, the above method also includes: determining multiple target words based on the threat detection result, wherein the multiple target words are words that exist as threats in the second log whose threat probability is greater than the preset probability; determining the number of threats for each target word in the multiple target words, wherein the number of threats represents the number of times the target word is used as a word that exists as a threat; updating the risk level of each target word based on the number of threats for each target word to obtain an updated risk level for each target word; updating the second word set based on the multiple target words and the updated risk level of each target word to obtain an updated second word set.

[0124] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0129] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0130] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0131] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0132] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0133] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A log processing method, characterized in that: include: Get multiple first logs; For each first log of the plurality of first logs, words in the first log are processed according to a first word set and a second word set to obtain a second log corresponding to the first log, wherein the word frequency of each word in the first word set is higher than the word frequency of each word in the second word set, and each word in the second word set is set with a risk level higher than a preset level; The threat probability of each second log is determined by the target detection model, and threat detection is performed on the second log whose threat probability is greater than the preset probability to obtain a threat detection result.

2. The method according to claim 1, characterized in that Processing the words in the first log according to the first word set and the second word set to obtain a second log corresponding to the first log includes: Determining a word in the first word set as a first word, and retaining the first word in the first log; Determine words in the second word set as second words, and replace the second words in the first log according to the risk level of each second word in the second word set; The words other than the first word and the second word in the first log are determined as third words, and the third words in the first log are replaced with a preset character string to obtain a second log corresponding to the first log.

3. The method according to claim 1, characterized in that Before processing the words in the first log according to the first word set and the second word set to obtain the second log corresponding to the first log, the method further includes: Obtain a plurality of historical logs, calculate the word frequency of each word in the plurality of historical logs, and obtain the word frequency of each word; Generate the first word set based on the words whose word frequency is greater than a first preset threshold, and generate the second word set based on the words whose word frequency is less than or equal to the first preset threshold and greater than a second preset threshold; The proportion of the words in the second word set in the preset threat log set is calculated, and the risk level of each word in the second word set is determined according to the proportion of each word in the second word set.

4. The method according to claim 3, characterized in that Before calculating the word frequency of each word in the plurality of historical logs to obtain the word frequency of each word, the method further includes: Replacing a character string of a preset type in the historical log by a regular expression to obtain a replaced historical log, wherein the character string of the preset type is a character string irrelevant to user behavior; The replaced historical log is segmented according to preset characters to obtain each word in the historical log.

5. The method according to claim 1, characterized in that Each second log includes at least one key-value pair, the key in the key-value pair represents an attribute of log data, the value in the key-value pair includes words in the second log, the value includes at least a first-class value and a second-class value, the first-class value represents words with a semantic relationship between contexts, and the second-class value represents words without a semantic relationship between contexts, and the threat probability of each second log is determined by the target detection model, including: For each key-value pair in each second log, convert the first-category value into an embedded vector using a first conversion model, and convert the second-category value and the key into embedded vectors respectively using a fully connected layer, so as to obtain a key-value pair vector set of each second log; For each second log, input the key-value pair vector set of the second log into the multi-head attention mechanism of the target detection model to obtain the threat probability of each key-value pair in the second log; The threat probability of the second log is determined according to the threat probability of each key-value pair in the second log.

6. The method according to any one of claims 1 to 5, characterized in that: The target detection model is trained by the following steps: Acquire a plurality of sample logs, and construct a key-value pair vector set of the plurality of sample logs according to the first word set and the second word set; Constructing a training sample set based on the key-value pair vector set of the multiple sample logs; Iteratively train a preset detection model based on the training sample set to obtain at least one intermediate model; The object detection model is determined based on the at least one intermediate model.

7. The method according to claim 6, characterized in that After iteratively training a preset detection model based on the training sample set to obtain at least one intermediate model, the method further includes: Obtaining a prediction result of each intermediate model in the at least one intermediate model, wherein the prediction result of the intermediate model represents a threat probability of each key-value pair in the plurality of sample logs predicted by the intermediate model; Determine the key-value pair with the highest threat probability in the prediction results of each intermediate model respectively, and obtain a key-value pair set; The second word set is updated according to the key-value pair set.

8. The method according to claim 1, characterized in that After performing threat detection on the second log whose threat probability is greater than the preset probability and obtaining the threat detection result, the method further includes: Determining a plurality of target words according to the threat detection result, wherein the plurality of target words are words that have threats in the second log and whose threat probabilities are greater than a preset probability; Determining the threat count of each target word among the plurality of target words, wherein the threat count represents the number of times the target word is used as a word that poses a threat; Update the risk level of each target word according to the number of threats of each target word to obtain the updated risk level of each target word; The second word set is updated according to the multiple target words and the updated risk level of each target word to obtain an updated second word set.

9. A log processing device, characterized in that: include: A first acquisition unit, used to acquire a plurality of first logs; a processing unit, configured to process, for each of the plurality of first logs, words in the first log according to a first word set and a second word set to obtain a second log corresponding to the first log, wherein a word frequency of words in the first word set is higher than a word frequency of words in the second word set, and each word in the second word set is set with a corresponding risk level; The detection unit is used to determine the threat probability of each second log through a target detection model, and perform threat detection on the second log whose threat probability is greater than a preset probability to obtain a threat detection result.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the log processing method described in any one of claims 1 to 8.

Images