Continuous automatic vulnerability mining method based on log data

Abstract

The invention discloses a continuous automatic vulnerability mining method based on log data, and belongs to the technical field of network and information security. According to the method, through quantitative information flow analysis and topology discovery based on data tracing, an application surface layer can be penetrated, and a dynamic generation interface, an application program interface and a deep vulnerability parameter which cannot be touched by a traditional technology can be accurately identified, so that the attack entry point coverage rate in real flow is remarkably improved, and panoramic coverage of an attack surface is realized; on the basis of a static semantic-dynamic time sequence double-layer learning model and a closed-loop feedback mechanism, novel attack modes including zero-day attack can be captured, understood and verified from real traffic nearly in real time, the threat response time is shortened from several days to several hours, and the adaptive threat immunity is constructed; through a risk intelligent scheduling mechanism driven by a multi-target genetic algorithm, test resources are preferentially allocated to an attack entry with key service and highest risk so as to significantly optimize computing resource configuration.

Description

Technical Field

[0001] This invention relates to the field of network and information security technology, and in particular to a method for continuous automated vulnerability discovery based on log data. Background Technology

[0002] As the core of interaction in modern information systems, the security of web applications is the cornerstone of cyberspace security. Currently, the main methods for detecting web application security are automated vulnerability scanning and manual penetration testing. Both methods face inherent limitations:

[0003] For automated vulnerability scanning, relying on predefined vulnerability signature libraries and web crawlers to discover attack entry points has inherent drawbacks:

[0004] (1) Incomplete coverage: It relies heavily on the crawler's traversal capabilities and pre-built dictionaries, and is almost powerless against "shadow APIs" or interfaces that are dynamically generated by complex front-end JavaScript and are not explicitly linked in the page, or interfaces that are deeply coupled with business logic. Empirical studies show that traditional crawlers typically have less than 50% coverage of attack entry points for modern single-page applications (SPAs).

[0005] (2) Untimely response: The vulnerability signature database is updated weekly or monthly. However, the time window from the exposure of new attacks (especially zero-day vulnerabilities) to their large-scale abuse has been drastically shortened to a few hours, showing a significant lag in timeliness.

[0006] (3) Resource idleness: The scanner will make indiscriminate and repeated attack attempts on all discovered entry points. Up to 78% of the verification behavior (according to research) targets no risk or low risk targets. These invalid probes lead to serious waste of computing resources.

[0007] (4) Blind spots in logic vulnerabilities: Traditional scanners are good at discovering technical vulnerabilities such as injection (e.g., SQLi, XSS), but they are extremely weak at detecting logical vulnerabilities that require understanding the context of business processes (e.g., insecure direct object reference IDOR, payment bypass, etc.).

[0008] Manual penetration testing, conducted by security experts, is highly accurate in discovering complex vulnerabilities, but its limitations are also significant, mainly as follows:

[0009] (1) High cost: It is essentially a intensive investment of expert knowledge and time, with a long testing cycle (usually more than 72 hours), high human resource costs, and difficulty in scaling up.

[0010] (2) Non-continuous: the test is usually carried out in the form of a project, with an average interval of more than 90 days, and cannot provide 7x24 hours of continuous security assurance, and there is a huge attack window in the rapid iteration of the development process.

[0011] In summary, the prior art, whether it is a machine-centered breadth-first scan (DAST) or an expert-centered depth-first test, has a fundamental contradiction: the complete coverage of attack entry points and the iterative timeliness of attack vectors cannot be achieved. The "impossible triangle" of resource consumption, response speed and continuity guarantee leads to the fact that modern web applications are always exposed to huge security risk openings. SUMMARY

[0012] The present application provides a continuous automated vulnerability mining method based on log data, aiming to break through the limitations of the "impossible triangle" of the prior art and provide a continuous automated vulnerability mining method based on log data and closed-loop self-evolution.

[0013] The technical scheme adopted by the present application is:

[0014] A continuous automated vulnerability mining method based on log data, the method comprising the following steps:

[0015] Step 1, real-time collection of multi-source log data and standardized processing to obtain standard log data and build a standardized log library;

[0016] Step 2, read the standard log data in the standardized log library, mine attack entry points from the read standardized log data based on quantitative information flow analysis of data provenance, and build and dynamically update an attack entry point knowledge base based on the mined attack entry points;

[0017] Step 3, based on the standard log data in the standardized log library, extract and classify the semantic embedding representation (i.e. log embedding representation) of real attack log data according to static semantic analysis, and then use the standard log data corresponding to each cluster center obtained by classification as the first attack vector;

[0018] Based on the standard log data in the standardized log library, construct the request sequence of each user session as time series data, and classify and predict the time series data as normal or malicious according to dynamic time series learning, and use the standard log data corresponding to the time series data classified and predicted as malicious as the second attack vector;

[0019] Based on the first attack vector and the second attack vector, build and dynamically update an attack vector knowledge base;

[0020] Step 4, data is extracted from the attack entry point knowledge base and the attack vector knowledge base, test cases are dynamically combined, and execution priorities are determined based on a risk scheduling strategy of a multi-objective genetic algorithm;

[0021] Wherein, the optimization objectives of the multi-objective genetic algorithm include: maximizing the vulnerability possibility based on the quantitative information flow score, maximizing the business value associated with the configuration management database, and minimizing the test execution cost;

[0022] Step 5, based on the execution priority of the test case, starting from the high execution priority, determine whether the test case has a vulnerability through multi-dimensional feature matching, machine learning model and out-of-band application security testing, and generate a report based on the determination result;

[0023] Step 6, update the attack vector knowledge base through the incremental learning model, and periodically trigger a new round of verification process to form a closed loop.

[0024] Further, in step 1, real-time multi-source log data from web servers, web application firewalls, intrusion detection systems, API gateways, and service meshes and audit logs in cloud-native environments are aggregated through traffic mirroring, file reading, and application programming interface (API) subscription.

[0025] Further, in step 1, standardization processing includes cleaning, deduplication, and standardization to map heterogeneous fields from different sources to standardized fields and filter out invalid probe logs.

[0026] Further, in step 2, the quantitative information flow analysis based on data provenance includes:

[0027] According to the scoring formula Sreflect=(α×I(Input;Output))+(β×Wkeyword)+(γ×∇Eresponse), the standard log data score Sreflect is calculated, and the attack entry point is determined based on the standard log whose score Sreflect is greater than or equal to the score threshold;

[0028] wherein, I (Input; Output) represents a statistical correlation value between the distribution of the input parameter value Input and the distribution of the output response value Output, i.e. the mutual information value between the two, such as the Pearson correlation coefficient, the Spearman rank correlation coefficient, etc.; W keyword represents a preset sensitive keyword weight for evaluating the tendency of the input parameter value triggering dangerous semantics in the response; ∇E response represents a response information entropy gradient for quantifying the disturbance degree of the change of the input parameter value on the response information complexity (measured by Shannon entropy) of the output response value; a, β and γ respectively represent preset weights of I (Input; Output), I (Input; Output) and ∇E response; the value range of the score Sreflect is [0, 1].

[0029] Further, the score threshold is dynamically set, and the working mode is divided into two categories according to the night and daytime working period: normal mode and strict mode, wherein the score threshold in the normal mode is lower than that in the strict mode. For example, the score threshold in the normal mode is set to 0.8, and the score threshold in the strict mode is set to 0.92.

[0030] Further, step 2 further comprises: mining attack entry points from the read standardized log data based on topology traversal, i.e. using a breadth-first search algorithm, constructing a call relationship graph of the parameterized entry point based on the request sequence in the standardized log to determine the attack entry point in the call relationship graph, to form a comprehensive and dynamically updated attack entry point library.

[0031] Further, in step 3, the first attack vector is specifically:

[0032] The data payload (i.e. attack payload) of the standard log data corresponding to the real attack log in the standardized log library is encoded into a log embedding representation using a pre-trained bidirectional encoder model based on Transformers;

[0033] The encoded log embedding representation is clustered based on cosine similarity to obtain the clustering results of different attack patterns;

[0034] The first attack vector is obtained based on the standard log data corresponding to each cluster center in the clustering results.

[0035] Further, in step 3, the dynamic time series learning is specifically: using a model based on a bidirectional long short-term memory network to classify and predict normal and malicious time series data.

[0036] Further, in step 4, the optimization target of the multi-objective genetic algorithm is set as:

[0037] TC={f1:max(Vulnerability Likelihood),f2:max(Business Impact),f3:min(Test Cost)}

[0038] Wherein, TC represents the execution degree of the test case, i.e., the multi-objective fitness, the greater the value is, the higher the execution priority is, f1~f3 represents three optimization objectives, Vulnerability Likelihood represents vulnerability likelihood, which is measured according to the result of data provenance quantitative information flow analysis, i.e., score Sreflect; Business Impact represents business impact, i.e., the business value associated with the configuration management database, and its measurement value is determined based on the normalized value of the business asset value and / or the normalized value of the criticality level of the application programming interface / service corresponding configuration management database or asset inventory; when both the business asset value and the criticality level are included, the measurement value of the business impact is obtained based on the weighted sum of the two; Test Cost represents the test execution cost, which is used to evaluate the expected time and computing resources required to execute the current test case, and the expected time and computing resources are normalized to have a value range of 0~1, and then the weighted sum is obtained to obtain the measurement value of the test execution cost.

[0039] Further, in step 6, the attack vector knowledge base is updated by incremental learning, specifically: the model parameters used for dynamic time series learning in step 3 are continuously optimized by incremental learning; the time series data constructed are classified and predicted as normal and malicious based on the periodically updated model, and the updating of the second attack vector is realized.

[0040] The technical scheme provided by the application at least brings the following beneficial effects:

[0041] (1) The attack surface is fully covered: through data provenance-based quantitative information flow analysis and topology discovery technology, the application can penetrate the application surface, accurately identify dynamic generated interfaces, hidden APIs and deep vulnerability parameters that cannot be touched by traditional technology, and improve the attack entry point coverage rate in real traffic to more than 95%, far exceeding the level of less than 50% of traditional crawlers.

[0042] (2) An adaptive threat immunity capability is built: based on the "static semantics-dynamic time series" double-layer learning model and the closed-loop feedback mechanism, the application can capture, understand and verify new attack patterns including zero-day attacks from real traffic in near real time, shorten the threat response time from several days to several hours, and realize the transformation from "passive response" to "active immunity".

[0043] (3) Achieved the optimization of resource utilization: through the risk intelligent scheduling mechanism driven by the multi-objective genetic algorithm, the test resources are preferentially allocated to the business-critical and highest-risk attack entry, effectively avoiding repeated and invalid testing on low-risk targets, significantly reducing the consumption of computing resources, and focusing security alerts on truly important risks.

[0044] (4) Provided real continuous security assurance: a closed-loop, self-evolving automated system was built, realizing 7x24 hours of uninterrupted vulnerability monitoring, mining and retesting, fundamentally solving the periodic and intermittent defects of manual testing, and providing continuous security verification for web applications that matches the speed of DevOps (Development and Operations). BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0046] Figure 1 The processing flow chart of a continuous automated vulnerability mining method based on log data provided by the embodiment of the present application;

[0047] Figure 2 The architecture diagram of system implementation of a continuous automated vulnerability mining method based on log data provided by the embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the following will combine the drawings in the embodiments of the present application to describe the technical solutions in the embodiments of the present application in detail and completely. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Usually, the components of the embodiments of the present application described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not to limit the scope of the claimed application, but only represents selected embodiments of the present application.

[0049] The embodiment of the present application provides a continuous automatic vulnerability mining method based on log data, which is a full-process closed-loop and self-evolution continuous automatic vulnerability mining method, and mainly solves the technical problems including incomplete attack entry coverage, attack vector update lag, serious verification resource waste and continuous detection capability loss.

[0050] Referring to Figure 1 The continuous automatic vulnerability mining method based on log data provided by the embodiment of the present application comprises the following steps: step one, collection and processing; step two, mining entry points; step three, extracting and learning vectors; step four, risk scheduling verification; step five, determination and analysis; step six, evolution and iteration; wherein, steps two and three are executed in parallel, the outputs of the two knowledge construction processes are used together to execute step four, then step five is executed, and the final result of verification, whether a vulnerability is found or not, will enter step six for optimizing the model and knowledge, and the next round of more intelligent verification is started through a "closed-loop feedback" node. The process highlights the core features of the present application, including continuity, iteration and adaptive learning.

[0051] The specific implementation of each step comprises:

[0052] Step one, collection and processing, that is, realizing multi-source log collection and standardized processing

[0053] Through traffic mirroring, file reading, API subscription, etc., real-time aggregation of multi-source log data from web servers, web application firewalls, intrusion detection systems, API gateways, and service meshes and audit logs in cloud-native environments. Clean, de-duplicate and standardize the original logs, establish a unified log data model, map heterogeneous fields from different sources to standardized fields (such as source IP address, HTTP request uniform resource identifier, user identity identifier, etc.), and filter out invalid probe logs (such as server errors with HTTP status code 5xx or basic probes with request body less than 128 bytes), to obtain the log data, and to build a standardized log library.

[0054] Step two: excavate the entry point, that is, realize the intelligent excavation of the attack entry point

[0055] In the processing of this step, the Data Provenance-based Quantitative Information Flow (DP-QIF) is proposed. The traditional technology only stays in the understanding of "reflection" that whether the user input appears in the response in the form of a string, while the DP-QIF of the present application aims to track the processing, transmission and transformation trajectory of the user input data inside the application (regarded as a black box) in depth, and accurately quantify its substantial influence on the final response based on information theory. Its scoring formula is:

[0056] Sreflect=(α×I(Input;Output))+(β×Wkeyword)+(γ×∇Eresponse)

[0057] Wherein, Sreflect represents the score of the user input data, i.e. the score of the log data, I(Input;Output) represents the statistical correlation between the distribution of the input parameter value and the distribution of the output response (such as the response body, the response header), i.e. the statistical correlation between the distribution of the input parameter value and the distribution of the output response based on mutual information. A high mutual information value means that the input is a strong channel that controls the output, and is likely to be a potential attack entry. Wkeyword is the weight of the sensitive keyword, which evaluates the tendency of the input to trigger dangerous semantics (such as error, exception) in the response. ∇Eresponse is the response information entropy gradient, which is used to quantify the degree of disturbance of the input value to the response information complexity (measured by Shannon entropy). A high gradient value indicates that the input has a significant impact on the system state. Wherein, α, β and γ are the weights of the three measurement values, i.e. the score of the user input data is obtained based on the weighted sum of the three, and the value range of the finally calculated score value is [0, 1].

[0058] Then based on the timing through the dynamically set threshold, the deep attack entry for injection or logical bypass that is ignored by the traditional scanner can be accurately identified, that is, the user input data with the score Sreflect exceeding the current threshold is determined as the attack entry point. At the same time, combined with the breadth-first search (BFS) algorithm, the call relationship graph of the parameterized entry point is constructed based on the request sequence in the log, the undisclosed hidden interface is found, and a comprehensive and dynamically updated attack entry point library is formed.

[0059] Among them, the specific setting of the threshold value can be: the threshold value is set to 0.8 in normal mode, and 0.92 in strict mode, wherein the night (such as 23:00 to 7:00) is set as the strict mode, and the rest of the time period is the normal mode.

[0060] Step three: extracting and learning vectors, that is, realizing attack vector extraction and classification

[0061] This step works with step six to build a "static semantics-dynamic timing" double-layer attack vector knowledge system.

[0062] Static semantic layer: for real attack logs, the BERT (Bidirectional Encoder Representations from Transformers) model pre-trained on a large amount of security corpus is used to encode the attack payload (Payload) into log embedding representation, so as to obtain a high-dimensional vector (such as 768 dimensions). This overcomes the defect that the traditional regular expression cannot understand attack variants (such as encoding, obfuscation), and realizes the accurate capture of attack "intent". Then the encoded log embedding representation is clustered by calculating the cosine similarity, and the log data corresponding to the clustering center obtained based on the clustering processing is used as the attack vector, so as to form a generalizable attack vector and store it in the attack vector knowledge base.

[0063] Dynamic timing layer: this layer focuses on identifying independent requests that seem harmless, but multiple requests form a logical attack chain in time. The request sequence of each user session is modeled as time series data. These sequences are analyzed by a model based on Bi-LSTM to learn the timing patterns of normal business processes and identify malicious activity sequences that deviate from normal patterns (such as payment process bypass, privilege escalation operations, etc.). These identified malicious timing patterns are classified as a kind of attack vector and stored in the attack vector knowledge base. That is, the malicious timing identification model based on Bi-LSTM first extracts features from the input time series data through Bi-LSTM, and then determines whether it is a malicious activity sequence through the classification output layer.

[0064] Step four: risk scheduling verification, that is, realizing dynamic cross-validation execution

[0065] To solve the problem of resource waste, the application introduces a risk-driven intelligent scheduling mechanism. The mechanism uses a multi-objective genetic algorithm (MOGA) to solve the optimal execution sequence of a large number of potential test cases. The core is the design of a multi-objective fitness function, which aims to optimize multiple conflicting objectives at the same time, and its expression is as follows:

[0066] TC={f1:max(Vulnerability Likelihood),f2:max(Business Impact),f3:min(Test Cost)}

[0067] Where TC represents the execution degree of the test case, i.e. the multi-objective fitness, f1~f3 represent the three optimization objectives, Vulnerability Likelihood represents the vulnerability likelihood, which is determined based on the score Sreflect calculated in step two; Business Impact represents the business impact, which is obtained by associating with the configuration management database (CMDB) or asset inventory, and the business asset value and criticality level corresponding to the API or service, i.e. the measurement value of business impact can be directly based on the normalized value of business asset value, or based on the normalized value of criticality level; It can also be the weighted sum of the normalized business asset value and the level system. Test Cost represents the test execution cost, which is used to evaluate the expected time and computing resources required to execute the test case, and to normalize the expected time and computing resources, adjusting their value range to 0~1, and then weighting and summing to obtain the quantitative value of the test cost.

[0068] Through the above design, the application raises the vulnerability mining from "technical discovery" to the height of "risk management", ensuring that the computing resources are always focused on the target that poses the greatest potential threat to the business. The test case is dynamically combined by Cartesian product from the attack entry point library and the attack vector library, and automatically executes fuzzing strategies such as encoding and parameter position mutation to try to bypass the defense.

[0069] Step five: judgment and analysis, i.e. vulnerability verification and result analysis

[0070] A multi-dimensional response analysis mechanism is used to reduce the false positive rate, including:

[0071] (1) In-Band analysis: combining static rule matching (e.g. detecting database error information in responses), dynamic behavior analysis (e.g. monitoring sudden spikes in server CPU usage), and multi-model fusion machine learning judgment. Using pre-trained random forest or SVM anomaly detection models, input response body length, keyword frequency, etc. Features, output the probability of vulnerability existence.

[0072] (2) Adaptive threshold judgment: when the probability exceeds the preset threshold, it is determined that the vulnerability exists. This threshold is not fixed, but uses an adaptive threshold algorithm to dynamically adjust according to the historical response baseline of each API endpoint to adapt to the normal behavior fluctuations of different interfaces and further reduce false positives.

[0073] Among them, the dynamic adjustment according to the historical response baseline of each API endpoint can adopt one of the following two ways:

[0074] Method 1: Based on the mean μ and standard deviation σ of historical response data (such as response length) to set a dynamic boundary. The threshold T is set as: T = μ ± k × σ, where k usually takes the value of 3.

[0075] Method 2: First calculate the two quartiles of historical response data: the first quartile Q1 and the third quartile Q3; and calculate the interquartile range: IQR = Q3-Q1; The upper threshold is set as T_max = Q3 + (1.5 × IQR), and the lower threshold is set as T_min = Q1 - (1.5 × IQR).

[0076] (3) Out-of-Band confirmation: For blind vulnerability (such as Blind SQLi, Blind SSRF), integrate OAST (Out-of-Band Application Security Testing) framework. The generated attack payload will contain a unique uniform resource locator (URL) pointing to a system-controlled callback server. If the backend successfully executes the payload and triggers an out-of-band interaction (such as DNS query or HTTP request), it can confirm the existence of the vulnerability.

[0077] The confirmed vulnerability will automatically generate a detailed report containing the reproduction steps and real-time alarm.

[0078] Step six: evolution and iteration, that is, incremental learning and knowledge base evolution

[0079] This step is responsible for the self-evolution and closed-loop feedback of the knowledge base. It continuously analyzes the newly added standardized logs and the verification results from step five.

[0080] In this embodiment, the knowledge base evolution mechanism is as follows: the Bi-LSTM model described in step three is continuously optimized through incremental learning. When the model identifies a new type of timing attack pattern that is significantly different from the known normal pattern and is verified as malicious (for example, the reconstruction error exceeds the dynamic threshold and the confidence is > 95%), the atomization upgrade of the attack vector knowledge base is automatically triggered, and the attack sequence is patterned and stored in the knowledge base. This update will immediately drive a new round of cross-validation. The organic combination of static semantic understanding (based on the BERT model) and dynamic timing learning (based on the Bi-LSTM model) and continuous evolution enable the present application to have the ability to self-improve from the "attack instruction level" and "attack tactic level" dimensions. The system also periodically performs regression testing on historical repaired vulnerabilities to ensure the effectiveness of the repair.

[0081] In one embodiment, a processing system for implementing a log data-based continuous automated vulnerability mining method of the present application includes three main parts: a data layer, a core system, and an output layer, forming a continuously running closed loop, as shown in Figure 2 The figure shows the main components of the system (data layer, core system, output layer) and the mutual relationship and data flow between the six core modules (M1~M6): starting from multi-source log input, after standard processing by the multi-source log collection and standardization module M1, the log stream continuously supplies the attack entry mining module M2 and the attack vector feature extraction and classification module M3, which update their respective knowledge bases. The dynamic cross-validation module M4 extracts information from these two knowledge bases to generate test cases, and the verification results obtained by the vulnerability verification and result analysis module M5 generate vulnerability reports on the one hand and send key feedback (red dashed arrow in the figure) back to the incremental learning and knowledge base evolution module M6 on the other hand, thereby realizing the self-evolution of the knowledge base and the closed loop operation of the system.

[0082] Among them, the data input source is implemented in the data layer, and the starting point of the system is the log collection and standardization module M1, which aggregates real-time log data from various data sources (DS), and stores it in the standardized log library after processing. Among them, the standardized log library is the data basis for all subsequent analysis of the system. Then, based on the standardized log stream, the system constructs two core knowledge in parallel: the attack entry point intelligent mining module (M2) continuously analyzes the log, identifies potential attack entries using the DP-QIF technology, and dynamically updates the attack entry point library. The attack vector feature extraction and classification module (M3) uses BERT and Bi-LSTM models to learn the static semantics and dynamic timing features of attacks from logs, and constructs and updates the attack vector knowledge base. The dynamic cross-validation execution module (M4) is the active verification engine of the system. It extracts data from the attack entry point library and the attack vector, dynamically combines to generate test cases, and determines the execution priority according to the risk scheduling algorithm of the multi-objective genetic algorithm (MOGA), that is. The vulnerability verification and result analysis module (M5) is responsible for analyzing the execution results of the verification engine. On the one hand, confirmed vulnerabilities will generate reports and alarms through the output layer. On the other hand, all verification results (whether successful or failed) will be fed back to the incremental learning and knowledge base evolution module (M6) as key feedback information. This feedback mechanism enables the system to learn from its own verification activities and continuously optimize its attack knowledge, thereby realizing a true closed-loop self-evolution.

[0083] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0084] The above only describes some embodiments of the present application. For those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1. A method for continuous automated vulnerability discovery based on log data, characterized in that, Includes the following steps: Step 1: Collect multi-source log data in real time and perform standardization processing to obtain standard log data and build a standardized log library; Step 2: Read standard log data from the standardized log library, and mine attack entry points from the read standardized log data based on quantitative information flow analysis of data source tracing; and build and dynamically update the attack entry point knowledge base based on the mined attack entry points. Step 3: Based on the standard log data in the standardized log library, extract and classify the semantic embedding representation of the real attack log data according to static semantic analysis, and then use the standard log data corresponding to each cluster center obtained by classification as the first attack vector. Based on standard log data in the standardized log library, the request sequence of each user session is constructed into time series data. The time series data is classified as normal or malicious based on dynamic time series learning. The standard log data corresponding to the time series data that is classified as malicious is used as the second attack vector. An attack vector knowledge base is constructed and dynamically updated based on the first and second attack vectors; Step 4: Extract data from the attack entry point knowledge base and the attack vector knowledge base, dynamically combine them to generate test cases, and determine the execution priority based on the risk scheduling strategy of the multi-objective genetic algorithm; The optimization objectives of the multi-objective genetic algorithm include: maximizing the vulnerability probability based on quantitative information flow scoring, maximizing the business value associated with the configuration management database, and minimizing test execution costs. Step 5: Based on the execution priority of the test cases, starting with the highest execution priority, determine whether vulnerabilities exist in the test cases through multi-dimensional feature matching, machine learning models, and out-of-band application security testing, and generate a report based on the determination results; Step 6: Update the attack vector knowledge base through incremental learning model and periodically trigger a new round of verification process to form a closed loop.

2. The continuous automated vulnerability discovery method based on log data as described in claim 1, characterized in that, In step 1, multi-source log data from web servers, web application firewalls, intrusion detection systems, API gateways, and service meshes and audit logs in cloud-native environments are aggregated in real time through traffic mirroring, file reading, and application programming interface (API) subscription.

3. A continuous automated vulnerability discovery method based on log data as described in claim 1 or 2, characterized in that, In step 1, the standardization process includes cleaning, deduplication, and standardization to map heterogeneous fields from different sources to normalized fields and filter out invalid probe logs.

4. The continuous automated vulnerability discovery method based on log data as described in claim 1, characterized in that, Step 2, the quantitative information flow analysis based on data tracing includes: The standard log data score Sreflect is calculated according to the scoring formula Sreflect=(α×I(Input;Output))+(β×Wkeyword)+(γ×∇Eresponse), and its value range is [0,1]. Then, the attack entry point is determined based on the standard logs whose score Sreflect is greater than or equal to the scoring threshold. Where I(Input;Output) represents the statistical correlation between the distribution of the input parameter value Input and the distribution of the output response value Output, i.e., the mutual information value between the two; Wkeyword represents the preset sensitive keyword weight; ∇Eresponse represents the response information entropy gradient, and α, β and γ represent the preset weights of I(Input;Output), I(Input;Output) and ∇Eresponse, respectively.

5. The continuous automated vulnerability discovery method based on log data as described in claim 1, characterized in that, The scoring threshold is dynamically set, and the daily work mode is divided into two categories according to the night and day work periods: normal mode and strict mode. The scoring threshold in normal mode is lower than that in strict mode.

6. The continuous automated vulnerability discovery method based on log data as described in claim 1, characterized in that, Step 2 also includes: mining attack entry points from the read standardized log data based on topological traversal, that is, using a breadth-first search algorithm to construct a call relationship graph of parameterized entry points based on the request sequence in the standardized log, so as to determine the attack entry points in the call relationship graph.

7. The continuous automated vulnerability discovery method based on log data as described in claim 1, characterized in that, In step 3, the first attack vector is specifically: A pre-trained Transformers-based bidirectional encoder model is used to encode the data payload of standard log data corresponding to real attack logs in a standardized log library into a log embedding representation. The encoded log embedding representation is clustered based on cosine similarity to obtain clustering results for different attack modes. The first attack vector is obtained based on the standard log data corresponding to each cluster center in the clustering results.

8. The continuous automated vulnerability discovery method based on log data as described in claim 1, characterized in that, In step 3, dynamic time series learning specifically involves using a model based on a bidirectional long short-term memory network to classify and predict whether the time series data is normal or malicious.

9. The continuous automated vulnerability discovery method based on log data as described in claim 1, characterized in that, In step 4, the optimization objective of the multi-objective genetic algorithm is set as follows: TC={f1:max(Vulnerability Likelihood),f2:max(Business Impact),f3:min(TestCost)} Where TC represents the execution degree of the test case, the larger the value, the higher the execution priority; f1~f3 represent three optimization objectives: Vulnerability Likelihood, which represents the likelihood of a vulnerability and is measured based on the results of quantitative information flow analysis of data source tracing; Business Impact, which represents the business value associated with the configuration management database and is determined based on the normalized value of the business asset value and / or the normalized value of the criticality level of the configuration management database or asset list corresponding to the application programming interface / service; when both business asset value and criticality level are included, the business impact metric is obtained based on the weighted sum of the two; and Test Cost, which represents the test execution cost and is used to assess the expected time and computing resources required to execute the current test case. The expected time and computing resources are normalized to a value range of 0~1, and then weighted summed to obtain the test execution cost metric.

10. The continuous automated vulnerability discovery method based on log data as described in claim 1, characterized in that, In step 6, updating the attack vector knowledge base through incremental learning specifically involves: continuously optimizing the model parameters used for dynamic time-series learning in step 3 through incremental learning; and using the periodically updated model to perform normal and malicious classification predictions on the constructed time-series data to update the second attack vector.

Images