A method for continuous automated vulnerability mining based on log data

Abstract

The application discloses a kind of based on log data's continuous automation vulnerability mining method, belong to network and information security technical field.The application can penetrate application surface layer, accurately identify dynamic generation interface, hidden application program interface and deep vulnerability parameter that traditional technology cannot touch, to significantly improve attack entry point coverage in real traffic, realize panoramic coverage of attack surface by quantitative information flow analysis and topology discovery based on data tracing.The application can capture, understand and verify new attack mode including zero-day attack from real traffic in near real time based on static semantics-dynamic time sequence double-layer learning model and closed-loop feedback mechanism, shorten threat response time from several days to several hours, build adaptive threat immunity ability;Through the risk intelligent scheduling mechanism driven by multi-objective genetic algorithm, test resources are preferentially allocated to business critical and highest risk attack entry, to significantly optimize computing resource allocation.

Description

Technical Field

[0001] This invention relates to the field of network and information security technology, and in particular to a method for continuous automated vulnerability discovery based on log data. Background Technology

[0002] As the core of interaction in modern information systems, the security of web applications is the cornerstone of cyberspace security. Currently, the main methods for detecting web application security are automated vulnerability scanning and manual penetration testing. Both methods face inherent limitations:

[0003] For automated vulnerability scanning, relying on predefined vulnerability signature libraries and web crawlers to discover attack entry points has inherent drawbacks:

[0004] (1) Incomplete coverage: It relies heavily on the crawler's traversal capabilities and pre-built dictionaries, and is almost powerless against "shadow APIs" or interfaces that are dynamically generated by complex front-end JavaScript and are not explicitly linked in the page, or interfaces that are deeply coupled with business logic. Empirical studies show that traditional crawlers typically have less than 50% coverage of attack entry points for modern single-page applications (SPAs).

[0005] (2) Untimely response: The vulnerability signature database is updated weekly or monthly. However, the time window from the exposure of new attacks (especially zero-day vulnerabilities) to their large-scale abuse has been drastically shortened to a few hours, showing a significant lag in timeliness.

[0006] (3) Resource idleness: The scanner will make indiscriminate and repeated attack attempts on all discovered entry points. Up to 78% of the verification behavior (according to research) targets no risk or low risk targets. These invalid probes lead to serious waste of computing resources.

[0007] (4) Blind spots in logic vulnerabilities: Traditional scanners are good at discovering technical vulnerabilities such as injection (e.g., SQLi, XSS), but they are extremely weak at detecting logical vulnerabilities that require understanding the context of business processes (e.g., insecure direct object reference IDOR, payment bypass, etc.).

[0008] Manual penetration testing, conducted by security experts, is highly accurate in discovering complex vulnerabilities, but its limitations are also significant, mainly as follows:

[0009] (1) High cost: It is essentially a intensive investment of expert knowledge and time, with a long testing cycle (usually more than 72 hours), high human resource costs, and difficulty in scaling up.

[0010] (2) Non-continuous: Tests are usually conducted in the form of projects with an average interval of more than 90 days, which cannot provide continuous security 24 / 7 and presents a huge attack window in the rapid iterative development process.

[0011] In summary, existing technologies, whether machine-based breadth-first scanning (DAST) or expert-based depth-first testing, suffer from a fundamental contradiction: the completeness of attack entry point coverage and the timeliness of attack vector iteration cannot be simultaneously achieved. These two aspects form an "impossible triangle" in terms of resource consumption, response speed, and continuity assurance, leaving modern web applications constantly exposed to significant security risks. Summary of the Invention

[0012] This invention provides a continuous automated vulnerability discovery method based on log data, aiming to break the limitations of the "impossible triangle" in existing technologies and provide a closed-loop, self-evolving continuous automated vulnerability discovery method based on log data.

[0013] The technical solution adopted in this invention is as follows:

[0014] A continuous automated vulnerability discovery method based on log data, comprising the following steps:

[0015] Step 1: Collect multi-source log data in real time and perform standardization processing to obtain standard log data and build a standardized log library;

[0016] Step 2: Read standard log data from the standardized log library, and mine attack entry points from the read standardized log data based on quantitative information flow analysis of data source tracing; and build and dynamically update the attack entry point knowledge base based on the mined attack entry points.

[0017] Step 3: Based on the standard log data in the standardized log library, extract and classify the semantic embedding representation (i.e., log embedding representation) of the real attack log data according to static semantic analysis, and then use the standard log data corresponding to each cluster center obtained by classification as the first attack vector.

[0018] Based on standard log data in the standardized log library, the request sequence of each user session is constructed into time series data. The time series data is classified as normal or malicious based on dynamic time series learning. The standard log data corresponding to the time series data that is classified as malicious is used as the second attack vector.

[0019] An attack vector knowledge base is constructed and dynamically updated based on the first and second attack vectors;

[0020] Step 4: Extract data from the attack entry point knowledge base and the attack vector knowledge base, dynamically combine them to generate test cases, and determine the execution priority based on the risk scheduling strategy of the multi-objective genetic algorithm;

[0021] The optimization objectives of the multi-objective genetic algorithm include: maximizing the vulnerability probability based on quantitative information flow scoring, maximizing the business value associated with the configuration management database, and minimizing test execution costs.

[0022] Step 5: Based on the execution priority of the test cases, starting with the highest execution priority, determine whether vulnerabilities exist in the test cases through multi-dimensional feature matching, machine learning models, and out-of-band application security testing, and generate a report based on the determination results;

[0023] Step 6: Update the attack vector knowledge base through incremental learning model and periodically trigger a new round of verification process to form a closed loop.

[0024] Furthermore, in step 1, multi-source log data from web servers, web application firewalls, intrusion detection systems, API gateways, and service meshes and audit logs in cloud-native environments are aggregated in real time through traffic mirroring, file reading, and application programming interface (API) subscription.

[0025] Furthermore, in step 1, the standardization process includes cleaning, deduplication, and standardization to map heterogeneous fields from different sources to normalized fields and filter out invalid probe logs.

[0026] Furthermore, in step 2, the quantitative information flow analysis based on data tracing includes:

[0027] The standard log data score Sreflect is calculated according to the scoring formula Sreflect=(α×I(Input;Output))+(β×Wkeyword)+(γ×∇Eresponse); then the attack entry point is determined based on the standard logs whose score Sreflect is greater than or equal to the scoring threshold.

[0028] Wherein, I(Input;Output) represents the statistical correlation between the distribution of the input parameter value Input and the distribution of the output response value Output, i.e., the mutual information value between the two, such as the Wilson correlation coefficient, Spearman rank correlation coefficient, etc.; Wkeyword represents the pre-set sensitive keyword weight, which is used to evaluate the tendency of the input parameter value to trigger dangerous semantics in the response; ∇Eresponse represents the response information entropy gradient, which is used to quantify the degree of perturbation of the response information complexity (measured by Shannon entropy) of the output response value by the change of the input parameter value; α, β and γ represent the pre-set weights of I(Input;Output), I(Input;Output) and ∇Eresponse, respectively; the value range of the score Sreflect is [0,1].

[0029] Furthermore, the scoring thresholds are dynamically set, dividing the daily work mode into two categories based on nighttime and daytime work periods: Normal Mode and Strict Mode. The scoring threshold for Normal Mode is lower than that for Strict Mode. For example, the scoring threshold for Normal Mode is set to 0.8, and the scoring threshold for Strict Mode is set to 0.92.

[0030] Furthermore, step 2 also includes: mining attack entry points from the read standardized log data based on topological traversal, that is, using a breadth-first search algorithm to construct a call relationship graph of parameterized entry points based on the request sequence in the standardized log, so as to determine the attack entry points in the call relationship graph and form a comprehensive and dynamically updated attack entry point library.

[0031] Furthermore, in step 3, the first attack vector is specifically:

[0032] A pre-trained Transformers-based bidirectional encoder model is used to encode the data payload (i.e., attack payload) of the standard log data corresponding to the real attack logs in the standardized log library into a log embedding representation;

[0033] The encoded log embedding representation is clustered based on cosine similarity to obtain clustering results for different attack modes.

[0034] The first attack vector is obtained based on the standard log data corresponding to each cluster center in the clustering results.

[0035] Furthermore, in step 3, dynamic time series learning specifically involves using a model based on a bidirectional long short-term memory network to classify and predict the time series data as normal or malicious.

[0036] Furthermore, in step 4, the optimization objective of the multi-objective genetic algorithm is set as follows:

[0037] TC={f1:max(Vulnerability Likelihood),f2:max(Business Impact),f3:min(Test Cost)}

[0038] Among them, TC represents the execution degree of the test case, i.e., multi-objective fitness. The larger the value, the higher the execution priority. f1~f3 represent three optimization objectives. Vulnerability Likelihood represents the vulnerability probability, which is measured based on the results of quantitative information flow analysis of data tracing, i.e., the score Sreflect. Business Impact represents the business impact, i.e., the business value associated with the configuration management database. Its measurement value is determined based on the normalized value of the business asset value and / or the normalized value of the criticality level of the configuration management database or asset list corresponding to the application programming interface / service. When both business asset value and criticality level are included, the measurement value of business impact is obtained based on the weighted sum of the two. Test Cost represents the test execution cost, which is used to evaluate the expected time and computing resources required to execute the current test case. The expected time and computing resources are normalized to make their value range 0~1, and then weighted to obtain the measurement value of test execution cost.

[0039] Furthermore, in step 6, updating the attack vector knowledge base through incremental learning specifically involves: continuously optimizing the model parameters used for dynamic time-series learning in step 3 through incremental learning; and performing normal and malicious classification predictions on the constructed time-series data based on the periodically updated model to update the second attack vector.

[0040] The technical solution provided by this invention brings at least the following beneficial effects:

[0041] (1) Achieved panoramic coverage of the attack surface: Through quantitative information flow analysis and topology discovery technology based on data tracing, this invention can penetrate the application surface and accurately identify dynamically generated interfaces, hidden APIs and deep vulnerability parameters that traditional technologies cannot reach, increasing the coverage of attack entry points in real traffic to more than 95%, far exceeding the level of less than 50% of traditional crawlers.

[0042] (2) An adaptive threat immunity capability has been constructed: Based on the "static semantics-dynamic temporal" two-layer learning model and closed-loop feedback mechanism, this invention can capture, understand and verify new attack patterns, including zero-day attacks, from real traffic in near real time, shortening the threat response time from several days to several hours, and realizing the transformation from "passive response" to "active immunity".

[0043] (3) Optimization of resource utilization: Through the risk intelligent scheduling mechanism driven by multi-objective genetic algorithm, test resources are preferentially allocated to the attack entry points that are critical to the business and have the highest risk, which effectively avoids repeated and invalid testing of low-risk targets, significantly reduces the consumption of computing resources, and focuses security alerts on truly important risks.

[0044] (4) Provides true continuous security assurance: It has built a closed-loop, self-evolving automated system that enables 24 / 7 uninterrupted vulnerability monitoring, discovery and retesting, fundamentally solving the periodic and intermittent defects of manual testing, and providing Web applications with continuous security verification that matches the speed of DevOps (Development and Operations). Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating a continuous automated vulnerability discovery method based on log data, provided in an embodiment of the present invention;

[0047] Figure 2 This is an architecture diagram of a system implementation of a continuous automated vulnerability discovery method based on log data, provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0049] This invention provides a continuous automated vulnerability discovery method based on log data. It is a closed-loop, self-evolving, continuous automated vulnerability discovery method that primarily addresses the following technical problems: incomplete attack entry point coverage, delayed attack vector updates, severe waste of verification resources, and lack of continuous detection capabilities. This invention achieves comprehensive discovery and verification of all potential attack entry points through in-depth analysis and data tracing of real application traffic logs, including hidden APIs and "shadow APIs" that traditional crawlers cannot discover, thus solving the problem of incomplete attack entry point coverage. It establishes an attack knowledge system capable of self-learning and evolving from real attack traffic, enabling adaptive capture and application of new and variant attack patterns to address the problem of delayed attack vector updates. It introduces an intelligent scheduling mechanism based on business risk and threat intelligence, precisely focusing computing resources on high-risk targets that pose the greatest potential threat to the business, greatly reducing ineffective testing and thus solving the problem of severe waste of verification resources. Finally, it constructs a 24 / 7 uninterrupted automated monitoring, learning, and verification pipeline, transforming security verification from an independent post-event remediation step into a real-time assurance process parallel to the application's lifecycle, thereby solving the problem of lack of continuous detection capabilities.

[0050] See Figure 1 This invention provides a continuous automated vulnerability discovery method based on log data, comprising: Step 1: data collection and processing; Step 2: entry point discovery; Step 3: vector extraction and learning; Step 4: risk scheduling and verification; Step 5: judgment and analysis; Step 6: evolution and iteration. Steps 2 and 3 are executed in parallel. The outputs of these two knowledge construction processes are used to execute Step 4, followed by Step 5. The final verification result, regardless of whether a vulnerability is found, proceeds to Step 6 for optimizing the model and knowledge, and initiates the next round of more intelligent verification through a "closed-loop feedback" node. This process highlights the core characteristics of this invention: continuity, iteration, and adaptive learning.

[0051] The specific implementation of the above steps includes:

[0052] Step 1: Collection and Processing, i.e., realizing multi-source log collection and standardized processing.

[0053] By employing methods such as traffic mirroring, file reading, and API subscription, real-time aggregation of log data from multiple sources, including web servers, web application firewalls, intrusion detection systems, API gateways, and service meshes and audit logs in cloud-native environments, is achieved. The raw logs are cleaned, deduplicated, and standardized to establish a unified log data model. Heterogeneous fields from different sources are mapped to normalized fields (such as source IP addresses, HTTP request Uniform Resource Identifiers, user identities, etc.), and invalid probe logs (such as server errors with HTTP status codes of 5xx or basic probes with request bodies less than 128 bytes) are filtered out. This process yields labeled log data, enabling the construction of a standardized log library.

[0054] Step 2: Discover entry points, i.e., achieve intelligent discovery of attack entry points.

[0055] In this step, a Data Provenance-based Quantitative Information Flow (DP-QIF) analysis is proposed. Traditional techniques understand "reflection" only by whether user input appears as a string in the response. In contrast, DP-QIF aims to deeply trace the processing, transmission, and transformation of user input data within the application (considered a black box), and precisely quantify its substantial impact on the final response based on information theory. Its scoring formula is as follows:

[0056] Sreflect=(α×I(Input;Output))+(β×Wkeyword)+(γ×∇Eresponse)

[0057] Here, Sreflect represents the rating of the user input data, i.e., the rating of the log data. I(Input;Output) represents the statistical correlation between the distribution of input parameter values ​​and the distribution of output response values ​​(such as response body and response header), i.e., the statistical correlation between the distribution of input parameter values ​​and the distribution of output response values ​​based on mutual information. A high mutual information value means that the input is a strong channel controlling the output and is very likely a potential attack entry point. Wkeyword is the sensitive keyword weight, which evaluates the tendency of the input to trigger dangerous semantics (such as error, exception) in the response. ∇Eresponse is the response information entropy gradient, which is used to quantify the degree of perturbation of the response information complexity (measured by Shannon entropy) by small changes in the input value. A high gradient value indicates that the input has a significant impact on the system state. α, β, and γ are the weights of the three metrics, i.e., the rating of the user input data is obtained based on the weighted sum of the three, and the final calculated rating value ranges from [0,1].

[0058] Then, based on time-series data and dynamically set thresholds, deep attack entry points used for injection or logic bypass that are ignored by traditional scanners can be accurately identified. Specifically, user input data with a score (Sreflect) exceeding the current threshold is identified as attack entry points. Simultaneously, by combining a breadth-first search (BFS) algorithm, a call relationship graph of parameterized entry points is constructed based on request sequences in logs, uncovering undisclosed hidden interfaces and forming a comprehensive, dynamically updated attack entry point library.

[0059] The specific threshold settings are as follows: in normal mode, the threshold is set to 0.8, and in strict mode, it is set to 0.92. In this case, nighttime (such as from 11 p.m. to 7 a.m.) is set to strict mode, and the rest of the time period is set to normal mode.

[0060] Step 3: Extract and learn vectors, i.e., perform attack vector extraction and classification.

[0061] This step works in conjunction with step six to construct a two-layer attack vector knowledge system of "static semantics - dynamic temporal sequence".

[0062] Static Semantic Layer: For real attack logs, the attack payload is encoded into log embedding representations using a BERT (Bidirectional Encoder Representations from Transformers) model pre-trained on a massive security corpus, resulting in high-dimensional vectors (e.g., 768-dimensional). This overcomes the limitation of traditional regular expressions in understanding attack variants (such as encoding and obfuscation), achieving accurate capture of attack "intent." Then, the encoded log embedding representations are clustered into attack patterns using cosine similarity calculation. The log data corresponding to the cluster centers obtained from this clustering process are then used as attack vectors, forming generalizable attack vectors, which are stored in an attack vector knowledge base.

[0063] Dynamic Temporal Layer: This layer focuses on identifying seemingly harmless independent requests that, when linked temporally, form logical attack chains. The request sequence of each user session is modeled as time-series data. These sequences are analyzed using a model based on Bidirectional Long Short-Term Memory (Bi-LSTM) networks to learn the temporal patterns of normal business processes and identify malicious activity sequences that deviate from these patterns (such as payment process bypassing, privilege escalation operations, etc.). These identified malicious temporal patterns are classified and stored in an attack vector knowledge base as attack vectors. In other words, the malicious temporal identification model based on Bi-LSTM networks first extracts features from the input time-series data using the Bi-LSTM network, and then uses a classification output layer to determine whether it is a malicious activity sequence.

[0064] Step 4: Risk scheduling verification, i.e., implementing dynamic cross-validation.

[0065] To address the problem of resource waste, this invention introduces a risk-driven intelligent scheduling mechanism. This mechanism employs a multi-objective genetic algorithm (MOGA) to solve for the optimal execution sequence of massive amounts of potential test cases. Its core lies in the design of a multi-objective fitness function, which aims to simultaneously optimize multiple conflicting objectives, as expressed below:

[0066] TC={f1:max(Vulnerability Likelihood),f2:max(Business Impact),f3:min(Test Cost)}

[0067] Here, TC represents the test case execution degree, i.e., multi-objective fitness; f1~f3 represent three optimization objectives; Vulnerability Likelihood represents the vulnerability probability, determined by the score Sreflect calculated in step two; Business Impact represents the business impact, obtained by associating with the Configuration Management Database (CMDB) or asset inventory to obtain the business asset value and criticality level corresponding to the API or service. That is, the measure of business impact can be directly based on the normalized value of business assets, or based on the normalized value of criticality level; or it can be a weighted sum of the normalized business asset value and the criticality level. Test Cost represents the test execution cost, used to assess the expected time and computing resources required to execute the test case. The expected time and computing resources are normalized, their value range is adjusted to between 0 and 1, and then weighted to obtain the quantitative value of the test cost.

[0068] Through the above design, this invention elevates vulnerability discovery from the level of "technical discovery" to the level of "risk management," ensuring that computing resources are always focused on targets that pose the greatest potential threat to business operations. Test cases are dynamically generated by combining attack entry point libraries and attack vector libraries through Cartesian product, and automatically execute fuzzing strategies such as encoding and parameter position mutation to attempt to bypass defenses.

[0069] Step 5: Judgment and Analysis, i.e., vulnerability verification and result analysis.

[0070] A multi-dimensional response analysis mechanism is employed to reduce the false alarm rate, including:

[0071] (1) In-band analysis: Combining static rule matching (such as detecting database error information in the response), dynamic behavior analysis (such as monitoring sudden increases in server CPU usage), and machine learning judgment through multi-model fusion. Using pre-trained random forest or SVM anomaly detection models, inputting features such as response body length and keyword frequency, outputting the probability of vulnerability existence.

[0072] (2) Adaptive threshold determination: When the probability exceeds a preset threshold, the vulnerability is determined to exist. This threshold is not fixed, but is dynamically adjusted according to the historical response baseline of each API endpoint using an adaptive threshold algorithm to adapt to the normal behavior fluctuations of different interfaces and further reduce false alarms.

[0073] Dynamic adjustment based on the historical response baseline of each API endpoint can be achieved using one of the following two methods:

[0074] Method 1: Set dynamic boundaries based on the mean μ and standard deviation σ of historical response data (e.g., response length). The threshold T is set as: T = μ ± k × σ, where k is typically 3.

[0075] Method 2: First, calculate the two quartiles of the historical response data: the first quartile Q1 and the third quartile Q3; and calculate the interquartile range: IQR = Q3 - Q1; set the upper threshold to T_max = Q3 + (1.5 × IQR) and the lower threshold to T_min = Q1 - (1.5 × IQR).

[0076] (3) Out-of-Band Verification: For blind SQL injection vulnerabilities (such as Blind SQLi, Blind SSRF), integrate the OAST (Out-of-Band Application Security Testing) framework. The generated attack payload will contain a unique Uniform Resource Locator (URL) pointing to the callback server controlled by the system. If the backend successfully executes the payload and triggers out-of-band interactions (such as DNS queries or HTTP requests), the vulnerability can be confirmed.

[0077] Confirmed vulnerabilities will automatically generate a detailed report containing reproduction steps and will trigger real-time alerts.

[0078] Step Six: Evolution and Iteration, i.e., achieving incremental learning and knowledge base evolution.

[0079] This step is responsible for the self-evolution and closed-loop feedback of the knowledge base. It continuously analyzes newly added standardized logs and the verification results from step five.

[0080] In this embodiment, the knowledge base evolution mechanism employed is as follows: the Bi-LSTM model described in step three is continuously optimized through incremental learning. When the model identifies a novel temporal attack pattern that is significantly different from known normal patterns and verified as malicious (e.g., the reconstruction error exceeds a dynamic threshold and the confidence level is >95%), an atomic upgrade of the attack vector knowledge base is automatically triggered, and the attack sequence is patterned and stored in the knowledge base. This update immediately drives a new round of cross-validation. This organic combination and continuous evolution of static semantic understanding (based on the BERT model) and dynamic temporal learning (based on the Bi-LSTM model) enables the present invention to have the ability to improve itself from both the "attack instruction level" and the "attack tactical level." The system also periodically performs regression testing on historically patched vulnerabilities to ensure the effectiveness of the patching.

[0081] In one embodiment, a processing system for implementing the continuous automated vulnerability mining method based on log data of the present invention includes three main parts: a data layer, a core system, and an output layer, forming a continuously running closed loop, such as... Figure 2 As shown in the diagram, this diagram illustrates the main components of the system (data layer, core system, output layer) and the interrelationships and data flow between the six core modules (M1~M6): Starting with multi-source log input, after standard processing by the multi-source log collection and standardization module M1, the log stream continuously supplies the attack entry point mining module M2 and the attack vector feature extraction and classification module M3, which update their respective knowledge bases. The dynamic cross-validation module M4 extracts information from these two knowledge bases to generate test cases. The validation results obtained by the vulnerability verification and result analysis module M5 generate vulnerability reports on the one hand, and send them back to the incremental learning and knowledge base evolution module M6 as key feedback (indicated by the red dashed arrows in the diagram) on the other hand, thereby realizing the self-evolution of the knowledge base and the closed-loop operation of the system.

[0082] The system's data input source is implemented at the data layer, starting with the log collection and standardization module M1. This module aggregates log data from various data sources (DS) in real time, processes it, and stores it in a standardized log library. This standardized log library forms the data foundation for all subsequent analyses. Based on the standardized log stream, the system constructs two core knowledge bases in parallel: the attack entry point intelligent mining module (M2) continuously analyzes logs, uses DP-QIF technology to identify potential attack entry points, and dynamically updates the attack entry point library; the attack vector feature extraction and classification module (M3) uses BERT and Bi-LSTM models to learn the static semantics and dynamic temporal features of attacks from logs, constructing and updating the attack vector knowledge base. The dynamic cross-validation execution module (M4) is the system's active verification engine. It extracts data from the attack entry point library and attack vectors, dynamically combines them to generate test cases, and determines the execution priority based on the risk scheduling algorithm of the multi-objective genetic algorithm (MOGA). The vulnerability verification and result analysis module (M5) is responsible for analyzing the execution results of the verification engine. Confirmed vulnerabilities generate reports and alerts through the output layer. On the other hand, all verification results (whether successful or not) are fed back as key feedback information to the incremental learning and knowledge base evolution module (M6). This feedback mechanism enables the system to learn from its own verification activities, continuously optimize its attack knowledge, and thus achieve true closed-loop self-evolution.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0084] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A method for continuous automated vulnerability mining based on log data, characterized in that, The method comprises the following steps: Step 1, real-time collection of multi-source log data and standardization processing to obtain standard log data and build a standardized log library; Step 2, reading the standard log data in the standardized log library, mining attack entry points from the read standard log data based on data source-based quantitative information flow analysis, and building and dynamically updating an attack entry point knowledge base based on the mined attack entry points; Step 3, based on the standard log data in the standardized log library, extracting and classifying the semantic embedding representation of the real attack log data according to static semantic analysis, and taking the standard log data corresponding to each cluster center obtained by classification as a first attack vector; Based on the standard log data in the standardized log library, constructing the request sequence of each user session as time series data, classifying and predicting normal and malicious according to dynamic time series learning, and taking the standard log data corresponding to the time series data classified and predicted as malicious as a second attack vector; Based on the first attack vector and the second attack vector, building and dynamically updating an attack vector knowledge base; Step 4, extracting data from the attack entry point knowledge base and the attack vector knowledge base, dynamically combining to generate test cases, and determining the execution priority based on a risk scheduling strategy of a multi-objective genetic algorithm; Wherein, the optimization objectives of the multi-objective genetic algorithm include: maximizing the vulnerability possibility based on the quantitative information flow score, maximizing the business value associated with the configuration management database, and minimizing the test execution cost; Step 5, based on the execution priority of the test cases, starting from the high execution priority, determining whether the test cases have vulnerabilities through multi-dimensional feature matching, machine learning models, and out-of-band application security testing, and generating a report based on the determination result; Step 6, updating the attack vector knowledge base through an incremental learning model, and periodically triggering a new round of verification process to form a closed loop.

2. The method of claim 1, wherein, In step 1, multi-source log data from web servers, web application firewalls, intrusion detection systems, API gateways, and service meshes and audit logs in cloud-native environments are aggregated in real time through traffic mirroring, file reading, and application programming interface (API) subscription.

3. The method of claim 1 or 2, wherein, In step 1, standardization processing includes cleaning, deduplication, and standardization to map heterogeneous fields from different sources to standardized fields and filter out invalid probe logs.

4. The method of claim 1, wherein, In step 2, data source-based quantitative information flow analysis includes: Calculating the standard log data score Sreflect according to the score formula Sreflect=(α×I(Input;Output))+(β×Wkeyword)+(γ×∇Eresponse), and the value range is [0, 1], and then determining the attack entry point based on the standard log whose score Sreflect is greater than or equal to the score threshold; Wherein, I (Input; Output) represents the statistical correlation value between the distribution of the input parameter value Input and the distribution of the output response value Output, that is, the mutual information value between the two; Wkeyword represents the preset weight of the sensitive keyword; ∇Eresponse represents the response information entropy gradient, and alpha, beta and gamma respectively represent the preset weights of I (Input; Output), I (Input; Output) and ∇Eresponse.

5. The method of continuous automated vulnerability mining based on log data as claimed in claim 1, wherein, The score threshold is dynamically set, and the working mode is divided into two categories according to the night and daytime working period: normal mode and strict mode, wherein the score threshold in the normal mode is lower than that in the strict mode.

6. The method of continuous automated vulnerability mining based on log data as claimed in claim 1, wherein, Step 2 further comprises: mining attack entry points from the read standardized log data based on topology traversal, that is, using a breadth-first search algorithm, constructing a call relationship graph of parameterized entry points based on the request sequence in the standardized log to determine the attack entry points in the call relationship graph.

7. The method of continuous automated vulnerability mining based on log data as claimed in claim 1, wherein, In step 3, the first attack vector is specifically: The data payload of the standard log data corresponding to the real attack log in the standardized log library is encoded into a log embedding representation by using a pre-trained bidirectional encoder model based on Transformers; The encoded log embedding representation is clustered based on cosine similarity to obtain the clustering results of different attack modes. The first attack vector is obtained based on the standard log data corresponding to each cluster center in the clustering results.

8. The method of continuous automated vulnerability mining based on log data as claimed in claim 1, wherein, In step 3, dynamic time series learning is specifically: using a model based on bidirectional long short-term memory network to classify and predict normal and malicious time series data.

9. The method of claim 1, wherein, In step 4, the optimization objectives of the multi-objective genetic algorithm are set as: TC={f1:max(Vulnerability Likelihood),f2:max(Business Impact),f3:min(TestCost)} Wherein, TC represents the execution degree of the test case, and the greater the value is, the higher the execution priority is; f1~f3 represent three optimization objectives, Vulnerability Likelihood represents vulnerability likelihood, which is measured according to the results of quantitative information flow analysis of data tracing; Business Impact represents the business value associated with the configuration management database, and its measurement value is determined based on the normalized value of the business asset value and / or the normalized value of the criticality level of the application programming interface / service corresponding configuration management database or asset list; when the business asset value and criticality level are included at the same time, the measurement value of business impact is obtained based on the weighted sum of the two; Test Cost represents the test execution cost, which is used to evaluate the expected time and computing resources required to execute the current test case, and the expected time and computing resources are normalized to have a value range of 0~1, and then the weighted sum is obtained to get the measurement value of the test execution cost.

10. The method of continuous automated vulnerability mining based on log data as claimed in claim 1, wherein, In step 6, the attack vector knowledge base is updated by incremental learning model, specifically: the model parameters used for dynamic time series learning in step 3 are continuously optimized by incremental learning; the time series data constructed is classified and predicted as normal and malicious based on the periodically updated model, and the second attack vector is updated.

Images