A vulnerability and threat scanning method and system based on artificial intelligence

Abstract

The present invention discloses a vulnerability and threat scanning method and system based on artificial intelligence, and specifically relates to the field of artificial intelligence technology. By extracting known threats and potential zero-day attack data from a real environment to train an AI model, combined with generating high-complexity and low-abnormality and low-complexity and high-abnormality simulated attack samples, the AI ​​model is tested for threat detection in diversified scenarios, and its response capability to high-complexity attacks is analyzed. By recording and counting false positives and false negatives, the accuracy of the model in identifying unknown new threat features is comprehensively evaluated. When it is detected that the model has inaccurate recognition of the unknown new threat features, by predicting the degree of abnormality within a fixed time period, an early warning signal is generated and the model is optimized, which can not only significantly improve the AI ​​model's detection capability for complex disguised attacks, but also promptly discover and repair the model's blind spots in the actual environment, effectively preventing zero-day vulnerabilities and advanced persistent threats from breaking through the system.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based vulnerability and threat scanning method and system. Background Art

[0002] AI-based vulnerability and threat scanning uses artificial intelligence technology to automatically detect computer systems, networks, and applications to identify potential security vulnerabilities and threats. Through machine learning and data analysis, AI can extract patterns from large amounts of historical data and real-time activities to detect complex threats that may be overlooked by traditional methods. Compared with traditional scanning tools, AI can not only identify known vulnerabilities, but also discover potential zero-day vulnerabilities and abnormal behaviors through predictive analysis.

[0003] The prior art has the following deficiencies:

[0004] If an AI model is overfitted to specific threat patterns in the training data, it may not respond to a completely new form of attack, such as a new malware that uses innovative encrypted communication methods. Attackers take advantage of AI's bias towards specific historical features and design camouflage methods to evade detection, making it impossible for the model to recognize the abnormality of this threat. At the same time, if the AI ​​model lacks generalization capabilities and cannot accurately identify unknown new threat features, attackers can easily exploit zero-day vulnerabilities or advanced persistent threats (APTs) to break into the system without AI being able to detect it, forming a blind spot. This blind spot can lead to long-term exposure of critical infrastructure, allowing attackers to steal sensitive data unhindered, disrupt system functions, and even implant backdoors, further expanding the attack surface. Summary of the invention

[0005] The purpose of the present invention is to provide a vulnerability and threat scanning method and system based on artificial intelligence to address the deficiencies in the background technology.

[0006] In order to achieve the above object, the present invention provides the following technical solution: a vulnerability and threat scanning method based on artificial intelligence, comprising the following steps:

[0007] S1: Extract training data of known threats and potential zero-day attacks from the real environment, and use the extracted training data to build and train the AI ​​model;

[0008] S2: Based on the trained AI model, generate simulated attack samples with different complexity and abnormality, wherein the simulated attack samples include high-complexity and low-abnormality samples and low-complexity and high-abnormality samples;

[0009] S3: After preprocessing, the simulated attack samples are directly brought into the trained AI model as input data to simulate the threat detection process in the actual operating environment and analyze the AI ​​model's response capability to highly complex attacks during the simulation.

[0010] S4: When the AI ​​model has a low response capability to high-complexity attacks, the output results of the AI ​​model are recorded, the classification results of highly abnormal scoring behaviors are compared, the false positives and false negatives of the AI ​​model are counted, and the abnormality of the output results of the AI ​​model is determined;

[0011] S5: Comprehensively analyze the AI ​​model’s ability to respond to highly complex attacks and the degree of abnormality in the AI ​​model’s output results, and evaluate the accuracy of the AI ​​model in identifying unknown new threat features;

[0012] S6: When the AI ​​model shows inaccuracy in identifying unknown new threat features, the abnormal degree of accuracy of the AI ​​model in identifying unknown new threat features within a fixed time period is predicted. If the abnormal degree is high, a warning signal is generated and the AI ​​model is optimized.

[0013] Preferably, in S3, the sample response time deviation index is generated after comparing the time difference between the model in processing high-complexity and low-anomaly samples and low-complexity and high-anomaly samples. The method for obtaining the sample response time deviation index is:

[0014] Constructing response time series for high-complexity and low-anomaly samples ; Response time series of low-complexity and high-anomaly samples ; Define the distance function , expressed as Euclidean distance: ;Construct an m×n distance matrix D, whose elements are: ; Calculate the cumulative cost matrix. Define the cumulative cost matrix C, which represents the minimum matching cost from the starting point to the current position: Initial condition: C(1,1)=D(1,1); Boundary condition: ; ; Recursive relation: ; Finally, C(m,n) is the sequence arrive The minimum matching cost; backtrack to find the best path P, start backtracking from (m,n), and select the path according to the minimum cumulative cost: ; The length of the optimal path is L, which represents the number of alignment steps of the two sequences. The sample response time deviation index is defined as the average cumulative cost on the path: ; In the formula, C(m,n) is the sequence arrive The cumulative matching cost, QS is the sample response time deviation index, and L is the length of the optimal path.

[0015] Preferably, in S4, the correlation between the anomaly score and the false negative rate is analyzed to generate an abnormal false negative correlation index to judge the abnormal degree of the output result of the AI ​​model. The method for obtaining the abnormal false negative correlation index is:

[0016] Constructing anomaly score sequences Represents the abnormal score of the zth sample, missing label sequence Indicates underreporting. Indicates no missed reports; discretize the anomaly score A into multiple intervals, and the missed report label L is already a binary classification with a value of {0,1}; for each , statistics A in the interval and The number of samples; calculate the joint probability ; where n is the total number of samples; the marginal probability of anomaly score: ; Marginal probability of missing a label: ; According to the joint probability and marginal probability, calculate the abnormal underreporting correlation index, the expression is: ; AS is the abnormal underreporting correlation index.

[0017] Preferably, in S5, the responsiveness of the AI ​​model to highly complex attacks and the degree of abnormality of the AI ​​model output results are comprehensively analyzed to evaluate the accuracy of the AI ​​model in identifying unknown new threat features, specifically:

[0018] The sample response time deviation index and the anomaly omission correlation index are normalized, and the accuracy coefficient of the AI ​​model in identifying unknown new threat characteristics is calculated based on the normalized sample response time deviation index and the anomaly omission correlation index.

[0019] Preferably, in S5, the acquired accuracy coefficient of the AI ​​model for identifying unknown new threat features is compared with the accuracy threshold under normal conditions preset according to historical data. If the accuracy coefficient of the AI ​​model for identifying unknown new threat features is greater than or equal to the accuracy threshold, it means that the AI ​​model has high accuracy in identifying unknown new threat features. In this case, an accurate recognition signal is generated, and the recognition result of the AI ​​model is classified as an accurate recognition result. If the accuracy coefficient of the AI ​​model for identifying unknown new threat features is less than the accuracy threshold, it means that the AI ​​model has low accuracy in identifying unknown new threat features. In this case, an inaccurate recognition signal is generated, and the recognition result of the AI ​​model is classified as an accurate recognition result.

[0020] Preferably, in S6, when the AI ​​model shows inaccuracy in identifying unknown new threat features, that is, the accuracy coefficient of the AI ​​model generating within a fixed time period for identifying unknown new threat features is less than the accuracy threshold, the accuracy coefficients less than the accuracy threshold generated within a subsequent fixed time period are collected, and a corresponding data set is established, and the mean and standard deviation of the data set are calculated. After analyzing the data set, the abnormal degree of the accuracy of the AI ​​model identifying unknown new threat features within the fixed time period is predicted based on the analysis results. If the abnormal degree is high, a warning signal is generated and the AI ​​model is optimized.

[0021] Preferably, if the mean of the accuracy coefficients in the data set is greater than or equal to the reference threshold of the mean of the accuracy coefficients, and the standard deviation of the accuracy coefficients is less than the reference threshold of the standard deviation of the accuracy coefficients, then no warning signal is generated, the model performance is normal, and no optimization is required;

[0022] If the mean value of the accuracy coefficient is greater than or equal to the reference threshold value of the mean value of the accuracy coefficient, and the standard deviation of the accuracy coefficient is greater than or equal to the reference threshold value of the standard deviation of the accuracy coefficient, a third-level warning signal is generated, indicating that the model performance is not stable enough, and the classification threshold of the model is adjusted;

[0023] If the mean value of the accuracy coefficient is less than the reference threshold value of the mean value of the accuracy coefficient, and the standard deviation of the accuracy coefficient is greater than or equal to the reference threshold value of the standard deviation of the accuracy coefficient, a first-level warning signal is generated, indicating that the model is in a state of performance degradation and requires urgent optimization;

[0024] If the mean accuracy coefficient is less than the reference threshold of the mean accuracy coefficient, and the standard deviation of the accuracy coefficient is less than the reference threshold of the standard deviation of the accuracy coefficient, a secondary warning signal is generated, and the model extraction method and the coverage of the training data need to be re-evaluated for comprehensive optimization.

[0025] The present invention also provides an artificial intelligence-based vulnerability and threat scanning system, comprising a data extraction module, a simulated attack generation module, a model test analysis module, an error analysis module, an accuracy assessment module, and an optimization module;

[0026] Data extraction module: extracts training data of known threats and potential zero-day attacks from the real environment, and uses the extracted training data to build and train the AI ​​model;

[0027] Simulated attack generation module: based on the trained AI model, generates simulated attack samples with different complexity and abnormality, including high-complexity and low-abnormality samples and low-complexity and high-abnormality samples;

[0028] Model testing and analysis module: After preprocessing, simulated attack samples are directly brought into the trained AI model as input data to simulate the threat detection process in the actual operating environment and analyze the AI ​​model's response capability to highly complex attacks during the simulation process;

[0029] Error analysis module: When the AI ​​model has a low response capability to highly complex attacks, the output results of the AI ​​model are recorded, the classification results of highly abnormal scoring behaviors are compared, the false positives and false negatives of the AI ​​model are counted, and the degree of abnormality of the output results of the AI ​​model is determined;

[0030] Accuracy assessment module: Comprehensively analyzes the AI ​​model's ability to respond to highly complex attacks and the degree of abnormality of the AI ​​model's output results, and evaluates the accuracy of the AI ​​model in identifying unknown new threat features;

[0031] Optimization module: When the AI ​​model shows inaccuracy in identifying unknown new threat features, the abnormal degree of accuracy of the AI ​​model in identifying unknown new threat features within a fixed time period is predicted. If the abnormal degree is high, a warning signal is generated and the AI ​​model is optimized.

[0032] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0033] 1. The present invention constructs an AI model by extracting known threats and potential zero-day attack data from the real environment, and combines the generation of high-complexity and low-abnormality and low-complexity and high-abnormality simulated attack samples to test the performance of the model under different complexity and abnormality conditions, deeply analyze the false positives and false negatives, and quantify the model's responsiveness and degree of abnormality. By introducing the sample response time deviation index and the abnormal false negative correlation index, the accuracy of the model in identifying unknown new threat features is comprehensively evaluated, and compared with the historical accuracy threshold to generate a model performance signal. When the performance is not up to standard, the degree of abnormality of the model is calculated based on the mean and standard deviation of the accuracy coefficient for a fixed time period, and different levels of early warning signals are generated and targeted optimization is performed to improve the model's detection capability and robustness for unknown threats.

[0034] 2. The present invention solves the problems of overfitting and lack of generalization ability of AI models in the prior art, effectively responds to the challenge of attackers evading detection through disguise and innovative means, and reduces the risk of zero-day vulnerabilities and advanced persistent threats to critical infrastructure. Through dynamic monitoring and graded early warning mechanisms, the present invention significantly improves the adaptability and response efficiency to new threats while ensuring the model detection performance, providing the system with comprehensive, accurate and dynamic security protection capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0036] Figure 1 The figure is a flow chart of the method of the present invention.

[0037] Figure 2 It is a system module diagram of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Example 1, please refer to Figure 1 and Figure 2 As shown, the vulnerability and threat scanning method based on artificial intelligence described in this embodiment includes the following steps:

[0040] S1: Extract training data of known threats and potential zero-day attacks from the real environment, and use the extracted training data to build and train the AI ​​model;

[0041] S2: Based on the trained AI model, generate simulated attack samples with different complexity and abnormality, wherein the simulated attack samples include high-complexity and low-abnormality samples and low-complexity and high-abnormality samples;

[0042] S3: After preprocessing, the simulated attack samples are directly brought into the trained AI model as input data to simulate the threat detection process in the actual operating environment and analyze the AI ​​model's response capability to highly complex attacks during the simulation.

[0043] S4: When the AI ​​model has a low response capability to high-complexity attacks, the output results of the AI ​​model are recorded, the classification results of highly abnormal scoring behaviors are compared, the false positives and false negatives of the AI ​​model are counted, and the abnormality of the output results of the AI ​​model is determined;

[0044] S5: Comprehensively analyze the AI ​​model’s ability to respond to highly complex attacks and the degree of abnormality in the AI ​​model’s output results, and evaluate the accuracy of the AI ​​model in identifying unknown new threat features;

[0045] S6: When the AI ​​model shows inaccuracy in identifying unknown new threat features, the abnormal degree of accuracy of the AI ​​model in identifying unknown new threat features within a fixed time period is predicted. If the abnormal degree is high, a warning signal is generated and the AI ​​model is optimized.

[0046] In S1, training data of known threats and potential zero-day attacks are extracted from the real environment, and the AI ​​model is built using the extracted training data to train the AI ​​model. Specifically:

[0047] The real environment refers to the dynamic data associated with the actual use scenario generated by the actual network system, application, device log and communication traffic. These data contain real user behavior, known attack patterns and potential unknown threat characteristics. When collecting training data, it is necessary to cover a variety of data sources to ensure that the model can adapt to different threat scenarios: Known threat data: network logs (such as traffic capture PCAP files). Security event management system (SIEM) records. Threat intelligence library (such as attack techniques recorded in CVE, MITRE ATT&CK framework). Malware sample library (such as VirusTotal, MalwareBazaar). Potential zero-day attack data: unclassified network traffic and logs. Highly abnormal behavior patterns, such as unconventional port access and traffic surges. Unknown behavior data collected from honeypots and sandbox environments. Network data: such as packet traffic, HTTP request logs, DNS records. System data: such as file change records, user login behavior. Time series data: captures patterns of behavior changes over time.

[0048] Use dedicated tools and platforms to collect data: Traffic monitoring tools: Wireshark, Zeek.

[0049] Log collection tools: ELK Stack, Splunk. Honeypot systems: Cowrie, Dionaea, used to capture unknown threats. Eliminate irrelevant or duplicate data (such as background noise, regular traffic). Fill or delete missing fields in logs. For encrypted traffic, decrypt it as much as possible to extract threat features.

[0050] Known threat data: Based on the classification of the threat intelligence library, samples are marked as specific threat types (such as DDoS, SQL injection). Zero-day attack data: Abnormal behavior data is labeled as "unknown threat" or further subdivided through cluster analysis.

[0051] Convert raw data into a format that AI models can process: Convert network data into feature vectors (such as packet size, protocol type) Convert system event logs into time series matrices.

[0052] Extract key features that help in threat classification: Network level: such as traffic direction (inbound / outbound), packet size distribution, protocol usage. User behavior: such as login frequency, geographic location changes. System level: such as file access patterns, CPU and memory usage mutations.

[0053] Use tools (such as PCA and Lasso regression) to filter out the features most relevant to threat detection and reduce the dimensions. Balance feature weights to avoid overfitting caused by feature bias. Build more advanced combined features. For example, summarize "port scanning behavior" as a feature sequence of continuous port connections.

[0054] Choose the appropriate AI algorithm based on the threat detection task: Supervised learning (known threat detection): decision tree, random forest, support vector machine. Unsupervised learning (zero-day attack detection): K-Means clustering, isolation forest. Deep learning (complex threat detection): LSTM (Long Short-Term Memory Network) captures time series threat patterns; CNN (Convolutional Neural Network) analyzes traffic characteristics.

[0055] Training set and test set division: divide the data into training set and test set in a ratio of 8:2 to ensure that the test set contains unknown samples. Model parameter tuning: use grid search or Bayesian optimization to adjust model hyperparameters (such as tree depth, learning rate). Cross-validation: use K-fold cross-validation to evaluate model performance and avoid overfitting. Evaluation indicators: Accuracy: the ratio of all samples correctly classified. Recall rate: the proportion of real threats detected by the model, especially focusing on high-risk attacks. F1 score: a comprehensive measure of the balance between accuracy and recall.

[0056] Deploy the trained model to the production environment to analyze and detect threats in real time. Integrated system: Combine with SIEM, IDS / IPS and other security devices. Real-time response: Configure automated threat response strategies. Continuously monitor the performance of the model in the real environment and record false positives and false negatives. Collect new threat data in real time and add it to the training set for model iteration. Regularly update the threat intelligence library (such as CVE numbers, new APT attack techniques) to ensure the timeliness of the model knowledge base. Use GANs to generate more complex simulated attack samples to improve the model's detection capabilities for zero-day attacks.

[0057] S2: Based on the trained AI model, simulated attack samples with different complexity and abnormality are generated, wherein the simulated attack samples include high-complexity and low-abnormality samples and low-complexity and high-abnormality samples.

[0058] High complexity and low anomaly samples: The samples are highly disguised and obfuscated, imitating legitimate behaviors. The behavioral characteristics are very close to normal traffic, with only some minor differences. Example scenario: Malicious data is transmitted using encrypted traffic (such as HTTPS). Malicious instructions are embedded in data packets disguised as legitimate services (such as DNS queries).

[0059] Low complexity and high anomaly samples: The behavior or characteristics of the samples are simple and clear, easy to understand. They are significantly different from normal behavior patterns. Example scenarios: Direct attempts to perform a large number of port scans in a short period of time. Unauthorized high-frequency access to external unknown IPs.

[0060] Extract samples from real data and generate simulated attack samples that meet the complexity and abnormality requirements through specific operations: High-complexity and low-abnormal sample generation: Modify subtle parameters in normal traffic, such as adding encryption layers and disguising as legitimate services. Simulate cross-domain access or use legitimate tools (such as PowerShell) to conduct lurking activities. Low-complexity and high-abnormal sample generation: Simply change behavioral characteristics, such as continuously trying to access high-privilege paths and sending an abnormally large number of requests in a short period of time.

[0061] Generate samples using attack simulation tools or scripts: Tools: Metasploit Framework: Generates diverse attack behaviors. Scapy: Customizes data packets to simulate abnormal traffic. Hping3: Generates specific network layer attack traffic.

[0062] Rule construction: High complexity and low anomaly rules: simulate normal HTTPS sessions and insert disguised malicious data. Use legitimate protocols (such as SMB) to deliver malicious payloads. Low complexity and high anomaly rules: continuously send invalid HTTP requests. Simulate high-frequency access behavior of atypical ports.

[0063] Use GANs to generate more complex samples: High-complexity and low-abnormal samples: The GAN model generates traffic that is almost the same as normal traffic, but with hidden attack features. Low-complexity and high-abnormal samples: Generate obviously abnormal attack behaviors by adding noise or random features. By clustering or detecting anomalies on unlabeled data, marking potential threat features, and then adjusting their complexity and abnormality to generate corresponding attack samples.

[0064] S3: After preprocessing, the simulated attack samples are directly brought into the trained AI model as input data to simulate the threat detection process in the actual operating environment and analyze the AI ​​model's response capability to high-complexity attacks during the simulation.

[0065] Convert attack samples (such as network traffic and logs) into feature vectors that can be recognized by the model. Extract features such as packet size, protocol type, IP address distribution, etc. from network traffic samples. Extract features such as user behavior time series and abnormal event frequency from system logs. For time series models (such as LSTM), adjust the sample length and fill in missing data. Normalize sample features to ensure that the feature value range is consistent.

[0066] High-complexity and low-anomaly samples: marked as "disguised attack". Low-complexity and high-anomaly samples: marked as "obvious attack". Keep the true label (ground truth) for subsequent comparative analysis. Clean the redundant or meaningless information in the sample to avoid interference with the model analysis results.

[0067] Configure the network traffic and log data distribution consistent with the production environment to build a near-real operating environment. Mix normal samples and simulated attack samples to ensure that the detection process is realistic: the proportion of high-complexity and low-abnormal samples is appropriately increased to simulate the potential impact of zero-day attacks. The proportion of low-complexity and high-abnormal samples is appropriately reduced to simulate rare but obvious threats.

[0068] Input the preprocessed simulated attack samples into the AI ​​model through data streams to simulate the real-time processing of traffic or logs in the production environment. Example: Input traffic data in chronological order to simulate continuous attacks. Test the performance of the model under high-load scenarios by batch inputting highly complex attack samples.

[0069] The sample response time deviation index is generated by comparing the time difference between the model in processing high-complexity and low-anomaly samples and low-complexity and high-anomaly samples. The method for obtaining the sample response time deviation index is as follows:

[0070] Constructing response time series for high-complexity and low-anomaly samples ; Response time series of low-complexity and high-anomaly samples ; Define the distance function , expressed as Euclidean distance: ;Construct an m×n distance matrix D, whose elements are: ; Calculate the cumulative cost matrix. Define the cumulative cost matrix C, which represents the minimum matching cost from the starting point to the current position: Initial condition: C(1,1)=D(1,1); Boundary condition: ; ; Recursive relation: ; Finally, C(m,n) is the sequence arrive The minimum matching cost; backtrack to find the best path P, start backtracking from (m,n), and select the path according to the minimum cumulative cost: ; The length of the optimal path is L, which represents the number of alignment steps of the two sequences. The sample response time deviation index is defined as the average cumulative cost on the path: ; In the formula, C(m,n) is the sequence arrive The cumulative matching cost, QS is the sample response time deviation index, and L is the length of the optimal path.

[0071] If the sample response time deviation index is larger, it means that there is a significant difference between the response time of the AI ​​model to high-complexity attacks and low-complexity attacks. The AI ​​model has a high response capability to high-complexity attacks, showing instability in response efficiency. Specifically, this may mean that the model takes longer to complete feature extraction and threat classification when facing high-complexity attacks, indicating that the model's processing capability for complex attacks is insufficient or there is a bottleneck in algorithm optimization. An excessively large sample response time deviation index may result in high-complexity attacks not being detected in a timely manner, thereby increasing the risk of the system being exploited.

[0072] On the contrary, if the sample response time deviation index is smaller, it means that the difference in response time between the AI ​​model and the high-complexity attack and the low-complexity attack is smaller, and the AI ​​model has a low response ability to high-complexity attacks, showing strong consistency and robustness. This shows that the model can efficiently complete threat identification when facing complex attacks, and the feature extraction and classification mechanism is relatively complete. However, a too small sample response time deviation index may also mean that the model has limited ability to distinguish complexity and abnormality features, resulting in complex attack samples not being fully identified, thereby potentially increasing the risk of underreporting.

[0073] S4: When the AI ​​model has a low response capability to high-complexity attacks, record the output results of the AI ​​model, compare the classification results of highly abnormal scoring behaviors, count the false positives and false negatives of the AI ​​model, and determine the degree of abnormality of the AI ​​model output results.

[0074] Record the AI ​​model's predicted label (normal / malicious) and its confidence for each input sample. Ground Truth for high-complexity attacks: labeled "malicious". Record the model output label (normal / malicious). Keep the anomaly score for each sample (such as the probability value or abnormality score in the model prediction). High anomaly score: indicates that the model has identified a significant deviation from normal behavior. Low anomaly score: may mean that the model failed to effectively identify sample anomalies. For each high-complexity sample, record the model's response time from input to output classification results.

[0075] The key features of the sample are retained to facilitate subsequent source tracing analysis (such as the protocol type of network traffic, packet size, etc.).

[0076] Accuracy: the rate at which the model correctly classifies high-complexity samples as "malicious". False Negative Rate (FNR): false negatives mean that the model fails to identify real high-complexity threats. False Positive Rate (FPR): false positives mean that the model mistakenly labels normal samples as malicious.

[0077] The classification results of behaviors with high anomaly scores are extracted separately and analyzed for consistency with the true labels: High anomaly score + classification as malicious: indicates that the model can correctly identify such threats. High anomaly score + classification as normal: may be a misjudgment of the model in a complex attack mode. Low anomaly score + classification as malicious: may reflect that the model is too sensitive to certain features.

[0078] Compare the distribution of anomaly scores between high-complexity attack samples and low-complexity attack samples: Analyze whether high-complexity attacks receive lower anomaly scores overall. Check whether there is any consistency between anomaly scores and classification results.

[0079] Collect statistics on the key features of false positive and false negative samples (such as protocol type, time features, etc.). Analyze whether there are specific features or patterns that lead to the concentration of false positives and false negatives. Analyze the time when false positives and false negatives occur to determine whether they are related to specific time periods or load conditions.

[0080] After analyzing the correlation between the anomaly score and the false negative rate, an abnormal false negative correlation index is generated to determine the degree of abnormality of the AI ​​model output results. The method for obtaining the abnormal false negative correlation index is as follows:

[0081] Constructing anomaly scoring sequences Represents the abnormal score of the zth sample, missing label sequence Indicates underreporting. Indicates no missed reports; discretize the anomaly score A into multiple intervals, and the missed report label L is already a binary classification with a value of {0,1}; for each , statistics A in the interval and The number of samples; calculate the joint probability ; where n is the total number of samples; the marginal probability of anomaly score: ; Marginal probability of missing a label: ; According to the joint probability and marginal probability, calculate the abnormal underreporting correlation index, the expression is: ; AS is the abnormal underreporting correlation index.

[0082] If the anomaly false negative correlation index is larger, it means that the correlation between the anomaly score and the false negative rate is stronger, that is, the anomaly score of the AI ​​model can well explain the false negative situation. This result may indicate that the model has a systematic problem in detecting high-complexity attacks: when the anomaly score is higher, the false negative rate may also be higher, reflecting that the model's classification rules for high-complexity threats fail to fully utilize the information in the anomaly score. A larger index indicates that the model output results are highly abnormal, and it is necessary to optimize the anomaly scoring mechanism or classification strategy to reduce the risk of systematic misjudgment.

[0083] On the contrary, if the anomaly underreporting correlation index is smaller, it means that the correlation between the anomaly score and the underreporting rate is weaker, that is, the anomaly score of the AI ​​model has a poor ability to explain the underreporting situation. This situation may indicate that the model does not make sufficient use of the anomaly score and fails to effectively map the abnormal characteristics to the threat level, resulting in random underreporting of abnormal samples during classification. A smaller index indicates that the degree of abnormality of the model output is low, but it may also reveal that the model's scoring mechanism needs to enhance its ability to express threat characteristics.

[0084] S5: Comprehensively analyze the AI ​​model’s ability to respond to highly complex attacks and the degree of abnormality in the AI ​​model’s output results to evaluate the accuracy of the AI ​​model in identifying unknown new threat characteristics.

[0085] The sample response time deviation index and the anomaly omission correlation index are normalized, and the accuracy coefficient of the AI ​​model in identifying unknown new threat characteristics is calculated based on the normalized sample response time deviation index and the anomaly omission correlation index.

[0086] For example, the present invention can use the following formula to calculate the accuracy coefficient of the AI ​​model in identifying unknown new threat features, and the calculation expression is: ; Where LK is the accuracy coefficient of the AI ​​model in identifying unknown new threat features, QS is the sample response time deviation index, and AS is the abnormal underreporting correlation index. is the proportional coefficient of the sample response time deviation index and the abnormal underreporting correlation index, and .

[0087] The obtained accuracy coefficient of the AI ​​model for identifying unknown new threat characteristics is compared with the accuracy threshold under normal conditions preset according to historical data. If the accuracy coefficient of the AI ​​model for identifying unknown new threat characteristics is greater than or equal to the accuracy threshold, it means that the AI ​​model has high accuracy in identifying unknown new threat characteristics. At this time, an accurate recognition signal is generated, and the recognition result of the AI ​​model is classified as an accurate recognition result; if the accuracy coefficient of the AI ​​model for identifying unknown new threat characteristics is less than the accuracy threshold, it means that the AI ​​model has low accuracy in identifying unknown new threat characteristics. At this time, an inaccurate recognition signal is generated, and the recognition result of the AI ​​model is classified as an accurate recognition result.

[0088] S6: When the AI ​​model shows inaccuracy in identifying unknown new threat features, the abnormal degree of accuracy of the AI ​​model in identifying unknown new threat features within a fixed time period is predicted. If the abnormal degree is high, a warning signal is generated and the AI ​​model is optimized.

[0089] When the AI ​​model shows inaccuracy in identifying unknown new threat features, that is, the accuracy coefficient of the AI ​​model generating within a fixed time period for identifying unknown new threat features is less than the accuracy threshold, the accuracy coefficients less than the accuracy threshold generated within subsequent fixed time periods are collected, and a corresponding data set is established. The mean and standard deviation of the data set are calculated and analyzed. The degree of abnormality in the accuracy of the AI ​​model identifying unknown new threat features within the fixed time period is predicted based on the analysis results. If the degree of abnormality is high, a warning signal is generated and the AI ​​model is optimized.

[0090] If the mean of the accuracy coefficient in the data set is greater than or equal to the reference threshold of the mean of the accuracy coefficient, and the standard deviation of the accuracy coefficient is less than the reference threshold of the standard deviation of the accuracy coefficient, a high mean indicates good overall recognition performance; a small standard deviation indicates that the accuracy distribution is stable and has no obvious fluctuations. At this time, no warning signal is generated, the model performance is normal, and no optimization is required. It is suitable to maintain the current configuration and continue monitoring.

[0091] If the mean value of the accuracy coefficient is greater than or equal to the reference threshold value of the mean value of the accuracy coefficient, and the standard deviation of the accuracy coefficient is greater than or equal to the reference threshold value of the standard deviation of the accuracy coefficient, a high mean value indicates that the overall performance is basically up to standard. A large standard deviation indicates that the model performance fluctuates significantly and may perform poorly in some cases. At this time, a third-level warning signal is generated, and the model performance stability is insufficient, and the recognition ability in specific scenarios needs to be optimized. The characteristics of highly volatile data points can be analyzed to adjust the classification threshold of the model or enhance sample training for specific threat scenarios.

[0092] If the mean of the accuracy coefficient is less than the reference threshold of the mean of the accuracy coefficient, and the standard deviation of the accuracy coefficient is greater than or equal to the reference threshold of the standard deviation of the accuracy coefficient, the low mean indicates poor overall performance. A large standard deviation indicates that the performance distribution is chaotic and that recognition fails seriously in certain time periods or specific scenarios. At this time, a first-level warning signal is generated, and the model is in a state of performance degradation and needs to be urgently optimized. The cause should be found from the time period or sample characteristics with more false positives and false negatives, and the model should be redesigned or new training data should be introduced.

[0093] If the mean of the accuracy coefficient is less than the reference threshold of the mean of the accuracy coefficient, and the standard deviation of the accuracy coefficient is less than the reference threshold of the standard deviation of the accuracy coefficient, the low mean indicates that the overall performance of the model does not meet the standard. A small standard deviation indicates that the distribution of model performance is stable but generally low. At this time, a secondary warning signal is generated, and the model has systemic defects and may lack the ability to detect specific threat features. The model's feature extraction method and the coverage of training data need to be re-evaluated and comprehensively optimized.

[0094] It should be noted here that the importance of the first-level warning signal is greater than that of the second-level warning signal, and the importance of the second-level warning signal is greater than that of the third-level warning signal. Relevant personnel can take corresponding handling measures according to different warning signal levels.

[0095] In this embodiment, by extracting known threats and potential zero-day attack data from the real environment, building and training the AI ​​model, simulated attack samples of different complexity and abnormality (such as high-complexity low-abnormality and low-complexity high-abnormality samples) are generated, and these samples are input into the trained AI model to simulate the actual operating environment for threat detection, and analyze the model's response to high-complexity attacks. When the model's response ability is low, its output results are recorded, the classification of high-abnormal scoring behaviors is compared, the false positives and missed negatives are counted, and the degree of abnormality of the model's output results is judged. Based on the comprehensive analysis of the model's response ability and output results, its accuracy in identifying unknown new threat features is evaluated. If the model's accuracy abnormality is high within a fixed time period, a predictive analysis is performed to generate an early warning signal, and its performance is improved by optimizing the model to continuously improve the detection capability of unknown new threats.

[0096] Embodiment 2, an artificial intelligence-based vulnerability and threat scanning system described in this embodiment includes a data extraction module, a simulated attack generation module, a model test analysis module, an error analysis module, an accuracy assessment module and an optimization module;

[0097] Data extraction module: extracts training data of known threats and potential zero-day attacks from the real environment, and uses the extracted training data to build and train the AI ​​model;

[0098] Simulated attack generation module: based on the trained AI model, generates simulated attack samples with different complexity and abnormality, including high-complexity and low-abnormality samples and low-complexity and high-abnormality samples;

[0099] Model testing and analysis module: After preprocessing, simulated attack samples are directly brought into the trained AI model as input data to simulate the threat detection process in the actual operating environment and analyze the AI ​​model's response capability to highly complex attacks during the simulation process;

[0100] Error analysis module: When the AI ​​model has a low response capability to highly complex attacks, the output results of the AI ​​model are recorded, the classification results of highly abnormal scoring behaviors are compared, the false positives and false negatives of the AI ​​model are counted, and the degree of abnormality of the output results of the AI ​​model is determined;

[0101] Accuracy assessment module: Comprehensively analyzes the AI ​​model's ability to respond to highly complex attacks and the degree of abnormality of the AI ​​model's output results, and evaluates the accuracy of the AI ​​model in identifying unknown new threat features;

[0102] Optimization module: When the AI ​​model shows inaccuracy in identifying unknown new threat features, the abnormal degree of accuracy of the AI ​​model in identifying unknown new threat features within a fixed time period is predicted. If the abnormal degree is high, a warning signal is generated and the AI ​​model is optimized.

[0103] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0104] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0105] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0106] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A vulnerability and threat scanning method based on artificial intelligence, characterized in that: The following steps are involved: S1: Extract training data of known threats and potential zero-day attacks from the real environment, and use the extracted training data to build and train the AI ​​model; S2: Based on the trained AI model, generate simulated attack samples with different complexity and abnormality, wherein the simulated attack samples include high-complexity and low-abnormality samples and low-complexity and high-abnormality samples; S3: After preprocessing, the simulated attack samples are directly brought into the trained AI model as input data to simulate the threat detection process in the actual operating environment and analyze the AI ​​model's response capability to highly complex attacks during the simulation. The step S3 specifically includes: generating a sample response time deviation index after comparing the time difference between the model in processing high-complexity and low-anomaly samples and low-complexity and high-anomaly samples. The method for obtaining the sample response time deviation index is: Constructing response time series for high-complexity and low-anomaly samples ; Response time series of low-complexity and high-anomaly samples ; Define the distance function , expressed as Euclidean distance: ;Construct an m×n distance matrix D, whose elements are: ; Calculate the cumulative cost matrix. Define the cumulative cost matrix C, which represents the minimum matching cost from the starting point to the current position: Initial condition: C(1,1)=D(1,1); Boundary condition: ; ; Recursive relation: ; Finally, C(m,n) is the sequence arrive The minimum matching cost; backtrack to find the best path P, start backtracking from (m,n), and select the path according to the minimum cumulative cost: , ; The length of the optimal path is L, which represents the number of alignment steps of the two sequences. The sample response time deviation index is defined as the average cumulative cost on the path: ; In the formula, C(m,n) is the sequence arrive The cumulative matching cost of is the sample response time deviation index, L is the length of the optimal path; S4: When the AI ​​model has a low response capability to high-complexity attacks, the output results of the AI ​​model are recorded, the classification results of highly abnormal scoring behaviors are compared, the false positives and false negatives of the AI ​​model are counted, and the abnormality of the output results of the AI ​​model is determined; The step S4 specifically includes: analyzing the correlation between the abnormality score and the false negative rate to generate an abnormal false negative correlation index, judging the abnormality degree of the output result of the AI ​​model, and obtaining the abnormal false negative correlation index by: Constructing anomaly scoring sequences , Represents the abnormal score of the zth sample, missing label sequence , =1 means missing report, =0 means no missed report; discretize the anomaly score A into multiple intervals, and the missed report label L is already a binary classification with a value of {0,1}; for each , statistics A in the interval and The number of samples; calculate the joint probability : ; where z is the total number of samples; the marginal probability of anomaly score: ; Marginal probability of missing a label: ; According to the joint probability and marginal probability, calculate the abnormal underreporting correlation index, the expression is: ; is the abnormal underreporting related index; S5: Comprehensively analyze the AI ​​model’s ability to respond to highly complex attacks and the degree of abnormality in the AI ​​model’s output results, and evaluate the accuracy of the AI ​​model in identifying unknown new threat features; S6: When the AI ​​model shows inaccuracy in identifying unknown new threat features, the abnormal degree of accuracy of the AI ​​model in identifying unknown new threat features within a fixed time period is predicted. If the abnormal degree is high, a warning signal is generated and the AI ​​model is optimized.

2. The vulnerability and threat scanning method based on artificial intelligence according to claim 1, characterized in that: In S5, the AI ​​model’s ability to respond to highly complex attacks and the degree of abnormality of the AI ​​model’s output results are comprehensively analyzed to evaluate the accuracy of the AI ​​model in identifying unknown new threat features. Specifically: The sample response time deviation index and the anomaly omission correlation index are normalized, and the accuracy coefficient of the AI ​​model in identifying unknown new threat characteristics is calculated based on the normalized sample response time deviation index and the anomaly omission correlation index.

3. The vulnerability and threat scanning method based on artificial intelligence according to claim 2, characterized in that: In S5, the acquired accuracy coefficient of the AI ​​model for identifying unknown new threat features is compared with the accuracy threshold under normal conditions preset according to historical data. If the accuracy coefficient of the AI ​​model for identifying unknown new threat features is greater than or equal to the accuracy threshold, it means that the AI ​​model has high accuracy in identifying the unknown new threat features. In this case, an accurate recognition signal is generated, and the recognition result of the AI ​​model is classified as an accurate recognition result. If the accuracy coefficient of the AI ​​model for identifying unknown new threat features is less than the accuracy threshold, it means that the AI ​​model has low accuracy in identifying the unknown new threat features. In this case, an inaccurate recognition signal is generated, and the recognition result of the AI ​​model is classified as an accurate recognition result.

4. The vulnerability and threat scanning method based on artificial intelligence according to claim 1, characterized in that: In S6, when the AI ​​model shows inaccuracy in identifying unknown new threat features, that is, the accuracy coefficient of the AI ​​model generating within a fixed time period for identifying unknown new threat features is less than the accuracy threshold, the accuracy coefficients less than the accuracy threshold generated within subsequent fixed time periods are collected, and a corresponding data set is established. The mean and standard deviation of the data set are calculated and analyzed. The abnormal degree of the accuracy of the AI ​​model identifying unknown new threat features within the fixed time period is predicted based on the analysis results. If the abnormal degree is high, a warning signal is generated and the AI ​​model is optimized.

5. The vulnerability and threat scanning method based on artificial intelligence according to claim 4, characterized in that: If the mean accuracy coefficient in the data set is greater than or equal to the reference threshold of the mean accuracy coefficient, and the standard deviation of the accuracy coefficient is less than the reference threshold of the standard deviation of the accuracy coefficient, no warning signal is generated, the model performance is normal, and no optimization is required; If the mean value of the accuracy coefficient is greater than or equal to the reference threshold value of the mean value of the accuracy coefficient, and the standard deviation of the accuracy coefficient is greater than or equal to the reference threshold value of the standard deviation of the accuracy coefficient, a third-level warning signal is generated, indicating that the model performance is not stable enough, and the classification threshold of the model is adjusted; If the mean value of the accuracy coefficient is less than the reference threshold value of the mean value of the accuracy coefficient, and the standard deviation of the accuracy coefficient is greater than or equal to the reference threshold value of the standard deviation of the accuracy coefficient, a first-level warning signal is generated, indicating that the model is in a state of performance degradation and requires urgent optimization; If the mean accuracy coefficient is less than the reference threshold of the mean accuracy coefficient, and the standard deviation of the accuracy coefficient is less than the reference threshold of the standard deviation of the accuracy coefficient, a secondary warning signal is generated, and the model extraction method and the coverage of the training data need to be re-evaluated for comprehensive optimization.

6. An artificial intelligence-based vulnerability and threat scanning system, used to implement an artificial intelligence-based vulnerability and threat scanning method according to any one of claims 1 to 5, characterized in that: It includes data extraction module, simulated attack generation module, model test analysis module, error analysis module, accuracy assessment module and optimization module; Data extraction module: extracts training data of known threats and potential zero-day attacks from the real environment, and uses the extracted training data to build and train the AI ​​model; Simulated attack generation module: based on the trained AI model, generates simulated attack samples with different complexity and abnormality, including high-complexity and low-abnormality samples and low-complexity and high-abnormality samples; Model testing and analysis module: After preprocessing, simulated attack samples are directly brought into the trained AI model as input data to simulate the threat detection process in the actual operating environment and analyze the AI ​​model's response capability to highly complex attacks during the simulation process; Error analysis module: When the AI ​​model has a low response capability to highly complex attacks, the output results of the AI ​​model are recorded, the classification results of highly abnormal scoring behaviors are compared, the false positives and false negatives of the AI ​​model are counted, and the degree of abnormality of the output results of the AI ​​model is determined; Accuracy assessment module: Comprehensively analyzes the AI ​​model's ability to respond to highly complex attacks and the degree of abnormality of the AI ​​model's output results, and evaluates the accuracy of the AI ​​model in identifying unknown new threat features; Optimization module: When the AI ​​model shows inaccuracy in identifying unknown new threat features, the abnormal degree of accuracy of the AI ​​model in identifying unknown new threat features within a fixed time period is predicted. If the abnormal degree is high, a warning signal is generated and the AI ​​model is optimized.

Images