Intelligent agent-based security vulnerability detection method and apparatus, and electronic device

Abstract

The invention discloses a security vulnerability detection method and device based on an agent and electronic equipment, and relates to the field of artificial intelligence and the technical field of information security, and the method comprises the steps: extracting a feature vector representing a business security state from operation data and a transaction log of a business system; inputting the feature vector as state information of the service system into a reinforcement learning agent, and outputting a vulnerability detection action for the service system through the reinforcement learning agent; according to the execution result of the vulnerability detection action, judging whether the service system has security vulnerabilities in the current service scene, and generating a detection result containing vulnerability risk levels; and generating a reward value based on the detection result, and feeding back the reward value to the reinforcement learning agent to drive the reinforcement learning agent to update. The technical problem of low financial service scene security detection efficiency in the prior art is solved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and information security technology, and more specifically, to a security vulnerability detection method, apparatus, and electronic device based on intelligent agents. Background Technology

[0002] With the accelerated digital transformation of the financial industry, various business systems are becoming increasingly complex, and transaction patterns and interaction logics are proliferating, making traditional security detection methods ineffective. Efficiently and accurately identifying security vulnerabilities at the business logic level, such as unauthorized access and data tampering, is crucial for ensuring the security and stable operation of financial services.

[0003] In related technologies, security detection mainly relies on two types of technical solutions. One is rule-matching tool scanning technology, which matches and analyzes system logs, code, or traffic using predefined security rules or keywords. The other is supervised learning model detection technology, which uses labeled historical vulnerability samples to train classification models for pattern recognition. However, both of these existing technical solutions have significant drawbacks. Tool scanning methods heavily rely on expert experience and fixed rules, making it difficult to cover dynamic and complex business logic scenarios, resulting in low detection efficiency and high false negative rates. Supervised learning methods are limited by the quantity and quality of labeled samples, lacking the ability to deeply analyze the constantly evolving new attack patterns and complex scenarios in financial business, leading to insufficient detection accuracy, and the models are unable to adapt to the continuous changes in the business environment. Therefore, existing technologies for security detection in financial business scenarios are inefficient, failing to achieve rapid response and deep coverage of large-scale, highly dynamic business environments while ensuring high accuracy.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a security vulnerability detection method, apparatus, and electronic device based on intelligent agents, to at least solve the technical problem of low security detection efficiency in financial business scenarios in the prior art.

[0006] According to one aspect of the embodiments of this application, a security vulnerability detection method based on an intelligent agent is provided, comprising: extracting feature vectors representing the security status of a business system from its operational data and transaction logs; inputting the feature vectors as state information of the business system into a pre-trained reinforcement learning intelligent agent, and outputting vulnerability detection actions for the business system through the reinforcement learning intelligent agent; wherein the reinforcement learning intelligent agent is constructed based on a value function approximation network and configured with a reward feedback mechanism corresponding to the business security objective; determining whether a security vulnerability exists in the current business scenario based on the execution result of the vulnerability detection action, and generating a detection result including the vulnerability risk level; generating a reward value based on the detection result, and feeding the reward value back to the reinforcement learning intelligent agent to drive the update of the reinforcement learning intelligent agent.

[0007] Optionally, the reinforcement learning agent is trained through the following steps: initializing the parameters of the value function approximation network and creating a cache for storing training experience data; constructing a simulation environment based on historical business data of the business system; performing iterative training in the simulation environment, wherein each iteration includes: determining the exploration probability based on the exploration rate parameter, selecting the target action corresponding to the value function approximation network from multiple preset vulnerability detection actions based on the exploration probability and executing it; after executing the target action, collecting the reward value returned by the simulation environment and the latest state information of the business system, and forming experience data stored in the cache; periodically randomly sampling a batch of experience data from the cache to update the parameters of the value function approximation network; when the fluctuation of the predicted value of the value function approximation network is lower than a first preset threshold in multiple consecutive training cycles, and the similarity of the selected target action sequence reaches a second preset threshold, the value function approximation network is determined to have converged, and a reinforcement learning agent is generated based on the converged value function approximation network.

[0008] Optionally, the exploration rate parameter is configured as a function that monotonically decreases from the initial value to the final value, and the value of the exploration rate parameter is dynamically calculated during training based on the number of iterations completed.

[0009] Optionally, the process of determining the reward value includes the following steps: when a security vulnerability defined as the first risk level by a preset rule is successfully identified, a first positive reward value is generated; when a security vulnerability defined as the second risk level by a preset rule is successfully identified, a second positive reward value is generated, wherein the first risk level is less than the second risk level, and the second positive reward value is less than the first positive reward value; when a security vulnerability is incorrectly identified, a negative reward value is generated.

[0010] Optionally, the feature vector includes at least one of the following feature components: a user-dimensional feature component, which includes at least the risk level identifier of the user account and the account balance; a product-dimensional feature component, which includes at least the risk level identifier of the financial product and the transaction amount limit information; and a transaction-dimensional feature component, which includes at least the transaction amount of this transaction and the cumulative number of transactions on the day.

[0011] Optionally, a reward value is generated based on the detection results, and the reward value is fed back to the reinforcement learning agent to drive the update of the reinforcement learning agent, including: after performing vulnerability detection actions on the business system, obtaining the latest status information of the business system; combining the latest status information of the business system, the vulnerability detection actions, and the reward value to form a target experience data; and updating the reinforcement learning agent based on the target experience data.

[0012] Optionally, after generating detection results containing vulnerability risk levels, the agent-based security vulnerability detection method further includes: sending information about target security vulnerabilities with vulnerability risk levels greater than preset levels to a human review interface; and calibrating the reward value corresponding to the target security vulnerability based on the confirmation or correction results returned by the human review interface.

[0013] According to another aspect of the embodiments of this application, a security vulnerability detection device based on an intelligent agent is also provided, comprising: a feature extraction unit, used to extract feature vectors representing the security status of a business system from the operational data and transaction logs of the business system; an action determination unit, used to input the feature vectors as state information of the business system into a pre-trained reinforcement learning agent, and output vulnerability detection actions for the business system through the reinforcement learning agent; wherein the reinforcement learning agent is constructed based on a value function approximation network and configured with a reward feedback mechanism corresponding to the business security objective; an action execution unit, used to determine whether there is a security vulnerability in the current business scenario of the business system based on the execution result of the vulnerability detection action, and generate a detection result including the vulnerability risk level; and a feedback unit, used to generate a reward value based on the detection result and feed the reward value back to the reinforcement learning agent to drive the update of the reinforcement learning agent.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the above-described agent-based security vulnerability detection method.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the above-described agent-based security vulnerability detection method.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implement the above-described agent-based security vulnerability detection method.

[0017] In this embodiment, the security monitoring system first extracts feature vectors representing the security status of the business system from its operational data and transaction logs. These feature vectors are then input as state information of the business system into a pre-trained reinforcement learning agent, which outputs vulnerability detection actions targeting the business system. The reinforcement learning agent is constructed based on a value function approximation network and configured with a reward feedback mechanism corresponding to the business security objective. Then, based on the execution results of the vulnerability detection actions, the system determines whether a security vulnerability exists in the current business scenario and generates a detection result including the vulnerability risk level. Finally, a reward value is generated based on the detection result and fed back to the reinforcement learning agent to drive its updates.

[0018] As described above, the security monitoring system extracts feature vectors representing the security status of business systems from their operational data and transaction logs. This transforms massive amounts of unstructured business data into structured inputs that can be efficiently processed by machines, providing accurate and unified data for subsequent intelligent analysis and reducing the manual and inefficient bottlenecks of traditional tools in complex data preprocessing and feature engineering. Secondly, the security monitoring system inputs these feature vectors as the business system's state information into a pre-trained reinforcement learning agent. This agent outputs vulnerability detection actions targeting the business system. By utilizing a reinforcement learning agent built based on value function approximation networks instead of fixed rule matching, the system can dynamically generate detection strategies adapted to specific scenarios based on historical experience and real-time environmental conditions, thereby improving the targeting and speed of detection actions. Furthermore, based on the execution results of vulnerability detection actions, the system determines whether the business system has security vulnerabilities in the current business scenario and generates detection results including vulnerability risk levels. This facilitates real-time, automated judgment and risk quantification of security status, transforming the lengthy process of traditional manual assessment into rapid structured output and shortening the cycle from detection to generating usable conclusions. Finally, the security monitoring system generates reward values ​​based on the detection results and feeds these rewards back to the reinforcement learning agent to drive its updates. This forms a closed-loop self-optimization mechanism, enabling the security monitoring system to continuously adjust and optimize the decision-making model of the reinforcement learning agent based on the effectiveness of each detection practice. This facilitates iterative self-improvement in detection accuracy and execution efficiency, reducing the inefficiency caused by lagging detection strategy updates and difficulty in continuously evolving with changes in business and threats. Ultimately, this solves the technical problem of low security detection efficiency in financial business scenarios in existing technologies. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a flowchart of an optional agent-based security vulnerability detection method according to an embodiment of this application;

[0021] Figure 2 This is an overall architecture diagram of an optional agent-based security vulnerability detection method according to an embodiment of this application;

[0022] Figure 3 This is a flowchart of an optional model deep training method according to an embodiment of this application;

[0023] Figure 4This is a flowchart of an optional greedy strategy evaluation according to an embodiment of this application;

[0024] Figure 5 This is a flowchart of an optional model convergence determination according to an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of an optional agent-based security vulnerability detection device according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] According to an embodiment of this application, a method embodiment for a security vulnerability detection method based on an intelligent agent is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0030] According to the embodiments of this application, a security monitoring system can be used as the execution subject of the agent-based security vulnerability detection method of this application. The system can be a software system or an embedded system combining software and hardware. Of course, the execution subject of the method in the embodiments of this application can also be other forms of execution subject, such as devices, equipment, etc. It should be known by those skilled in the art that this application does not particularly limit the specific form of the execution subject.

[0031] Figure 1 This is a schematic diagram of a security vulnerability detection method based on intelligent agents according to an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0032] Step S101: Extract feature vectors from the operational data and transaction logs of the business system to characterize the business security status.

[0033] Optionally, the operational data of the business system may include metrics reflecting the real-time status of the system, such as server central processing unit (CPU) utilization, memory usage, network connections, and application programming interface (API) call frequency. Transaction logs can record every financial transaction, such as login, transfer, payment, and query, and may include information such as timestamps, user identifiers, transaction type, transaction amount, involved accounts, and transaction result status codes.

[0034] Optionally, a feature vector can refer to an ordered set of numerical or categorical data, where the data at each position (dimension) represents a specific aspect of the business security status. For example, a feature vector can contain values ​​for multiple dimensions such as user risk level, account balance, transaction amount, and number of transactions in a given day.

[0035] Optionally, the security monitoring system can predefine a set of rules related to common financial business vulnerabilities such as unauthorized access and transaction tampering. It can then parse event or field values ​​that conform to these rules from operational data and transaction logs, encode the parsing results into numerical or category labels, and combine them into feature vectors. Alternatively, the security monitoring system can apply Natural Language Processing (NLP) techniques or sequence pattern mining algorithms to analyze unstructured log text, automatically extracting pattern features such as abnormal API call sequences and high-frequency error code combinations, and converting them into feature vectors. In essence, the security monitoring system can transform the unstructured, massive amounts of operational data and transaction logs from financial business systems into machine-processable, structured features that reflect the system's security posture.

[0036] Step S102: The feature vector is input as the state information of the business system into the pre-trained reinforcement learning agent, and the reinforcement learning agent outputs vulnerability detection actions for the business system; wherein, the reinforcement learning agent is constructed based on the value function approximation network and is configured with a reward feedback mechanism corresponding to the business security goal.

[0037] Alternatively, a reinforcement learning agent can refer to a decision-making entity that follows a reinforcement learning framework and learns optimal policies through interaction with its environment. The environment can be a simulated or real business system, and the agent learns how to more effectively discover vulnerabilities by performing detection actions and observing system feedback.

[0038] Alternatively, a value function approximation network can refer to a neural network model used to approximate a state value function or an action value function. For example, a Deep Q-Network (DQN) is a commonly used value function approximation network that takes state information as input and outputs the expected cumulative reward (Q-value) for each possible action in that state.

[0039] Optionally, vulnerability detection actions can refer to specific operations that a reinforcement learning agent chooses to perform in a particular state. For example, actions could include "simulating an API request by user A to access user B's account records without authorization," "replaying a transaction by modifying specific parameters," or "checking whether a configuration item meets the security baseline."

[0040] Optionally, the reward feedback mechanism can define the immediate evaluation signal (reward value) that the agent receives from the environment after performing an action. The calculation of the reward value is linked to business security objectives; for example, a positive reward is given for successfully discovering a vulnerability, while zero or negative rewards are given for not discovering or falsely reporting a vulnerability.

[0041] By using business security states, such as feature vectors, as state inputs, the security monitoring system enables reinforcement learning agents to evaluate the long-term value of different actions, such as detection strategies, based on their internal value function approximation network. It then outputs the current optimal or exploratory vulnerability detection action, facilitating the shift from static rule matching to dynamic intelligent decision-making. Configuring a reward feedback mechanism ensures that the agent's behavioral goals align with actual business security objectives, providing direction for continuous optimization.

[0042] Step S103: Based on the execution result of the vulnerability detection action, determine whether there is a security vulnerability in the business system in the current business scenario, and generate a detection result including the vulnerability risk level.

[0043] Optionally, the execution results may include the response status code of the business system after performing the vulnerability detection action, the returned data content, changes in log records, abnormal fluctuations in system performance indicators, etc.

[0044] Optionally, the vulnerability risk level can refer to a classification of the potential harm of a discovered security vulnerability, such as "critical," "high-risk," "medium-risk," and "low-risk." The classification criteria can be based on the Common Vulnerability Scoring System (CVSS) score or defined in conjunction with the potential impact of specific business scenarios, such as the risk of financial loss or the scope of data leakage.

[0045] The security monitoring system automates the analysis and judgment of the execution feedback of intelligent agents' actions, quickly identifying the existence of security vulnerabilities and performing preliminary risk classification on the discovered vulnerabilities. It transforms raw, potentially complex system responses such as error codes and abnormal data into structured detection reports that can be directly used to guide remediation. Automating vulnerability confirmation and classification helps avoid the significant time consumed by traditional manual analysis, shortening the cycle from detection to reaching a usable conclusion.

[0046] Step S104: Generate a reward value based on the detection results and feed the reward value back to the reinforcement learning agent to drive the update of the reinforcement learning agent.

[0047] Optionally, the detection results can be used to generate a quantified reward signal, which is then fed back to the reinforcement learning agent. Based on this reward value, the agent updates the parameters of the value function approximation network through its internal machine learning algorithm. This allows the agent to learn from each detection practice: strengthening strategies that successfully discover vulnerabilities and weakening strategies that are ineffective or lead to false positives. By continuously driving the agent to update, its detection accuracy and efficiency can be gradually improved, adapting to constantly changing business logic and new attack methods, thereby facilitating the dynamic evolution of detection capabilities.

[0048] For example, Figure 2 The overall architecture of the agent-based security vulnerability detection method is shown. The implementation process of the agent-based security vulnerability detection method can include: data input and acquisition, preprocessing and feature extraction, agent core processing operations, vulnerability detection, and result generation. This implementation process begins with defining the system to be detected and data input and acquisition, which can be performed by the data acquisition module to collect raw data. The raw data then enters the data preprocessing stage, where the feature extraction device can sort out the detection points and extract key features to complete the transformation from business system data to feature vectors representing the business security status. The extracted feature vectors are transmitted as input to the agent core device, which can integrate a reinforcement learning agent including model deep training, greedy policy evaluation, and model convergence judgment functions. The reinforcement learning agent generates initial result output based on the input. This initial result output is further processed through a professional review and confirmation stage, which can generate environmental feedback and provide reward signals based on the environmental feedback. The provided reward signals are transmitted to the reinforcement learning training device, which updates the model parameters and drives model reinforcement learning based on the reward signals, thereby completing one training iteration and optimization of the reinforcement learning agent. The entire implementation process can form an automated closed-loop system from data collection, feature extraction, agent decision-making, result review to feedback learning.

[0049] During the data input and collection process, it is first necessary to identify the system to be tested and confirm that the operational data, transaction logs, and configuration information of the target financial business system are all input into the agent-based security vulnerability detection device. Then, the data acquisition module in the agent-based security vulnerability detection device receives the raw data. The sources of the raw data include real-time transaction data streams and historical log files recorded in the database.

[0050] In the preprocessing and feature extraction process, data preprocessing is first required. The raw data received by the data acquisition module is sent to the preprocessing module for data cleaning, noise removal, data normalization, and format standardization to ensure the raw data conforms to the data format standards required for the input of the reinforcement learning agent model. Then, the key detection points are identified. Based on the specific business rules of the target financial business system, key detection points related to security detection are designed. For example, in the scenario of purchasing wealth management products, multiple steps are involved, including amount calculation, risk matching, and quota control. Examples of key business rules and security detection points to be identified are shown in Table 1.

[0051] Table 1

[0052]

[0053] Finally, key feature extraction is performed. From the preprocessed data, key features associated with security detection points are extracted to construct a feature vector representing the business security status. The feature vector extraction can cover dimensions such as abnormal transaction patterns, illegal application programming interface (API) call sequences, or sensitive configuration changes.

[0054] Taking online banking as an example, consider a scenario in a payment system where user A attempts to view and access user B's transfer records. First, an example of the original transaction record data can be obtained from the system logs, with the following structure:

[0055] {

[0056] # Query the user's SessionID;

[0057] "user_session": "XXXXXX";

[0058] # Target user's account name;

[0059] "target_account": "XXXXXXX";

[0060] # Target access path;

[0061] "api_endpoint": "XXX / api / transfer / XXXXXX / records";

[0062] # Request parameters and timestamp;

[0063] "request_params": {"request":"XXXXXXX","timeStamp": "XXXXXXX"};

[0064] # System returns status and data;

[0065] "system_response": {"status": XXX, "data": "XXXXXXXXXXXXXXXXXX"};

[0066] }

[0067] Based on these raw records, a structured feature vector dataset can be extracted and constructed. This feature vector includes feature components of multiple dimensions. A specific example is shown below: Key Feature Vector Dataset:

[0068] state_features = {

[0069] # Feature components of the user dimension;

[0070] "user_risk_level": "X", # Risk level indicator for user account;

[0071] "user_balance": XXXXX.XXX, # User account balance;

[0072] "user_daily_limit": XXXX.XX, # Daily limit for user accounts;

[0073] # Feature components of the product dimension;

[0074] "product_risk_level": "X", # Risk level indicator for financial products;

[0075] "product_min_amount": XXXX.XX, # Minimum transaction amount for financial products;

[0076] "product_max_amount": XXXX.XX, # The maximum transaction amount for financial products;

[0077] # Feature components of the transaction dimension;

[0078] "purchase_amount": XXXX.XX, # The transaction amount for this transaction;

[0079] "current_time": "XX:XX:XX", # The transaction time of this transaction;

[0080] "transaction_count": X # Total number of transactions for the day;

[0081] }

[0082] The security monitoring system constructs a reinforcement learning agent based on a value function approximation network and configured with a reward feedback mechanism. It utilizes the exploration and exploitation strategies of the reinforcement learning agent to achieve intelligent vulnerability detection, which can effectively improve the efficiency and accuracy of security detection in financial business scenarios.

[0083] The security monitoring system is based on a security vulnerability detection method for intelligent agents that is deeply integrated with financial business scenarios. By combining the analysis of architects' and information security experts' experience with a sampling review and confirmation mechanism for target security vulnerabilities, the detection accuracy of security vulnerabilities in core financial business scenarios such as "unauthorized access" and "transaction tampering" far exceeds that of general detection tools.

[0084] In some optional embodiments, the reinforcement learning agent is trained through the following steps: initializing the parameters of the value function approximation network and creating a cache for storing training experience data; constructing a simulation environment based on historical business data of the business system; performing iterative training in the simulation environment, wherein each iteration includes: determining the exploration probability based on the exploration rate parameter, selecting a target action corresponding to the value function approximation network from multiple preset vulnerability detection actions based on the exploration probability and executing it; after executing the target action, collecting the reward value returned by the simulation environment and the latest state information of the business system, and forming experience data stored in the cache; periodically randomly sampling a batch of experience data from the cache to update the parameters of the value function approximation network; when the fluctuation range of the predicted value of the value function approximation network is lower than a first preset threshold in multiple consecutive training cycles, and the similarity of the selected target action sequence reaches a second preset threshold, the value function approximation network is determined to have converged, and a reinforcement learning agent is generated based on the converged value function approximation network.

[0085] Optionally, the parameters of the value function approximation network can refer to the adjustable weights and biases that constitute the value function approximation network. Initialization is assigning initial values ​​to the parameters of the value function approximation network, for example, by random initialization or pre-trained model initialization.

[0086] Optionally, a simulation environment is a program that simulates a real business system and its interaction logic. It receives action instructions from the agent, simulates the system response, and returns new states and rewards. Building a simulation environment based on historical business data means that training can be conducted in a safe, controllable, and quickly reconfigurable simulation scenario, avoiding the risks and costs of trial and error on a real production system.

[0087] Optionally, the experience replay buffer is a cache used to store experience data generated by the interaction between the agent and the simulation environment. Randomly sampling small batches of experience data for training helps to break the temporal correlation between data, improving training stability and data utilization.

[0088] The security monitoring system provides a stable and repeatable training method for reinforcement learning agents by defining the initialization, iterative training with exploration in a simulation environment, updating network parameters by sampling data from the experience replay buffer, and determining convergence based on a preset threshold. This helps the agent systematically learn the complex relationship between the business environment and security vulnerabilities and helps avoid instability or divergence during the training process.

[0089] For example, the core processing operations of an agent may include deep model training, greedy policy evaluation, and model convergence determination. Feature vectors are input into the reinforcement learning agent to conduct deep model training.

[0090] Figure 3The flowchart shown illustrates deep model training, which can include iterative training from initialization to convergence.

[0091] Step a, network parameter initialization configuration. Initialize the value function to approximate the network's weight parameters. The learning rate α is set to 0.001, the discount factor γ to 0.99, and the experience replay buffer capacity D to 100,000. The parameters of the target network corresponding to the synchronous initialization value function approximation network are then determined. The target network is set to update once every 1000 training steps.

[0092] Step b, Training environment preparation. Configure training environment parameters, setting the total number of training cycles to 100,000 and the maximum number of steps per training cycle to 1,000. Reset the simulation environment to its initial state s0, clear the historical interaction record, and prepare to start the training cycle.

[0093] Step c, Strategy action selection and execution. Select action a according to the greedy strategy determined by the current exploration rate parameter ε. Execute action a in the simulation environment and observe the immediate reward value r1 returned by the simulation environment and the latest state information s1 of the business system. Record the complete interaction experience tuple (s, a, r1, s1).

[0094] Step d: Experience data storage management. Store the experience tuple (s, a, r1, s1) into the experience replay buffer. When the experience replay buffer reaches its capacity limit, update its contents using a first-in, first-out (FIFO) strategy. Maintain a balance between data diversity and timeliness within the experience replay buffer.

[0095] Step e, batch training data sampling. Randomly sample small batches of empirical data from the empirical replay buffer, with the batch size set to 32. This ensures the randomness and representativeness of the sampled data, avoiding training bias.

[0096] Step f: Network parameter optimization and update. Calculate the target Q value:

[0097] Formula (1);

[0098] in, Indicates the target Q value. This indicates an immediate reward upon performing the action. Then, the environment returns an immediate feedback signal. The discount factor is used to represent the discount factor, which can be selected from (0,1). For example, it can be selected as 0.99. The discount factor is used to reduce the weight of future rewards in the current decision, reflecting the balance between long-term and short-term benefits. Indicates the next state Next, iterate through all possible actions. Select the target network (parameter is...) Output The action with the largest value value. Represents the target network Value function, input is state and actions The output is the predicted expected cumulative reward. The parameters of the target network... It will be copied from the current learning network periodically to stabilize the training process.

[0099] The loss function that minimizes the mean squared error can be expressed as:

[0100] Formula (2);

[0101] in, Represents the loss function. The parameters represent the values ​​of the approximation network (e.g., a deep Q-network). This indicates the number of samples in the mini-batch training data. Used to indicate the first The target Q value for each sample Indicates the current network parameters Under, state and actions The corresponding predicted Q-value. The network parameters are updated using the Adaptive Moment Estimation (Adam) optimizer by minimizing the mean squared error loss function to approximate the value function. .

[0102] Step g: The target network is updated synchronously. After every 1000 training steps, the parameters of the network are approximated by the current value function. Parameters copied to the target network To maintain the relative stability of the target network and ensure the convergence of the training process.

[0103] Step h: Training effect evaluation. Evaluate the Q-value stability, policy consistency, and performance of the current value function approximation network to determine if it has reached the preset convergence criterion. If the value function approximation network has not reached the convergence criterion, return to the action selection and continue the training loop. When the fluctuation of the predicted value of the value function approximation network is lower than the first preset threshold over several consecutive training cycles, and the similarity of the selected target action sequence reaches the second preset threshold, the value function approximation network is deemed to have converged. A reinforcement learning agent is then generated based on the converged value function approximation network, the training model is saved, and training is complete.

[0104] In some optional embodiments, the exploration rate parameter is configured as a function that monotonically decreases from an initial value to a final value, and the value of the exploration rate parameter is dynamically calculated during training based on the number of iterations completed.

[0105] Optionally, in a greedy strategy, the exploration rate parameter ε (epsilon) can represent that the agent randomly selects an action (exploration) with a probability of ε, and selects the action currently considered optimal with a probability of 1-ε (exploitation).

[0106] Alternatively, a monotonically decreasing function indicates that the value of the exploration rate parameter only decreases and does not increase as the number of training iterations increases. For example, linear decay, exponential decay, or piecewise decay can be used.

[0107] The safety monitoring system encourages the agent to try a wide range of possible detection actions by setting a high exploration rate in the early stages of training, so as to fully understand the environment and discover potential effective strategies. As the number of training steps increases, the exploration rate decays linearly or non-linearly to a lower final value, allowing the agent to gradually rely on the learned knowledge and focus on optimizing and utilizing the most effective detection strategies.

[0108] For example, Figure 4 A flowchart of the greedy policy evaluation process is shown. The greedy policy evaluation process may include:

[0109] Step a, State Feature Extraction. Obtain the current simulation environment state s, extract system operation status data, business indicator data, and security feature vectors from the current simulation environment state s, and construct a state feature representation of the current simulation environment state s to provide complete simulation environment information for the policy decision-making of the reinforcement learning agent.

[0110] Step b, Dynamic exploration rate calculation. The current value of the exploration rate parameter ε is dynamically calculated based on the training progress. The exploration rate parameter ε is updated using a linear decay strategy, which can be expressed as:

[0111] Formula (3);

[0112] in, Used to represent the exploration rate, This is used to represent the initial exploration rate, and its value can be 1.0. This is used to represent the final exploration rate, and its value can be 0.01. Used to indicate the number of training iterations that have been completed. This is used to represent the total number of decay steps, and its value can be 10000.

[0113] Step c, Exploration and Utilization Decision-Making. Generate a random number ρ that follows a uniform distribution in the interval (0,1). Compare the random number ρ with the exploration rate parameter ε. When the random number ρ is less than or equal to the exploration rate parameter ε, the reinforcement learning agent enters exploration mode; when the random number ρ is greater than the exploration rate parameter ε, the reinforcement learning agent enters utilization mode. This comparison operation achieves an intelligent balance between exploration and utilization.

[0114] Step d, Exploration mode action selection. In exploration mode, the reinforcement learning agent randomly selects a vulnerability detection action 'a' from a pre-defined vulnerability detection action space consisting of multiple vulnerability detection actions.

[0115] Step e, Optimal Action Selection Based on Pattern Exploitation. In the exploit mode, the reinforcement learning agent uses the current value function approximation network (Q-network) to compute the Q-values ​​of all possible vulnerability detection actions. The reinforcement learning agent selects the vulnerability detection action with the highest Q-value as the target action. ,Right now =argmax a Q(s,a; Make full use of existing knowledge.

[0116] Step f, Action Execution and Feedback Collection. Execute the selected vulnerability detection action a or target action in the simulation environment. Observe the state transitions in the simulation environment caused by performing vulnerability detection actions and the immediate reward value feedback returned by the simulation environment to collect data on the actual effect of policy execution.

[0117] Step g, Policy Performance Evaluation and Optimization. The policy performance of the reinforcement learning agent is evaluated based on the execution results of the vulnerability detection actions. The decay policy parameter of the exploration rate ε is adjusted according to the reward signal and state value change trend returned by the simulation environment to continuously optimize the performance of the reinforcement learning agent.

[0118] For example, Figure 5 The flowchart for determining model convergence is shown. Figure 5 This demonstrates a multi-dimensional judgment process from initial evaluation to final confirmation or return to training. The model convergence judgment process may include:

[0119] Step a, Q-value stability in-depth analysis. Calculate the coefficient of variation of the Q-value of the value function approximation network over consecutive training cycles, and analyze the convergence stability of the Q-function of the value function approximation network. Set a first preset threshold. When the Q-value fluctuation range is less than the first preset threshold for 10 consecutive training cycles At that time, it is assumed that the Q-value of the value function approximation network reaches a stable state.

[0120] Step b: Rigorous test of strategy consistency. Statistical analysis of strategy change frequency. Frequency of strategy changes The calculation can be referenced from formula (4):

[0121] Formula (4);

[0122] in, [·] indicates an indicator function. This represents the action chosen in state s during the t-th training epoch. The convergence of the policy is evaluated based on the frequency of policy changes. Less than the second preset threshold And it remained stable for 20 consecutive training cycles. Second preset threshold. .

[0123] Step c, comprehensive performance evaluation. Analyze the sliding window average of the reward curve and calculate the reward growth rate. When the reward growth rate The absolute value is less than the third preset threshold. When this condition is met for five consecutive sliding window periods, the performance of the decision value function approximation network reaches saturation. Third preset threshold. .

[0124] Step d: Verify training progress. Confirm that the number of training cycles has reached the minimum required number of training cycles. Ensure the value function approximation network is adequately trained. Check auxiliary indicators such as training time and resource consumption to comprehensively evaluate the training progress of the value function approximation network.

[0125] Step e, multi-dimensional comprehensive convergence judgment. The convergence state of the value function approximation network is weighted and judged based on the combined results of Q-value stability analysis, policy consistency test, and performance index evaluation. Only when all convergence criteria of Q-value stability, policy consistency, and performance index are simultaneously met is the training of the value function approximation network confirmed to have reached convergence.

[0126] Step f: The early stopping mechanism is intelligently triggered. The performance trend of the value function approximation network during training is monitored. If the validation set performance continues to decline for 100 consecutive training epochs, the early stopping mechanism is automatically triggered to stop training the value function approximation network and prevent overfitting.

[0127] Step g: Final Convergence Confirmation. When all convergence criteria are met, the training of the value function approximation network is confirmed to be complete. Save the final model parameters and training state of the value function approximation network, and generate a training report for the value function approximation network.

[0128] Step h: Continue executing training instructions. For models that have not met the convergence criteria, return to the model depth training process for further optimization. Adjust training hyperparameters or training strategies until the value function approximation network satisfies the convergence condition.

[0129] In some optional embodiments, the process of determining the reward value includes the following steps: when a security vulnerability defined as a first risk level by a preset rule is successfully identified, a first positive reward value is generated; when a security vulnerability defined as a second risk level by a preset rule is successfully identified, a second positive reward value is generated, wherein the first risk level is less than the second risk level, and the second positive reward value is less than the first positive reward value; when a security vulnerability is incorrectly identified, a negative reward value is generated.

[0130] Optionally, the first and second risk levels can be an ordered classification of vulnerability risk levels. For example, "critical" and "high risk" can be defined as the first risk level (higher), and "medium risk" and "low risk" can be defined as the second risk level (lower). The level classification can be determined based on CVSS score thresholds or internal risk assessment strategies.

[0131] Optionally, the first positive reward value and the second positive reward value correspond to quantitative incentive signals for different risk levels. For example, the first positive reward value corresponding to the first risk level (higher) should be greater than the second positive reward value corresponding to the second risk level (lower) to reflect the risk difference.

[0132] By assigning differentiated positive reward values ​​for vulnerability discovery at different risk levels, the reward function explicitly conveys the priority of business security goals to the agent: encouraging it to prioritize the discovery of vulnerabilities with greater potential harm. Simultaneously, assigning negative reward values ​​for incorrect identifications penalizes the agent for generating invalid alerts, prompting it to learn more accurate discrimination capabilities. This refined reward design closely aligns the optimization goals of the reinforcement learning agent with real-world security operation and maintenance needs, ultimately guiding the agent to learn a high-quality detection strategy that efficiently discovers high-value vulnerabilities while maintaining a low false positive rate.

[0133] In some optional embodiments, the feature vector includes at least one of the following feature components: a user-dimensional feature component, which includes at least the risk level identifier of the user account and the account balance; a product-dimensional feature component, which includes at least the risk level identifier of the financial product and transaction amount limit information; and a transaction-dimensional feature component, which includes at least the transaction amount of the current transaction and the cumulative number of transactions on the day.

[0134] Optionally, the feature components of the user dimension can refer to data describing the attributes of the business operation initiator, such as the user account's risk tolerance rating, current account balance, and daily or single transaction limit.

[0135] Optionally, the feature components of the product dimension can refer to data that describes the attributes of the business operation object, such as the risk level label of wealth management products or loan products, the minimum / maximum purchase or investment amount set for the product, etc.

[0136] Optionally, the feature components of the transaction dimension can refer to data that describes the attributes of a single business operation, such as the transaction amount of this request, the timestamp of the transaction, and the number of transactions that the user has made in that natural day or session.

[0137] By limiting the feature vector to include at least user-dimensional (e.g., risk level, balance), product-dimensional (e.g., risk level, amount limit), and transaction-dimensional (e.g., transaction amount, frequency) features, it is beneficial to ensure that the state information of the input agent covers the subject and object attributes that may affect core security logic such as permissions, limits, and risk matching. Through multi-dimensional feature design, the reinforcement learning agent can be provided with rich information to determine whether business logic is abnormal, enabling the agent to learn deeper and more granular vulnerability patterns. For example, it can identify compound anomalies such as "low-risk users purchasing high-risk products" or "transaction amounts reaching boundary conditions," thereby improving the accuracy of detection and its coverage of complex scenarios.

[0138] In some optional embodiments, a reward value is generated based on the detection results, and the reward value is fed back to the reinforcement learning agent to drive the update of the reinforcement learning agent, including: after performing a vulnerability detection action on the business system, obtaining the latest status information of the business system; combining the latest status information of the business system, the vulnerability detection action, and the reward value to form a target experience data; and updating the reinforcement learning agent based on the target experience data.

[0139] Optionally, the target experience data can refer to the transition or experience tuple within the reinforcement learning framework. The update operation can employ a Deep Q-Network (DQN) algorithm or a Double DQN algorithm.

[0140] By combining the latest state information of the business system after performing vulnerability detection actions, the executed vulnerability detection actions, and the resulting reward values ​​into a complete set of target experience data, the learning sample format for each interaction can be standardized. Using this target experience data to update the approximate network parameters of the value function within the reinforcement learning agent allows the agent to revise its state-action value estimates based on real interaction consequences, facilitating the transformation of experience and knowledge from concrete detection practices to abstract policy models.

[0141] In some optional embodiments, after generating detection results including vulnerability risk levels, the agent-based security vulnerability detection method further includes: sending information about target security vulnerabilities with vulnerability risk levels greater than preset levels to a human review interface; and calibrating the reward value corresponding to the target security vulnerability based on the confirmation or correction results returned by the human review interface.

[0142] Alternatively, the human review interface can refer to a software interface that allows information security experts or business architects to view vulnerability details reported by an automated system and provide feedback such as confirmation, correction, or rejection.

[0143] Optionally, calibrating the reward value can refer to adjusting the reward value initially automatically generated by the system based on the results of manual review. For example, if the system falsely reports a normal operation as a high-risk vulnerability and awards a negative reward, but manual review confirms that the operation does indeed contain a novel vulnerability not covered by the rules, the reward value can be calibrated to a larger positive reward. Conversely, if the system falsely reports a harmless anomaly as a vulnerability and awards a positive reward, and manual review confirms it is a false alarm, the reward value should be calibrated to a negative reward.

[0144] By sending information about target security vulnerabilities with a risk level exceeding a preset level to a human review interface, high-risk cases can undergo final verification by human experts. This helps correct potential serious misjudgments by the model and ensures the accuracy of detection results in key areas. The reward value corresponding to the target security vulnerability is calibrated based on the confirmation or correction results returned by the human review interface. This allows human expert judgment to be quantified into more precise reward signals, which are then fed back to the reinforcement learning agent. This calibration mechanism enables the agent to learn from the demonstrations or corrections of human experts, quickly correcting erroneous understandings and reinforcing correct patterns. This accelerates the agent's learning process and makes its decision-making logic closer to the expert's actual judgment standards.

[0145] For example, the vulnerability detection and result generation process includes:

[0146] Step a, Result Analysis and Reward Generation. Based on the model feedback from the reinforcement learning agent, determine whether the business system has security vulnerabilities in the current business scenario. Summarize all detection results and generate a complete security report including vulnerability details, vulnerability risk level, and remediation recommendations. The security report is delivered to the corresponding business architect and information security manager through the output interface.

[0147] The relevant business architects and information security managers conduct sampling reviews and confirmations of the model's findings. Initially, 30% to 50% of the target security vulnerabilities with a risk level of high or above can be manually reviewed and marked, based on the sample size. After the reinforcement learning agent's training accuracy exceeds 80%, the number of target security vulnerabilities sampled and marked can be appropriately reduced.

[0148] If the security vulnerability discovered by the reinforcement learning agent is confirmed to be accurate, then detailed information about the target security vulnerability is analyzed, including triggering conditions, system characteristic data, and vulnerability risk level. Vulnerability risk levels are categorized as critical, high, medium, low, and no risk.

[0149] The instant reward value r obtained for this vulnerability detection action is calculated based on the preset reward function.

[0150] The reward function R can be designed as follows:

[0151] R = α × reward for vulnerability discovery count + β × reward for vulnerability risk level - γ × penalty for false alarms;

[0152] Among them, α=0.6, β=0.3, γ=0.1.

[0153] Step b: Model reinforcement learning and feedback update.

[0154] When a reinforcement learning agent discovers a security vulnerability, it is given a positive reward, and different reward coefficients are set according to the risk level of the target security vulnerability.

[0155] The reward value generated based on the detection results is fed back to the reinforcement learning device. The reinforcement learning device calculates the gradient based on the reward value, updates the model parameters such as the value function approximation network within the reinforcement learning agent, and records the latest state information of the business system after the vulnerability detection action is executed. And the updated parameter combinations.

[0156] In the next detection, the reinforcement learning agent adopts a parameter strategy optimized based on reward value feedback, continuously accumulating high-quality vulnerability samples and improving the detection accuracy of the reinforcement learning agent.

[0157] The reinforcement learning mechanism of a reinforcement learning agent can feed the results of human review as new samples into the reinforcement learning agent model. The results of human review are used to generate or calibrate reward values, and the value function approximates the network parameters of the reinforcement learning agent based on these reward values, enabling the agent to continuously adapt to reinforcement learning. The reinforcement learning agent can continuously optimize itself as financial business and security threats evolve.

[0158] See Figure 6 According to another aspect of the embodiments of this application, a security vulnerability detection device based on intelligent agents is also provided, including: a feature extraction unit, an action determination unit, an action execution unit, and a feedback unit.

[0159] The system comprises the following components: a feature extraction unit, which extracts feature vectors representing the security status of the business system from its operational data and transaction logs; an action determination unit, which inputs the feature vectors as the state information of the business system into a pre-trained reinforcement learning agent, and outputs vulnerability detection actions for the business system through the reinforcement learning agent; the reinforcement learning agent is constructed based on a value function approximation network and configured with a reward feedback mechanism corresponding to the business security objective; an action execution unit, which determines whether there are security vulnerabilities in the current business scenario based on the execution results of the vulnerability detection actions, and generates detection results including the vulnerability risk level; and a feedback unit, which generates reward values ​​based on the detection results and feeds the reward values ​​back to the reinforcement learning agent to drive its updates.

[0160] Optionally, the action determination unit includes: a first processing subunit for initializing the parameters of the value function approximation network and creating a cache for storing training experience data; a second processing subunit for constructing a simulation environment based on historical business data of the business system; a third processing subunit for iterative training in the simulation environment, wherein each iteration includes: determining the exploration probability based on the exploration rate parameter, selecting the target action corresponding to the value function approximation network from multiple preset vulnerability detection actions based on the exploration probability and executing it; after executing the target action, collecting the reward value returned by the simulation environment and the latest status information of the business system, and forming experience data stored in the cache; periodically randomly sampling a batch of experience data from the cache to update the parameters of the value function approximation network; and a fourth processing subunit for determining that the value function approximation network has converged when the fluctuation range of the predicted value of the value function approximation network is lower than a first preset threshold in multiple consecutive training cycles, and the similarity of the selected target action sequence reaches a second preset threshold, and generating a reinforcement learning agent based on the converged value function approximation network.

[0161] Optionally, the third processing subunit includes a parameter configuration module, which configures the exploration rate parameter as a function that monotonically decreases from the initial value to the final value. The value of the exploration rate parameter is dynamically calculated during training based on the number of iterations completed.

[0162] Optionally, the third processing subunit includes: a first processing module, used to generate a first positive reward value when a security vulnerability defined as a first risk level by a preset rule is successfully identified; a second processing module, used to generate a second positive reward value when a security vulnerability defined as a second risk level by a preset rule is successfully identified, wherein the first risk level is less than the second risk level, and the second positive reward value is less than the first positive reward value; and a third processing module, used to generate a negative reward value when a security vulnerability is incorrectly identified.

[0163] Optionally, the feature extraction unit includes: a first extraction subunit for extracting user-dimensional feature components, which at least include the user account's risk level identifier and account balance; a second extraction subunit for extracting product-dimensional feature components, which at least include the financial product's risk level identifier and transaction amount limit information; and a third extraction subunit for extracting transaction-dimensional feature components, which at least include the transaction amount of the current transaction and the cumulative number of transactions on the day.

[0164] Optionally, the feedback unit includes: a fifth processing subunit, used to obtain the latest status information of the business system after performing vulnerability detection actions on the business system; a sixth processing subunit, used to combine the latest status information of the business system, the vulnerability detection actions, and the reward value to form a target experience data; and a seventh processing subunit, used to update the reinforcement learning agent based on the target experience data.

[0165] Optionally, the agent-based security vulnerability detection device further includes: an information sending unit, used to send information about target security vulnerabilities with a vulnerability risk level greater than a preset level to a manual review interface; and a calibration unit, used to calibrate the reward value corresponding to the target security vulnerability based on the confirmation or correction results returned by the manual review interface.

[0166] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the above-described agent-based security vulnerability detection method.

[0167] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the above-described agent-based security vulnerability detection method.

[0168] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implement the above-described agent-based security vulnerability detection method.

[0169] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0170] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0172] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0174] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0175] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A security vulnerability detection method based on intelligent agents, characterized in that, include: Extract feature vectors from the operational data and transaction logs of the business system to characterize the business security status; The feature vector is used as the state information of the business system and input into a pre-trained reinforcement learning agent. The reinforcement learning agent outputs vulnerability detection actions for the business system. The reinforcement learning agent is constructed based on a value function approximation network and is configured with a reward feedback mechanism corresponding to the business security objective. Based on the execution result of the vulnerability detection action, determine whether the business system has a security vulnerability in the current business scenario, and generate a detection result including the vulnerability risk level; A reward value is generated based on the detection results, and the reward value is fed back to the reinforcement learning agent to drive the update of the reinforcement learning agent.

2. The security vulnerability detection method based on intelligent agents according to claim 1, characterized in that, The reinforcement learning agent is trained through the following steps: Initialize the parameters of the value function approximation network and create a cache for storing training experience data; A simulation environment is constructed based on the historical business data of the aforementioned business system; Iterative training is performed in the simulation environment, wherein each iteration includes: determining the exploration probability based on the exploration rate parameter; selecting and executing a target action corresponding to the value function approximation network from multiple preset vulnerability detection actions based on the exploration probability; after the target action is executed, collecting the reward value returned by the simulation environment and the latest status information of the business system, and storing the resulting experience data in the cache; periodically sampling a batch of experience data from the cache to update the parameters of the value function approximation network. When the fluctuation range of the predicted value of the value function approximation network is lower than the first preset threshold in multiple consecutive training cycles, and the similarity of the selected target action sequence reaches the second preset threshold, the value function approximation network is determined to have converged, and the reinforcement learning agent is generated based on the converged value function approximation network.

3. The security vulnerability detection method based on intelligent agents according to claim 2, characterized in that, The exploration rate parameter is configured as a function that monotonically decreases from the initial value to the final value, and the value of the exploration rate parameter is dynamically calculated during training based on the number of iterations completed.

4. The security vulnerability detection method based on intelligent agents according to claim 2, characterized in that, The process of determining the reward value includes the following steps: When a security vulnerability defined as the first risk level by preset rules is successfully identified, a first positive reward value is generated. When a security vulnerability defined as the second risk level by a preset rule is successfully identified, a second positive reward value is generated, wherein the first risk level is less than the second risk level, and the second positive reward value is less than the first positive reward value; When a security vulnerability is incorrectly identified, a negative reward value is generated.

5. The security vulnerability detection method based on intelligent agents according to claim 1, characterized in that, The feature vector includes at least one of the following feature components: User-dimensional feature components include at least the user account's risk level identifier and account balance; The product-level feature components should at least include the risk level identification and transaction amount limit information of the financial product; The characteristic components of the transaction dimension include at least the transaction amount of this transaction and the cumulative number of transactions on that day.

6. The security vulnerability detection method based on intelligent agents according to claim 1, characterized in that, Generating a reward value based on the detection results and feeding the reward value back to the reinforcement learning agent to drive the update of the reinforcement learning agent includes: After performing the vulnerability detection action on the business system, obtain the latest status information of the business system; The latest status information of the business system, the vulnerability detection action, and the reward value are combined to form a target experience data; The reinforcement learning agent is updated based on the target experience data.

7. The security vulnerability detection method based on intelligent agents according to claim 1, characterized in that, After generating detection results that include vulnerability risk levels, the agent-based security vulnerability detection method further includes: Information on target security vulnerabilities whose risk level is greater than a preset level is sent to the manual review interface; Based on the confirmation or correction results returned by the manual review interface, the reward value corresponding to the target security vulnerability is calibrated.

8. A security vulnerability detection device based on intelligent agents, characterized in that, include: The feature extraction unit is used to extract feature vectors that characterize the security status of the business from the operational data and transaction logs of the business system. An action determination unit is used to input the feature vector as the state information of the business system into a pre-trained reinforcement learning agent, and output a vulnerability detection action for the business system through the reinforcement learning agent; wherein, the reinforcement learning agent is constructed based on a value function approximation network and is configured with a reward feedback mechanism corresponding to the business security objective; The action execution unit is used to determine whether the business system has a security vulnerability in the current business scenario based on the execution result of the vulnerability detection action, and to generate a detection result including the vulnerability risk level. A feedback unit is used to generate a reward value based on the detection result and feed the reward value back to the reinforcement learning agent to drive the update of the reinforcement learning agent.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device containing the computer-readable storage medium performs the agent-based security vulnerability detection method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the agent-based security vulnerability detection method according to any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the agent-based security vulnerability detection method according to any one of claims 1 to 7.

Images