XSS automatic testing method based on curiosity driving and deep reinforcement learning

Abstract

The invention relates to the technical field of network security vulnerability automatic testing, and discloses a curiosity-driven and deep reinforcement learning-based XSS automatic testing method, which comprises the steps of constructing an input environment for injection point exploration, constructing an intelligent agent with an internal reward mechanism, and constructing an XSS automatic testing system. A state flow diagram is constructed to record and manage a state flow path accessed by the intelligent agent in the exploration process, and injection point information is obtained; transmitting the injection point information to an intelligent agent in an attack vector generation model, and constructing a state space, an action space and a reward function in the attack vector generation model; and carrying out attack verification on the response page after the attack vector generation model sends the attack request to the target page, calculating a reward according to a verification result, and feeding back the reward to the attack vector generation model. According to the method, high coverage rate and high accuracy are achieved in the aspect of identifying the XSS injection points in the Web application, meanwhile, manual intervention is not needed in the training and testing process of the intelligent agent, and the testing efficiency and reliability are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of automated testing of network security vulnerabilities, and in particular to an XSS automated testing method based on curiosity-driven and deep reinforcement learning. Background Art

[0002] XSS attack, or Cross-Site Scripting, is the most common type of network attack. It is known for its wide influence and high risk, and has long been ranked in the top three of the network security vulnerability threat list. Faced with the increasing number of malicious activities on the Internet and the need for comprehensive testing of large-scale applications or systems, manual penetration testing has shown significant defects, including but not limited to consuming a lot of time and resources, limited coverage, and lack of continuity. In response to these shortcomings, automated testing has gradually emerged, and as one of the important research directions of network automated attack technology, XSS automated testing technology plays a key role in promoting network threat detection technology and building a secure network environment.

[0003] XSS vulnerability testing mainly includes two stages: injection point exploration and attack vector generation. Excellent XSS detection technology needs to meet three requirements at the same time: accuracy of injection point exploration, success rate of attack vector generation, and minimized human intervention. Therefore, how to accurately identify XSS injection points and dynamically build rich and effective attack vectors to achieve automated attacks is a pressing issue in current Web application security testing.

[0004] Traditional XSS automated testing tools usually use web crawlers as the main means to identify XSS vulnerability injection points, and obtain pages containing injection points by recursive website crawling. However, this crawling method is difficult to handle dynamic navigation hidden pages in event-driven dynamic web applications. This feature leads to incomplete collection of injection point information, which in turn reduces the detection rate of vulnerabilities. Lekies et al. proposed a fully automated XSS detection system, including a taint-aware JavaScript engine and a corresponding document object model (DOM) implementation generation method, which identified 6167 unique vulnerabilities in the top 5000 websites in Alexa, distributed on 480 domain names. Parameshwaran et al. proposed a testing platform DEXTERJS, which performs character-accurate taint tracking during browser context execution through source-to-source rewriting, enabling it to identify information flows in web pages that may be vulnerable to XSS attacks. Melicher et al. trained a deep neural network (DNN) through taint tracking and more than 18 billion JavaScript functions to predict whether JavaScript functions are vulnerable to DOMXSS attacks. In these studies, the similarity of response pages is usually compared by calculating DOM structure similarity. Recently, the XSS attack vector generation method based on reinforcement learning has provided a feasible way to improve the accuracy and detection efficiency of XSS vulnerability identification, achieving an escape rate of more than 85% on Web application firewalls such as SafeDog. However, these XSS automated detection systems still require manual intervention throughout the testing cycle to train the agent, and rely on manual generation of attack strings between the agent and the Web application and obtaining response results, thus failing to achieve true automated detection. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides an XSS automated testing method based on curiosity-driven and deep reinforcement learning.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0007] An XSS automated testing method based on curiosity-driven and deep reinforcement learning includes the following steps:

[0008] The input environment for injection point exploration is constructed based on the structure, content, and interactive response information of the web page. The curiosity-driven reinforcement learning method is used to construct an intelligent agent with an intrinsic reward mechanism. A state flow graph is constructed to record and manage the state flow path visited by the intelligent agent during the exploration process to obtain injection point information.

[0009] Pass the injection point information to the intelligent agent in the attack vector generation model, and construct the state space, action space and reward function in the attack vector generation model;

[0010] The attack vector generation model is used to verify the response page after sending the attack request to the target page, and the reward is calculated based on the verification result and fed back to the attack vector generation model.

[0011] Optionally, when constructing an input environment for injection point exploration based on the structure, content, and interactive response information of the Web page, it includes:

[0012] The normalized compression distance is used to calculate the DOM structure similarity of two pages, and the page logic is deduplicated based on the DOM structure similarity.

[0013] As an alternative, when using the curiosity-driven reinforcement learning method to construct an intelligent agent with an intrinsic reward mechanism, the intrinsic reward function is specifically:

[0014]

[0015] Among them, R represents the intrinsic reward function, r max Indicates the maximum reward value, num ip Indicates the number of points, num s Indicates the number of visits to the set state, r nor Indicates that r neg indicates, s indicates the state;

[0016] The Q value is updated as follows:

[0017]

[0018] Among them, Q(s,a) represents the updated Q value when reaching the new state s' by executing action a, α represents, γ represents, and Q(s',a') represents.

[0019] As an alternative, an XSS automated testing method based on curiosity-driven and deep reinforcement learning is characterized in that when a state flow graph is constructed to record and manage the state flow paths visited by the agent during the exploration process, the state flow graph is updated in the following manner:

[0020]

[0021] Where, curiosity(s,a) represents, count(s,a) represents the number of times action a is executed in state s, and curiosity(s) represents, A s represents the action set of state s, a i Indicates state s i sequence of actions.

[0022] Optionally, when the injection point information is passed to the intelligent agent in the attack vector generation model, it includes:

[0023] The objective function for establishing a deep reinforcement learning method is:

[0024]

[0025] Among them, π * represents the objective function, argmax represents the maximum value function, E represents the expectation, ρ π represents the distribution of actions taken in the current state under strategy π, s t represents the state of the environment at time t, a t represents the action selected by the agent at time t, r(s t ,a t ) means in state s t Next select action a t The reward of environmental feedback, α represents the weight coefficient, H represents entropy, π(a|s t ) means in state s t The probability of taking action a under

[0026] The action-value function for building a deep reinforcement learning method is:

[0027]

[0028] Among them, Q π (s,a) represents the action value function, γ represents;

[0029] The state value function for establishing the deep reinforcement learning method is:

[0030]

[0031] Among them, V π (s) represents the state value function;

[0032] The iteration method of the action value function is:

[0033]

[0034] Among them, p(s'|s,a) represents, Q π (s',a') indicates;

[0035] The iteration method of the state value function is:

[0036]

[0037] The strategy optimization method of deep reinforcement learning method is:

[0038]

[0039] Among them, π new represents the optimized strategy, arg min represents the minimum function, π represents the strategy, Π represents the strategy set, D KL represents the distance between the policy and the energy-based model, Z represents the normalization function, and π old Indicates the original policy.

[0040] Optionally, when constructing the state space in the attack vector generation model, include:

[0041] A modular design approach is adopted, where an action is represented as a single modification of one of the modules of the XSS attack vector, so that each mutation operation performed on the attack vector is directly associated with the reward feedback provided by the environment.

[0042] Optionally, when constructing the action space in the attack vector generation model, include:

[0043] The state is modeled as a state feature vector, where the state feature vector includes static attack features, dynamic attack features, attack vector type, current action number features, and attack result features.

[0044] Optionally, the reward function in the attack vector generation model is constructed as follows:

[0045]

[0046] Among them, r(s t ,a t ) represents the reward at time t, MAX_STEP represents the maximum number of attacks that can be attempted in each round, and aux rwd represents the auxiliary reward based on edit distance, L represents the length of the module involved in the mutation, lev (before,after) (i,j) represents the edit distance between the original attack vector i and the attack vector j in the response page.

[0047] Alternatively, the edit distance is calculated as:

[0048]

[0049] Where a represents the string of the original attack vector i, and b represents the string of the attack vector j in the response page.

[0050] Optionally, when performing attack verification on a response page after the attack vector generation model sends an attack request to a target page, the following steps are included:

[0051] Set the attack code of all attack vectors to a pop-up window function;

[0052] Monitor whether a pop-up window event occurs; if so, the attack is considered successful; otherwise, proceed to the next step;

[0053] Traverse the DOM syntax tree of the page to search for the pop-up function; if the search is successful, the attack is considered successful; otherwise, the attack is considered failed.

[0054] The present invention has the following beneficial effects:

[0055] The present invention achieves high coverage and high accuracy in identifying XSS injection points in Web applications. At the same time, no human intervention is required during the training and testing process of the intelligent agent, which greatly improves the test efficiency and reliability. The core components of the model include the curiosity-driven injection point exploration algorithm IPExplore and the deep reinforcement learning-driven attack vector generation method AVGenerate. On the one hand, IPExplore effectively solves the common problems of low path coverage and repeated crawling in traditional XSS exploration methods by introducing a curiosity-driven reinforcement learning mechanism, and can adaptively discover hidden injection points in dynamic Web applications to ensure that effective attack information is fully collected. On the other hand, the AVGenerate method generates high-quality attack vectors through dynamic adjustment and real-time feedback, showing a higher attack success rate than traditional Web crawler methods, and can successfully bypass mainstream XSS detection models. This adaptive generation not only improves the effectiveness of vulnerability discovery, but also ensures the diversity and concealment of attack vectors. Finally, by combining these two innovative components, the present invention provides a comprehensive XSS vulnerability detection and attack vector generation solution, showing the ability to achieve fast convergence without human intervention, proving its effectiveness and robustness in practical applications, and providing new ideas and methods for the research of automated attack technology in the field of network security. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of an XSS automated testing method based on curiosity-driven and deep reinforcement learning;

[0057] Figure 2 This is a schematic diagram of an XSS automated testing method framework based on curiosity-driven and deep reinforcement learning;

[0058] Figure 3 Exploring the flow diagram for XSS injection points;

[0059] Figure 4 This is a schematic diagram of the XSS attack vector mutation process;

[0060] Figure 5 This is the flow chart of the attack verification module. DETAILED DESCRIPTION

[0061] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0062] Due to the widespread nature of XSS attacks, traditional manual testing methods can no longer meet the needs. It has become particularly important to achieve test automation, improve test coverage, and enhance the effectiveness of XSS vulnerability security testing. However, the current mainstream XSS automated testing methods may repeatedly crawl similar web pages and still require manual intervention. To this end, the present invention proposes an XSS vulnerability automated testing method based on curiosity-driven and deep reinforcement learning to solve the above problems.

[0063] The technical solution adopted by the present invention specifically includes three stages: injection point exploration, attack vector generation and attack verification.

[0064] Firstly, an adaptive exploration method for injection points based on curiosity-driven reinforcement learning is proposed, called IPExplore (Injection Point Exploration). Curiosity-driven reinforcement learning encourages agents to explore uncertain or unknown states by providing intrinsic rewards (i.e., curiosity value). IPExplore uses this mechanism to distinguish response pages based on DOM structure similarity to improve the efficiency and accuracy of XSS injection point exploration.

[0065] Then, a deep reinforcement learning algorithm SAC (Soft Actor-Critic) was introduced to dynamically generate attack vectors, called AVGenerate (attack vector generation). The functional process of AVGenerate consists of four key modules: intelligent agent, action space, state space construction and reward function, to achieve dynamic and adaptable XSS attack vector generation and improve the success rate of attacks.

[0066] Finally, by combining these two technologies, an automated attack verification method for XSS vulnerabilities was designed, which eliminated human intervention, automated the attack process, and provided a new solution for the research on network automated attack technology.

[0067] The present invention first identifies and extracts possible XSS injection points from Web applications through the injection point exploration component. In this process, the model applies a curiosity-driven reinforcement learning mechanism to efficiently detect unknown states, ensuring that hidden injection points that may be overlooked by traditional methods can be discovered. Then, after identifying the injection point, the model enters the attack vector generation stage, dynamically generating efficient attack vectors through deep reinforcement learning technology to adapt to the structure and characteristics of different Web applications, thereby maximizing the success rate of the attack. Finally, the attack verification component is responsible for verifying the effectiveness of the generated attack vectors for the identified XSS injection points, ensuring that the attack vectors generated by the model are not only theoretically effective, but also can successfully trigger vulnerabilities in actual tests. The implementation of this series of processes enables XSSExplore to efficiently and comprehensively complete automated testing of XSS vulnerabilities in Web applications.

[0068] like Figure 1 and Figure 2 As shown, an XSS automated testing method based on curiosity-driven and deep reinforcement learning provided by an embodiment of the present invention includes the following steps S1 to S3:

[0069] S1. Build the input environment for injection point exploration based on the structure, content and interactive response information of the web page, use the curiosity-driven reinforcement learning method to build an intelligent agent with an intrinsic reward mechanism, and build a state flow graph to record and manage the state flow path visited by the intelligent agent during the exploration process to obtain injection point information;

[0070] In an optional embodiment of the present invention, Figure 3 As shown, IPExplore consists of three components: environment, curiosity-driven reinforcement learning, and SFG. In step S1, IPExplore uses curiosity value as an intrinsic reward to guide the agent's exploration, and uses normalized compression distance to deduplicate similar pages. Normalized compression distance is a general similarity measure that is independent of features, and it will be used to calculate the DOM structure similarity of two pages in the following text. In addition, in order to prevent the agent from falling into a local optimum and not exploring any new states for a long time, the present invention proposes the construction of a state flow graph (SFG). It uses the global information maintained by SFG to further guide the agent to conduct more comprehensive exploration. IPExvole uses curiosity-driven reinforcement learning and SFG to continuously optimize the learning strategy.

[0071] This embodiment uses the environment module to define the specific context of the interaction between the agent and the Web application. In this process, the structure, content and interactive response information of the Web page constitute the input environment for injection point exploration. The agent judges the characteristics of each page through the perception of the environment and screens out possible XSS injection points, thus laying the foundation for subsequent steps.

[0072] In order to improve the efficiency of IPExplore in web page status recognition, the hash algorithm is first used to hash the DOM structure of the page to uniquely represent the page status. In the specific operation, the logical consistency of the two pages is judged by calculating the similarity of the DOM structures, and the normalized compression distance algorithm is used to implement the similarity calculation. When the similarity exceeds the set threshold, the two pages are considered to have the same data processing logic and are classified into the same state; otherwise, they are represented as different states. This method effectively reduces system performance loss by deduplicating page logic.

[0073] This embodiment uses a curiosity-driven reinforcement learning mechanism to encourage the agent to explore unknown states in depth and avoid staying in known states. Through the intrinsic reward system (i.e., curiosity value), the module enables the agent to prioritize paths that have not been visited or explored, thereby finding potential injection points in the widest possible range, avoiding repeated crawling of similar page content, and improving exploration efficiency and coverage.

[0074] In the curiosity-driven reinforcement learning module, IPExplore addresses the sparse reward problem by building an agent with an intrinsic reward mechanism. Specifically, the agent is rewarded not only when it achieves its goal (such as discovering a new state) during execution, but also at each step of exploration, and its reward value is dynamically adjusted based on the frequency of visits to the current state. The higher the visit frequency, the smaller the reward value, which encourages the agent to focus its exploration on new states in order to cover more states and discover hidden injection points. This reward mechanism is particularly critical in XSS tasks, because the dynamic update of the application makes the exploration of different states complex and important.

[0075] The intrinsic reward function designed in this embodiment is specifically:

[0076]

[0077] Among them, R represents the intrinsic reward function, r max Indicates the maximum reward value, num ip Indicates the number of points, num s Indicates the number of visits to the set state, r nor represents the generally high reward given when a new state is explored but does not contain a data injection point, r neg represents the negative reward used to punish the agent for exploring invalid states, s represents the state; r max 、r nor and r negThey are set to 100, 50, and -10, respectively. The specific values ​​of 100, 50, and -10 were chosen based on empirical testing and the need to balance exploratory incentives with practical constraints. max represents the maximum reward, which provides a strong incentive for the agent to discover a new state containing a data injection point. The reward for exploring a state decreases as the number of visits increases, which means that actions that are performed less frequently have a higher curiosity value. Therefore, actions that are performed less frequently are more likely to be selected, which increases the chances of the intelligent agent discovering new states.

[0078] IPExplore uses the Q-learning reinforcement learning algorithm to optimize its exploration strategy. The Q value represents the expected future reward for each state-action pair: Q: S×A→R. The Q value is stored in the Q table, and the agent selects the action that maximizes the long-term reward by querying the Q table. For each state s, when the new state s' is reached by executing action a, the Q value is updated as follows:

[0079]

[0080] Among them, Q(s,a) represents the updated Q value when the new state s' is reached by executing action a, α represents the learning rate, γ represents the decay factor, and Q(s',a') represents the updated Q value when the new state s' is reached by executing action a.

[0081] At each given state, the agent checks whether the state exists in the table. If the given state does not exist in the table, the agent first adds the new state and its corresponding action, and then selects the best action from the available actions in the state. In order to manage the dynamic changes in the state-action space, the system maintains a global action set to manage all states uniformly.

[0082] This embodiment uses a state flow graph (SFG) to record and manage the state flow paths visited by the agent during the exploration process to prevent the agent from being stuck in a local optimum. When the agent encounters a duplicate page or reaches a deadlock, the SFG will guide the agent to move to a new path to ensure the comprehensiveness and coherence of the injection point exploration, thereby significantly improving the ability to discover hidden injection points.

[0083] This embodiment chooses to construct a state flow graph to record state transitions and provide navigation guidance for the agent when necessary. The definition of SFG is as follows: SFG can be represented as a triple (r, s, E), where r represents the initial state, s is the state set, and E represents the transition between page states. During the exploration process, if a new transition relationship is found, the SFG will be updated.

[0084] During state selection, if no new state is found within a period of time, SFG determines the state with the highest curiosity, and the state flow graph is updated as follows:

[0085]

[0086] Among them, curiosity(s,a) represents the curiosity value obtained by executing action a in a single state s, count(s,a) represents the number of times action a is executed in state s, and curiosity(s) represents the number of times action a is executed in action set A. s The average curiosity value of each action obtained after all actions are executed in s represents the action set of state s, a i Indicates state s i IPExplore will navigate directly to this state and record the actions from state s 0 To state s t+1 The action sequence (a 0 ,a 1 ,...,a t ). Subsequently, the agent continues to explore, improving exploration efficiency and coverage.

[0087] S2, passing the injection point information to the intelligent agent in the attack vector generation model, and constructing the state space, action space and reward function in the attack vector generation model;

[0088] In an optional embodiment of the present invention, step S2 passes the obtained URL injection point information to the intelligent agent in AVGenerate after the IPExpore module extracts the injection point. AVGenerate builds a state space, action space, and reward function, which is specially designed for automatic XSS vulnerability discovery. This setting enables the agent to automatically adjust the attack vectors during the interaction with the web application and adapt them to the specific vulnerability context by analyzing the response to the attack attempt.

[0089] This embodiment uses an intelligent agent to automatically generate and adjust attack vectors in a given Web application environment. The intelligent agent consists of a deep reinforcement learning algorithm SAC and an action space. SAC is used to enhance the decision-making ability of the intelligent agent, enabling it to self-adjust its strategy according to the feedback of attack success or failure during the attack process, and finally generate the optimal attack vector.

[0090] The SAC algorithm increases the randomness of strategy selection through entropy regularization, thereby significantly improving exploration ability and convergence speed. The introduction of entropy means that strategy selection will be more random, avoiding the strategy from converging to the local optimal state too early, which is suitable for hierarchical XSS attack vector mutation tasks.

[0091] The objective function for establishing a deep reinforcement learning method is:

[0092]

[0093] Among them, π * represents the objective function, argmax represents the maximum value function, E represents the expectation, ρ π represents the distribution of actions taken in the current state under strategy π, s t represents the state of the environment at time t, a t represents the action selected by the agent at time t, r(s t ,a t ) means in state s t Next select action a t The reward of environmental feedback, α represents the weight coefficient, H represents entropy, π(a|s t ) means in state s t The probability of taking action a under

[0094] In addition to the basic goal of learning a policy that maximizes the expected cumulative reward, it also requires that the entropy of each action chosen by the policy is maximized.

[0095] The action-value function for building a deep reinforcement learning method is:

[0096]

[0097] Among them, Q π (s,a) represents the action value function, γ represents the discount factor;

[0098] The state value function for establishing the deep reinforcement learning method is:

[0099]

[0100] Among them, V π (s) represents the state value function;

[0101] With the basic definition of value function and strategy, the next step is to execute the two processes of strategy evaluation and strategy optimization alternately until the value function and strategy are close to the optimal.

[0102] Strategy evaluation is to accurately predict the benefits that the strategy brings to the agent. However, it cannot be calculated directly by the above two formulas. It needs to be calculated through value iteration. The iteration method of the action value function is:

[0103]

[0104] Among them, p(s'|s,a) represents the state transition probability distribution of the environment, Q π(s',a') represents the action-value function before updating;

[0105] The iteration method of the state value function is:

[0106]

[0107] When the two value functions converge, the evaluation of the current strategy is over, and the next step is strategy optimization. In theory, the ideal strategy of SAC is a strategy represented by an energy-based model (Energy Based Policy, EBP), because EBP has superior strategy representation capabilities. However, EBP cannot be sampled, so SAC finally chose to use Gaussian distribution π to approximate EBP. In the subsequent strategy optimization, try to make this Gaussian distribution close to EBP, and the distance between π and EBP is calculated using KL divergence. The strategy optimization method is:

[0108]

[0109] Among them, π new represents the optimized strategy, arg min represents the minimum function, π represents the strategy, Π represents the strategy set, D KL represents the distance between the policy and the energy-based model, Z represents the normalization function, and π old Indicates the original policy.

[0110] In its specific implementation, SAC uses two deep neural networks to simulate the action-value function and policy selector: the input of the network is the state-action pair, and the output is the value of the action; the input of the policy network is the environment state, and the output is the Gaussian distribution of optional actions under this state.

[0111] This embodiment uses the action space to define the optional behavior or mutation strategy of the intelligent agent, so that the attack vector can be flexibly adjusted and optimized in each iteration. This module provides a variety of mutation strategies to make the generated attack vector more diverse and targeted, which can adapt to the defense characteristics of different Web applications and effectively bypass the detection system.

[0112] In order to train an autonomous reinforcement learning agent to perform XSS automated attacks, it is necessary to formalize the learning problem, that is, to define a reasonable action space, state space, and reward function. Although there are many types of XSS attack vectors, they follow specific grammatical rules and characteristics. Typically, an XSS attack vector includes components such as HTML tags, attributes, pseudo-protocols, events, attack codes, and special symbols.

[0113] All attack vector mutation strategies constitute the action space of the agent. Due to the diversity of mutation strategies applied in different XSS attack vector modules, each strategy is almost always different from each other. In order to avoid improper use of mutation strategies that may render the attack vector invalid, the present invention designs an independent mutation strategy for each module. The construction principle of the action space is to make the actions as atomic as possible, that is, an action only represents a single modification to a certain module of the XSS attack vector. This design ensures that the rewards provided by the environment can be directly associated with the mutation operations performed on the attack vector, and the final action space includes 26 actions. Table 1 lists examples of different target modules.

[0114] Table 1. Table of different target modules

[0115]

[0116]

[0117]

[0118]

[0119]

[0120] This modular action space design allows each mutation operation to be more accurately associated with reward feedback, which helps the reinforcement learning agent to more efficiently explore and generate vectors with a higher attack success rate when generating attack vectors. This also enhances the strategy optimization effect during the XSS automated attack process and improves the adaptability and success rate of the attack.

[0121] In this embodiment, the state space construction is responsible for providing sufficient information support for the intelligent agent. The state space contains multiple attributes of the attack vector, such as static features, dynamic features, and context information, so that the intelligent agent can judge the attack effect and adjust the strategy at each step. By constructing the state space, AVGenerate can better capture the response characteristics of the Web application, thereby more accurately guiding the generation of attack vectors.

[0122] In the process of automated attack, the agent selects the best mutation strategy according to the current environment state. The present invention models the state s as a feature vector s = (f 1 ,f 2 ,…,f n), where each position is a scalar value, indicating the presence or type of the feature represented by the position. Since the automated attack model of the present invention is a black box model, the state vector can only be determined by the attack features of the sample itself and the dynamic features during the mutation process. The final state vector consists of five parts: static attack features, dynamic attack features, attack vector type, current action number, and attack result, a total of 44 bits. The specific description of each module is as follows.

[0123] Static attack features: refers to the features contained in the initial XSS attack vector. Table 2 summarizes the common features of XSS attack samples, where the bold features in the sample are such features. These features occupy the first 15 bits of the state vector, and each bit is 0 or 1, indicating the absence or presence of the feature, respectively.

[0124] Table 2. XSS attack sample characteristics

[0125]

[0126]

[0127] Dynamic attack signature: changes with the dynamic change of the attack vector. The present invention sets 26 mutation actions, and different mutation actions correspond to different target modules. Therefore, the dynamic attack signature contains 26 bits, indicating whether the conditions for executing each action are met, and the value is 1 or 0, indicating satisfaction or non-satisfaction, respectively. Figure 4 The dynamic changes of the attack vector are shown, and the part of the selected mutation module in each iteration is highlighted.

[0128] Attack vector type: indicates six types of attack vectors. The value range of the corresponding state vector bit is the type of vulnerability output location that the attack vector matches, including: between HTML common tags, between HTML special tags, within HTML common attributes, within HTML special attributes, within JS code, and within CSS styles. The specific description of the corresponding attack vectors is shown in Table 3.

[0129] Table 3. Attack vector description table

[0130]

[0131]

[0132] Current action number: indicates the action number corresponding to the mutation strategy adopted in the current round. The value range is Attack result: refers to whether the attack vector successfully triggers the vulnerability. The value is 1 if successful and 0 if failed.

[0133] In summary, the state vector designed by the present invention contains a total of 44 bits, including 15 static attack features, 26 dynamic attack features, 1 attack vector type, 1 current action number and 1 attack result. Table 4 lists the features and value ranges corresponding to all state vector bits.

[0134] Table 4. Description of all state vectors

[0135]

[0136]

[0137]

[0138] In this embodiment, the reward function is a key part of the attack vector optimization process, helping the agent to evaluate and adjust the vector generation strategy in real time during the attack process. The reward function is set based on the feedback of the attack, for example, when the attack vector successfully bypasses the defense mechanism, it provides a positive reward, and when it fails, it makes adjustments to guide the agent to mutate the attack vector in the direction of a higher success rate.

[0139] In reinforcement learning, the agent always learns how to maximize the cumulative reward, which is set according to the following formula:

[0140]

[0141] Among them, r(s t ,a t ) is the reward at time t, γ is the discount factor, R t When designing a reward function, you need to ensure that the agent can maximize the reward and achieve the goal. In addition, the reward can only be used to convey the goal you want to achieve, not how to achieve it, so the reward setting should not be too detailed.

[0142] In order to solve the sparse reward problem, the present invention not only uses the attack result as a positive feedback reward signal, but also introduces an auxiliary reward based on edit distance. This reward guides the mutation process of the attack vector. The reward is calculated based on the effect of the current strategy. The specific method is to compare the original attack vector payload brfore Attack vector payload in the response page after The larger the edit distance (i.e., the lower the similarity), the smaller the reward assigned. This design reflects the negative correlation between similarity and reward: as the similarity decreases (i.e., the attack vector is significantly modified or filtered on the target website), the reward obtained by the model decreases, indicating that the current mutation strategy may be ineffective. Conversely, a higher similarity (close to 100%) indicates that the mutation strategy successfully bypasses the filtering mechanism, so a higher reward is given to encourage the continued use of the strategy.

[0143] The edit distance is calculated as:

[0144]

[0145] Where a represents the string of the original attack vector i, and b represents the string of the attack vector j in the response page.

[0146] The final reward function in this embodiment is specifically:

[0147]

[0148] Among them, r(s t ,a t ) represents the reward at time t, MAX_STEP represents the maximum number of attacks that can be attempted in each round, and aux rwd represents the auxiliary reward based on edit distance, L represents the length of the module involved in the mutation, lev (before,after) (i,j) represents the edit distance between the original attack vector i and the attack vector j in the response page.

[0149] S3. Perform attack verification on the response page after the attack vector generation model sends the attack request to the target page, and calculate the reward based on the verification result and feed it back to the attack vector generation model.

[0150] In an optional embodiment of the present invention, attack verification plays a key role in the final testing and confirmation of the generated attack vector. Specifically, attack verification confirms the effect of the attack by executing the generated attack vector at the injection point and analyzing the system response page. The verification mechanism uses a method based on semantic analysis, which can accurately detect whether the XSS attack has successfully invaded the target system. If the attack verification confirms that the XSS vulnerability is successfully triggered, it means that the injection point and attack vector are valid. This result will be fed back to the previous module to optimize and improve the generation effect and accuracy of the model. In addition, if the verification fails, the system adjusts the strategy based on the feedback, regenerates and tests the new attack vector to achieve a more efficient automated attack cycle. Therefore, attack verification ensures the practicality and reliability of the automated attack system throughout the entire process.

[0151] The main responsibility of the attack verification module is to analyze the response page after AVGenerate sends an attack request to the target page, and to determine the attack result by analyzing the injection of the attack vector in the DOM syntax tree structure of the response page. Based on the attack result, the module calculates the reward and feeds back the reward value and environment status to AVGenerate. The specific process of this module is as follows Figure 5 shown.

[0152] To facilitate attack verification, the present invention sets the attack code of all attack vectors as a pop-up window function. Using a pop-up window function for attack verification has a key advantage: the pop-up window event is an intuitive and clear indication of successful code execution, which is very suitable for automated verification process. The specific steps are as follows:

[0153] Monitoring pop-up window events: The attack verification module first uses WebDriver to monitor whether a pop-up window event occurs. If a pop-up window event occurs, it indicates that the attack is successful.

[0154] Traversing the DOM syntax tree: If there is no pop-up event, the module will search for the pop-up function by traversing the DOM syntax tree of the page, because in some cases, although the injected attack code is successful, it will not trigger the pop-up immediately, but requires a specific event to trigger. Therefore, if the pop-up function is found in the DOM tree, it means that the attack code has been successfully injected into the page structure, ensuring that the attack code can be successfully executed.

[0155] Attack result determination: If the code injection fails, the attack is determined to have failed.

[0156] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0159] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0160] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. An XSS automated testing method based on curiosity-driven and deep reinforcement learning, characterized in that: The following steps are involved: The input environment for injection point exploration is constructed based on the structure, content, and interactive response information of the web page. The curiosity-driven reinforcement learning method is used to construct an intelligent agent with an intrinsic reward mechanism. A state flow graph is constructed to record and manage the state flow path visited by the intelligent agent during the exploration process to obtain injection point information. Pass the injection point information to the intelligent agent in the attack vector generation model, and construct the state space, action space and reward function in the attack vector generation model; The attack vector generation model is used to verify the response page after sending the attack request to the target page, and the reward is calculated based on the verification result and fed back to the attack vector generation model.

2. According to claim 1, a curiosity-driven and deep reinforcement learning-based XSS automated testing method is characterized in that: When constructing the input environment for injection point exploration based on the structure, content, and interactive response information of the web page, it includes: The normalized compression distance is used to calculate the DOM structure similarity of two pages, and the page logic is deduplicated based on the DOM structure similarity.

3. According to the XSS automated testing method based on curiosity-driven and deep reinforcement learning according to claim 1, it is characterized in that: When using curiosity-driven reinforcement learning methods to build an intelligent agent with an intrinsic reward mechanism, the intrinsic reward function is specifically: Among them, R represents the intrinsic reward function, num ip Indicates the number of points, num s Indicates the number of visits to the set state; r max Indicates that the state explored by the agent is a new state and the page contains a data injection point, giving the maximum reward value; r nor Indicates the generally high reward given when a new state is explored but does not contain a data injection point; r neg Represents negative reward, which is used to punish the agent for exploring invalid states; s represents state; The Q value is updated as follows: Among them, Q(s,a) represents the initial Q value, α represents the learning rate, γ represents the decay factor, and Q(s',a') represents the updated Q value when reaching the new state s' by executing action a.

4. According to claim 1, a curiosity-driven and deep reinforcement learning-based XSS automated testing method is characterized in that: When constructing a state flow graph to record and manage the state transition paths visited by the agent during the exploration process, the state flow graph is updated as follows: Where count(s,a) represents the number of times action a is executed in state s, curiosity(s,a) represents the curiosity value obtained after executing action a in a single state s, and decays with the increase in the number of times action a is executed; A s represents the action set of state s, a t Indicates state s i action sequence, curiosity(s) represents the action set A s The average curiosity value of each action obtained after all actions are executed.

5. According to claim 1, a curiosity-driven and deep reinforcement learning-based XSS automated testing method is characterized in that: When passing the injection point information to the intelligent agent in the attack vector generation model, it includes: The objective function for establishing a deep reinforcement learning method is: Among them, π * represents the objective function, argmax represents the maximum value function, E represents the expectation, ρ π represents the distribution of actions taken in the current state under strategy π, s t represents the state of the environment at time t, a t represents the action selected by the agent at time t, r(s t ,a t ) means in state s t Next select action a t The reward of environmental feedback, α represents the weight coefficient, H represents entropy, π(a|s t ) means in state s t The probability of taking action a under The action-value function for building a deep reinforcement learning method is: Among them, Q π (s,a) represents the action value function, γ represents the discount factor; The state value function for establishing the deep reinforcement learning method is: Among them, V π (s) represents the state value function; The iteration method of the action value function is: Among them, p(s'|s,a) represents the state transition probability distribution of the environment, Q π (s',a') represents the action-value function before updating; The iteration method of the state value function is: The strategy optimization method of deep reinforcement learning method is: Among them, π new represents the optimized strategy, arg min represents the minimum function, π represents the strategy, Π represents the strategy set, D KL represents the distance between the policy and the energy-based model, Z represents the normalization function, and π old Indicates the original policy.

6. According to claim 1, a curiosity-driven and deep reinforcement learning-based XSS automated testing method is characterized in that: When constructing the state space in the attack vector generation model, we include: A modular design approach is adopted, where an action is represented as a single modification of one of the modules of the XSS attack vector, so that each mutation operation performed on the attack vector is directly associated with the reward feedback provided by the environment.

7. According to claim 1, a XSS automated testing method based on curiosity-driven and deep reinforcement learning is characterized in that: When constructing the action space in the attack vector generation model, we include: The state is modeled as a state feature vector, where the state feature vector includes static attack features, dynamic attack features, attack vector type, current action number features, and attack result features.

8. According to claim 1, a curiosity-driven and deep reinforcement learning-based XSS automated testing method is characterized in that: The reward function in constructing the attack vector generation model is as follows: Among them, r(s t ,a t ) represents the reward at time t, MAX_STEP represents the maximum number of attacks that can be attempted in each round, and aux rwd represents the auxiliary reward based on edit distance, L represents the length of the module involved in the mutation, lev (before,after) (i,j) represents the edit distance between the original attack vector i and the attack vector j in the response page.

9. According to claim 8, a curiosity-driven and deep reinforcement learning-based XSS automated testing method is characterized in that: The edit distance is calculated as: Where a represents the string of the original attack vector i, and b represents the string of the attack vector j in the response page.

10. According to claim 1, a XSS automated testing method based on curiosity-driven and deep reinforcement learning is characterized in that: When performing attack verification on the response page after the attack vector generation model sends an attack request to the target page, it includes: Set the attack code of all attack vectors to a pop-up window function; Monitor whether a pop-up window event occurs; if so, the attack is considered successful; otherwise, proceed to the next step; Traverse the DOM syntax tree of the page to search for the pop-up function; if the search is successful, the attack is considered successful; otherwise, the attack is considered failed.

Images