Attack optimization method and device of confrontation system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an attack optimization method and device for a confrontation system, electronic equipment and a storage medium. The method comprises the steps of obtaining system environment data in the confrontation system; inputting the system environment data into an attack generator for data processing to obtain an initial attack strategy, and obtaining change environment data of the confrontation system after the action based on the initial attack strategy; inputting the change environment data into an attack discriminator for data processing to obtain an initial attack discrimination result corresponding to the initial attack strategy; obtaining an action state value according to the initial attack judgment result and the system environment data, obtaining an initial action reward based on the initial attack judgment result and a reward model, and performing model updating on the attack generator according to the initial action reward and the action state value, and an optimized attack strategy corresponding to the change environment data is generated by using the updated attack generator, so that the virtual attack with strong pertinence can be better and adaptively generated in a complex dynamic network environment.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an attack optimization method, device, electronic device and storage medium for a countermeasure system. Background Art

[0002] With the continuous development of information technology and the increasing number of network security threats, attack simulation technology has become a core link in network attack and defense experiments and security protection research. Attack simulation technology aims to simulate various network attack scenarios to evaluate the effectiveness and potential vulnerabilities of existing defense measures. Therefore, in the face of complex virtual network environments, generating more accurate and efficient attacks can better improve the accuracy and reliability of defense against various precision attacks.

[0003] In related technologies, to generate appropriate simulated attacks in a virtual network environment, corresponding attack rules are usually formulated in advance according to different virtual network environments, so as to match the real-time virtual network with the attack rules, thereby determining and generating corresponding simulated attacks to perform attack operations in the virtual network environment. However, since rule-driven attack simulation can only simulate known vulnerabilities and attack patterns, it is difficult to cope with complex and changeable network environments, and thus it is impossible to generate highly targeted virtual attacks. Summary of the invention

[0004] The embodiments of the present application provide an attack optimization method, device, electronic device and storage medium for a countermeasure system, which can improve the accuracy and targeting of virtual attacks generated in a complex virtual network environment that changes in real time.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes an attack optimization method for a countermeasure system, the method comprising: Acquiring system environment data in the confrontation system; Inputting the system environment data into the attack generator for data processing to obtain an initial attack strategy, and obtaining the changed environment data of the adversarial system after the initial attack strategy is applied; Inputting the changed environment data into an attack discriminator for data processing to obtain an initial attack discrimination result corresponding to the initial attack strategy; An action state value is obtained according to the initial attack discrimination result and the system environment data, an initial action reward is obtained based on the initial attack discrimination result and a reward model, a model of the attack generator is updated according to the initial action reward and the action state value, and an optimized attack strategy corresponding to the changed environment data is generated using the updated attack generator.

[0006] In some embodiments, the attack generator includes an attack optimization model, and the steps of constructing the attack optimization model include: Generate a state space based on the defense state information of the adversarial system, the attack characteristics of the attack generator, and the changed environmental parameters of the corresponding system after being attacked; Generate an action space based on the attack type, attack intensity, and attack timing of the attack generator; generating an attack optimization function based on the changing environmental parameters, and generating a reward model based on the attack optimization function; Initial decision parameters of the value decision network are obtained, and the attack optimization model is generated based on the initial decision parameters, the state space, the action space, and the reward model.

[0007] In some embodiments, generating an attack optimization function based on the changing environmental parameters includes: Obtaining an attack success parameter according to the changed environmental parameter, and generating an attack success reward function; Generating an attack concealment reward function according to the attack concealment parameter obtained from the changing environment parameter; Generating an attack and destruction reward function according to the attack and destruction parameters obtained from the changing environmental parameters; generating an attack failure penalty function according to the attack failure parameters obtained from the changing environment parameters; The attack success reward function, the attack concealment reward function, and the attack destruction reward function are accumulated, and the attack failure penalty function is subtracted to obtain the attack optimization function.

[0008] In some embodiments, obtaining an action state value according to the initial attack discrimination result and the system environment data, obtaining an initial action reward based on the initial attack discrimination result and a reward model, and updating the attack generator model according to the initial action reward and the action state value include: Obtaining the action state value corresponding to the initial attack discrimination result and the system environment data in the value decision network, and obtaining the maximum predicted action state value corresponding to the changed environment data in the value decision network; Obtaining an initial action reward based on the initial attack discrimination result and the reward model; The initial attack discrimination result and the initial decision parameter corresponding to the system environment data are updated based on the initial action reward, the action state value and the maximum predicted action state value.

[0009] In some embodiments, the inputting the changed environment data into an attack discriminator for data processing to obtain an initial attack discrimination result corresponding to the initial attack strategy includes: In the attack discriminator, the change environment data is formatted to obtain sequence change environment data; Performing attention processing and feedforward processing on the sequence-changing environmental data in sequence to obtain environmental processing data; Performing result classification and discrimination processing on the environmental processing data to obtain a successful attack result corresponding to the initial attack strategy; Performing result regression and discrimination processing on the environmental processing data to obtain an attack performance prediction result corresponding to the initial attack strategy; The initial attack identification result is obtained based on the attack success result and the attack performance prediction result.

[0010] In some embodiments, after obtaining the initial attack identification result based on the attack success result and the attack performance prediction result, the method further includes: Using a cross entropy function, based on the true label corresponding to the initial attack strategy and the attack performance prediction result, a classification loss value is calculated; Using a mean square error function, based on the true label and the attack performance prediction result, a regression loss value is calculated; The attack discriminator is updated based on the classification loss value and the regression loss value.

[0011] In some embodiments, obtaining system environment data in the confrontation system includes: Acquire current initial system environment data in the adversarial system, wherein the initial system environment data includes network topology information, vulnerability scanning information, and traffic pattern information; The initial system environment data is sequentially cleaned, normalized, and characterized to obtain system environment characteristic data; Performing environment conversion processing on the system environment feature data to obtain the system environment data that matches the input data of the attack generator.

[0012] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an attack optimization device for a confrontation system, the device comprising: An acquisition module, used for acquiring system environment data in the confrontation system; An attack generation module, used for inputting the system environment data into an attack generator for data processing, obtaining an initial attack strategy, and obtaining the changed environment data of the adversarial system after the initial attack strategy acts; An attack discrimination module, used for inputting the changed environment data into an attack discriminator for data processing to obtain an initial attack discrimination result corresponding to the initial attack strategy; An attack optimization module is used to obtain an action state value according to the initial attack discrimination result and the system environment data, obtain an initial action reward based on the initial attack discrimination result and a reward model, update the attack generator model according to the initial action reward and the action state value, and use the updated attack generator to generate an optimized attack strategy corresponding to the changed environment data.

[0013] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the attack optimization method of the countermeasure system as described in the first aspect.

[0014] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, the attack optimization method of the adversarial system described in the first aspect is implemented.

[0015] The attack optimization method, device, electronic device and storage medium of the adversarial system proposed in the embodiments of the present application include: first, obtaining system environment data in the adversarial system; then, inputting the system environment data into an attack generator for data processing to obtain an initial attack strategy, and obtaining the changed environment data of the adversarial system after the initial attack strategy is applied; next, inputting the changed environment data into an attack discriminator for data processing to obtain an initial attack discrimination result corresponding to the initial attack strategy; finally, obtaining an action state value based on the initial attack discrimination result and the system environment data, obtaining an initial action reward based on the initial attack discrimination result and a reward model, updating the attack generator model based on the initial action reward and the action state value, and using the updated attack generator to generate an optimized attack strategy corresponding to the changed environment data. The embodiment of the present application combines adversarial network technology and reinforcement learning technology, uses an attack generator combined with reinforcement learning technology to generate an initial attack strategy corresponding to the real-time system environment data of the adversarial system, and uses an attack discriminator to evaluate and discriminate the changed environment data of the adversarial system after the initial attack strategy acts on it to obtain an initial attack discrimination result, and then uses the initial attack discrimination result to optimize and improve the attack generator, so as to dynamically adapt to changes in the target system, generate diversified attack strategies, and continuously improve the effectiveness and concealment of the attack through a closed-loop optimization mechanism, so as to overcome the problems of lack of environmental adaptability, single attack method and incomplete evaluation mechanism of existing attack generation technology, and provide a more efficient, intelligent and dynamically optimized network attack and defense confrontation model, which can better adaptively generate highly targeted virtual attacks in a complex dynamic network environment.

[0016] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of an attack optimization method for a countermeasure system provided in one embodiment of the present application.

[0018] Figure 2 yes Figure 1 Flow chart of step 101 in FIG.

[0019] Figure 3 It is a flow chart for constructing an attack optimization model provided in one embodiment of the present application.

[0020] Figure 4 yes Figure 3 Flow chart of step 303 in FIG.

[0021] Figure 5 yes Figure 1 Flow chart of step 103 in FIG.

[0022] Figure 6 This is a flowchart of updating an attack discriminator provided by yet another embodiment of the present application.

[0023] Figure 7 yes Figure 1 Flow chart of step 104 in FIG.

[0024] Figure 8 It is a schematic flowchart of an attack optimization process for a countermeasure system provided by another embodiment of the present application.

[0025] Fig. 9 It is a structural diagram of an attack optimization device for a countermeasure system provided in one embodiment of the present application.

[0026] Fig.10 It is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0030] With the continuous development of information technology and the increasing number of network security threats, attack simulation technology has become a core link in network attack and defense experiments and security protection research. Attack simulation technology aims to simulate various network attack scenarios to evaluate the effectiveness and potential vulnerabilities of existing defense measures. Therefore, in the face of complex virtual network environments, generating more accurate and efficient attacks can better improve the accuracy and reliability of defense against various precision attacks.

[0031] In related technologies, to generate appropriate simulated attacks in a virtual network environment, corresponding attack rules are usually formulated in advance according to different virtual network environments, so as to match the real-time virtual network with the attack rules, thereby determining and generating corresponding simulated attacks to perform attack operations in the virtual network environment. However, since rule-driven attack simulation can only simulate known vulnerabilities and attack patterns, it is difficult to cope with complex and changeable network environments, and thus it is impossible to generate highly targeted virtual attacks.

[0032] In order to improve the accuracy and targeting of virtual attacks generated in a complex virtual network environment that changes in real time, the embodiment of the present application combines adversarial network technology and reinforcement learning technology, uses an attack generator combined with reinforcement learning technology to generate an initial attack strategy corresponding to the real-time system environment data of the adversarial system, and uses an attack discriminator to evaluate and discriminate the changing environment data of the adversarial system after the initial attack strategy acts on it to obtain an initial attack discrimination result, and then uses the initial attack discrimination result to optimize and improve the attack generator, so as to dynamically adapt to changes in the target system, generate diversified attack strategies, and continuously improve the effectiveness and concealment of the attack through a closed-loop optimization mechanism, so as to overcome the problems of lack of environmental adaptability, single attack method and incomplete evaluation mechanism of existing attack generation technology, and provide a more efficient, intelligent and dynamically optimized network attack and defense confrontation model, which can better adaptively generate highly targeted virtual attacks in a complex dynamic network environment.

[0033] The following will first describe in detail the attack optimization method of the confrontation system in the embodiment of the present application. Figure 1 , which is an optional flow chart of the attack optimization method for the countermeasure system provided in the embodiment of the present application, Figure 1 The method in the embodiment may include but is not limited to steps 101 to 104. Figure 1 The order of steps 101 to 104 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs. The attack optimization method of the confrontation system provided in the embodiment of the present application can be applied to a server provided with a virtual confrontation system, or it can be applied to any server or intelligent terminal connected to the confrontation system.

[0034] Step 101: Obtain system environment data in the adversarial system.

[0035] The following is a detailed description of step 101.

[0036] In some embodiments, when responding to the generation of a virtual attack in a virtual confrontation system, it is first necessary to use an environment perception module to obtain system environment data in the confrontation system, so as to facilitate the subsequent use of the system environment data to generate a highly targeted virtual attack.

[0037] The main task of the environment perception module is to collect and process the environmental data of the target system and generate an environmental feature vector that can be used by the attack recommendation generator. These features include the network topology, vulnerability information, traffic patterns, etc. of the target system, which can provide effective condition information for subsequent attack generation. The functions of the environment perception module mainly include: target system scanning and data collection, data preprocessing and feature extraction, and environmental feature vector generation. Its core goal is to enable the attack generator to generate more accurate and effective attack methods based on actual conditions through real-time perception of the target system.

[0038] The following will further describe how to use the environment perception module to obtain system environment data in the adversarial system.

[0039] Reference Figure 2 , obtaining system environment data in the adversarial system, including the following steps 201 to 203.

[0040] Step 201: Acquire the current initial system environment data in the adversarial system.

[0041] Step 202: The initial system environment data is sequentially cleaned, normalized, and feature screened to obtain system environment feature data.

[0042] Step 203: Performing environment conversion processing on the system environment feature data to obtain system environment data that matches the input data of the attack generator.

[0043] Steps 201 to 203 are described in detail below.

[0044] In some embodiments, the environment perception module first obtains the current initial system environment data in the adversarial system, where the initial system environment data includes network topology information, vulnerability scanning information, and traffic pattern information.

[0045] Among them, the purpose of the network topology scan corresponding to obtaining the network topology information is to draw the topology of the target network and identify the connection relationship between the nodes in the network and the type of nodes. Tools such as Nmap can be used to scan the IP address range, port status, etc. of the target system. Topology scanning provides the network infrastructure information for subsequent attack generation, helping the generator understand the attack path and potential attack surface. The representation of network topology usually adopts a graph structure, in which nodes represent network devices (such as routers, switches, hosts, etc.) and edges represent the connection relationship between devices. The network topology graph can be represented as a graph G=(V, E), where, , , represents the i-th node in the network; For Node The connecting edges between .

[0046] The vulnerability scan corresponding to the vulnerability scan information is to identify potential vulnerabilities in the target system through tools (such as Nessus, OpenVAS, etc.). The vulnerability scanner will conduct a comprehensive scan of the target system's operating system, applications, and services to find known vulnerabilities, unpatched security vulnerabilities, etc. Vulnerability information is one of the core data for generating attack methods, which directly affects which attack strategies the attack generator chooses. Vulnerability risk assessment is usually based on the severity and exposure of the vulnerability. The commonly used assessment method is CVSS (Common Vulnerability Scoring System). The CVSS calculation formula is shown in the following formula (1).

[0047] (1) Among them, AV represents the attack complexity of the vulnerability, AC represents the access control requirements of the attack, and Au represents the authentication requirements of the vulnerability.

[0048] The traffic analysis corresponding to the traffic pattern information is to collect the network traffic data of the target system and analyze its communication mode, bandwidth usage, source / destination address of the traffic, and other information. These traffic characteristics provide an important reference for generating attack paths and bypassing defenses. Traffic analysis is usually performed using tools such as Wireshark and Tcpdump. The traffic characteristics used in the embodiments of the present application mainly include: packet size, traffic protocol, source and destination IP, transmission timestamp, etc. The traffic characteristics are represented by the time series model as follows: , Indicates the size of the packet. Indicates the source address and destination address, Indicates a timestamp.

[0049] Next, the initial system environment data is cleaned, normalized, and feature-screened in turn to obtain system environment feature data, so as to facilitate conversion into a data format that the attack generator can understand and process.

[0050] Data cleaning is used to remove redundant, inconsistent or missing data. For example, in traffic analysis, some abnormal and incomplete packets may be collected. These data need to be removed to avoid affecting subsequent feature extraction and attack generation.

[0051] For normalization, different data sources may have different dimensions (e.g., the unit of traffic is bytes, and the vulnerability score is from 0 to 10), so the features need to be normalized or standardized. The normalized features can be unified to the range of [0, 1] to ensure that different features are processed at the same scale. The normalization formula is shown in the following formula (2).

[0052] (2) Among them, X is the original eigenvalue, Xmin and Xmax are the minimum and maximum values ​​of the eigenvalue respectively.

[0053] Regarding feature screening, there may be some redundant or irrelevant features in large-scale environmental data. Through feature selection methods (such as principal component analysis (PCA), correlation analysis, etc.), features that have a greater impact on attack generation can be screened out, thereby reducing computational complexity and improving the efficiency of the generator.

[0054] After that, the system environment feature data is transformed into the environment to obtain the system environment data that matches the input data of the attack generator. That is, after preprocessing and feature extraction, the environment perception module finally converts the extracted features into an environment feature vector as the input of the attack generator. The environment feature vector is a high-dimensional vector that contains the network structure, vulnerability information, traffic characteristics, etc. of the target system. The vector should reflect the current state of the target system as comprehensively and accurately as possible.

[0055] Assume that the feature vector of the target system is represented as E=[E1, E2, …, Ei, …, En], where each Ei is a different environmental feature, such as network topology, vulnerability score, traffic pattern, etc. In the next step, the attack recommendation generator will generate an attack method based on the feature vector.

[0056] Through the above steps 201 to 203, the environment perception module is used to accurately obtain the current network topology information, vulnerability scanning information and traffic pattern information in the adversarial system, and then cleans, normalizes, features screens and environmental conversion processes, thereby ensuring the accuracy, consistency and efficiency of the obtained system environment data, providing high-quality input data for the attack generator, thereby facilitating the subsequent attack strategy generation by the attack generator, improving the pertinence and effectiveness of the attack strategy, and optimizing the utilization of computing resources, making the attack generation process more intelligent and efficient, thereby improving the reliability and accuracy of attack optimization.

[0057] Step 102: Input the system environment data into the attack generator for data processing to obtain an initial attack strategy, and obtain the changed environment data of the adversarial system after the initial attack strategy is applied.

[0058] The following is a detailed description of step 102.

[0059] In some embodiments, after obtaining the current accurate system environment data of the adversarial system, the attack generator is used to process the system environment data to obtain an initial attack strategy corresponding to the system environment data.

[0060] The attack generator is responsible for generating targeted attack strategies based on the environmental feature vectors provided by the environmental perception module. To achieve this goal, the embodiment of the present application adopts a conditional generative adversarial network (CGAN) and combines the idea of ​​reinforcement learning (RL) to optimize the generated attack method, so that it can not only adapt to the current status of the target system, but also self-optimize in the process of playing with the defense system. Through the architecture of CGAN, the attack generator can generate attack methods that conform to the system environment data, and continuously adjust and optimize the attack strategy based on the feedback of the attack discriminator. Reinforcement learning plays a role in further improving the effectiveness of the attack method in this process. The attack generator can obtain rewards or penalties based on the success or failure of the attack, which drives the attack generator to adaptively improve the success rate of the attack.

[0061] The input of the attack generator is the environmental feature vector c and the noise vector z from the environmental perception module, where c is the system environmental data of the target system and z is random noise used to introduce diversity. The environmental feature vector contains the network topology, vulnerability information, traffic pattern, etc. of the adversarial system, while the noise vector ensures that the generated attack has a certain degree of randomness and innovation. The calculation formula is: , where c1, c2, …, cn are different environmental characteristics, such as vulnerability risks of adversarial systems, traffic characteristics, network topology, etc.

[0062] During the attack optimization process, the attack generator generates an attack method by learning the input environment feature vector c and noise z. The output of the generator is G(z|c), which is the initial attack strategy generated for the system environment data. The attack generator is trained through a multi-layer neural network, and its structure can be a fully connected layer, a convolutional layer, an LSTM, etc., which is selected according to the task requirements.

[0063] The goal of the attack generator is to generate an attack method that is as realistic as possible so that the attack discriminator can "believe" that the attack is effective and it is difficult to distinguish between true and false.

[0064] The output of the attack generator is an attack chain that contains multiple attack steps, including: attack target selection, selecting the system components or services that need to be attacked; attack path planning, planning the specific steps of the attack, including the order of exploiting vulnerabilities, possible strategies for bypassing defenses, etc.; attack tools and techniques. Based on the characteristics of the target system, the generator will recommend applicable attack tools or techniques, such as DDoS attacks, SQL injections, XSS, etc.

[0065] In addition, an attack optimization model generated by a reinforcement learning framework is also provided in the attack generator, so that a more targeted attack strategy can be generated by using the attack optimization model. The following will further describe how to construct the attack optimization model.

[0066] Reference Figure 3 , the steps of constructing the attack optimization model include the following steps 301 to 304.

[0067] Step 301: Generate a state space based on the defense state information of the adversarial system, the attack characteristics of the attack generator, and the changed environmental parameters of the corresponding system after being attacked.

[0068] Step 302: Generate an action space based on the attack type, attack intensity, and attack timing of the attack generator.

[0069] Step 303: Generate an attack optimization function based on the changing environmental parameters, and generate a reward model based on the attack optimization function.

[0070] Steps 301 to 303 are described in detail below.

[0071] In some embodiments, the reinforcement learning optimization module is designed to optimize the generated attack method through the reinforcement learning algorithm. The generator and the discriminator improve the performance of the attack method through mutual competition and alternating training, and the reinforcement learning optimization module provides an efficient learning mechanism, which enables the generator to self-adjust according to feedback signals in a complex environment, optimize the attack strategy, and improve the success rate, concealment and adaptability of the attack.

[0072] First, the state space is generated based on the defense state information represented in the system environment data of the adversarial system, the attack characteristics of the last attack generator, and the changed environmental parameters of the corresponding system after being attacked.

[0073] The state space represents the environmental information perceived by the agent (attack generator), which describes the current state of the system. In the embodiment of the present application, the state space can be described by the following dimensions.

[0074] (1) The defense status of the target system, including the current defense strategy of the target system, the type of defense mechanism, the configuration status of the defense, the detection mechanism, vulnerability information, network configuration, etc. This information helps the generator understand the current defense environment so that it can choose the appropriate attack method. Among them, the type of defense mechanism includes IDS (intrusion detection system), IPS (intrusion protection system), firewall type, etc.; the configuration status of the defense includes rule set, policy configuration, traffic filtering standard, etc.

[0075] (2) Characteristics of the current attack method, including the attack type, attack intensity, attack timing, etc. selected by the attack generator. The characteristics of the attack method help the generator determine whether the current attack meets the defense conditions of the target system; attack type: such as DoS (denial of service attack), SQL injection, cross-site scripting attack (XSS), buffer overflow, etc.; and attack intensity: such as traffic size, attack duration, etc.

[0076] (3) Environmental feedback information, including the discriminator’s evaluation results on the attack effect, such as attack success rate, concealment, destructiveness and other indicators; success rate: whether the attack has broken through the target defense system; concealment: whether the attack has successfully bypassed the detection of the defense system; destructiveness: the degree of damage caused by the attack, such as system crash, data leakage, etc.

[0077] In summary, the state space St is composed of several parts: St = (defense state, attack characteristics, environmental feedback). This state space will be converted into a form suitable for inputting deep learning models for the training of reinforcement learning algorithms.

[0078] Next, an action space is generated based on the attack type, attack intensity, and attack timing of the attack generator. The action space defines all possible attack methods that the generator can choose. Each action represents a choice of attack strategy, and the generator selects an action to execute based on the input of the current state space. In order to simplify the processing, the embodiment of the present application divides the action space into the following categories.

[0079] (1) Attack type selection, such as DoS attack, SQL injection, XSS, etc. Different attack types target different defense mechanisms. Therefore, the attack type is an important factor affecting the success rate of the attack. The attack types in the action space generally include: DoS, SQL Injection, XSS, etc.

[0080] (2) Attack intensity selection: According to the defense strength and defense strategy of the target system, the generator can select different attack intensities, such as low intensity, medium intensity, and high intensity. The attack intensity in the action space generally includes: Low, Medium, and High.

[0081] (3) The timing of the attack can affect the stealth and success rate of the attack. For example, you can choose to launch an attack when the defense system is restarted or the system load is high. The attack timing in the action space generally includes: Early, Mid, and Late.

[0082] Therefore, the action space At includes the combination of these choices: At=(attack type, attack intensity, attack timing), and the size of the action space depends on different combinations of attack type, intensity, and timing.

[0083] Afterwards, based on the changed environmental parameters generated in the confrontation system after being acted upon by the virtual attack, a suitable attack optimization function is generated, so as to facilitate the subsequent generation of a reward model based on the attack optimization function. The following will further describe how to generate the attack optimization function.

[0084] Reference Figure 4, generating an attack optimization function based on the changing environment parameters, including the following steps 401 to 405.

[0085] Step 401: Obtain attack success parameters according to the changing environment parameters and generate an attack success reward function.

[0086] Step 402: Generate an attack concealment reward function based on the attack concealment parameters obtained by changing the environmental parameters.

[0087] Step 403: Generate an attack and destruction reward function based on the attack and destruction parameters obtained by changing the environmental parameters.

[0088] Step 404: Generate an attack failure penalty function based on the attack failure parameters obtained by changing the environmental parameters.

[0089] Step 405: Accumulate the attack success reward function, the attack concealment reward function and the attack destruction reward function, and subtract the attack failure penalty function to obtain the attack optimization function.

[0090] Steps 401 to 405 are described in detail below.

[0091] The reward function plays a crucial role in reinforcement learning and determines the feedback of the generator after executing each action. By designing a reasonable reward function, the generator can be guided to optimize towards a better attack strategy. Therefore, it is necessary to set a suitable attack optimization function. In the attack generation task of the embodiment of the present application, the attack optimization function should consider the following factors.

[0092] (1) Reward for successful attack: The attack success parameter is obtained based on the changing environment parameters, and the attack success reward function is generated. That is, if the attack successfully breaks through the defense of the target system, the system will give the generator a positive reward. For example, successfully destroying the target system or leaking sensitive data. That is, rsuccess = +1 (attack successful) (2) Reward for attack stealth. Based on the attack stealth parameters obtained by changing the environmental parameters, the attack stealth reward function is generated. That is, if the attack can successfully bypass the monitoring mechanism of the target system, such as IDS, IPS, etc., the generator will receive a stealth reward. The stronger the attack stealth, the higher the reward. That is, if the attack is not detected, the reward is positive: rstealth=+1 (the attack is highly stealthy); if the attack is detected, the reward is negative: rstealth=−1 (the attack is detected).

[0093] (3) Reward for attack damage: Based on the attack damage parameters obtained by changing the environmental parameters, the attack damage reward function is generated. That is, if the attack successfully causes greater damage, such as system crash, data leakage, etc., the generator will receive a higher reward. That is, rdamage=+2 (the attack causes greater damage), rdamage=+1 (the attack causes certain damage).

[0094] (4) Penalty for attack failure: Generate an attack failure penalty function based on the attack failure parameters obtained by changing the environmental parameters. That is, if the attack fails or the defense system successfully intercepts the attack, the generator will be penalized. That is, rfailure = −1 (attack failure) Therefore, the attack success reward function, the attack concealment reward function and the attack destruction reward function are accumulated, and the attack failure penalty function is subtracted to obtain the attack optimization function. Further, based on the attack optimization function, the reward function rt can be expressed as shown in the following formula (3).

[0095] (3) Among them, α, β, γ, and δ are hyperparameters adjusted according to the requirements of specific tasks to balance the importance of different reward items.

[0096] Through the above steps 401 to 405, by generating reward functions for attack success, concealment, destructiveness and failure penalty according to the changing environmental parameters, and accumulating these functions to obtain the attack optimization function, a multi-dimensional evaluation and optimization of the attack strategy is achieved. The design of this multi-reward mechanism enables the attack generator to not only focus on the success rate of the attack, but also comprehensively consider the concealment and destructiveness of the attack, ensuring that the generated attack strategy can effectively bypass the detection mechanism and cause greater damage while breaking through the target system defense. At the same time, the introduction of the attack failure penalty function avoids the generation of invalid or inefficient attack strategies, and improves the efficiency and intelligence of the attack generator. Through this multi-dimensional reward mechanism, the attack generator can adaptively generate the optimal attack strategy in a complex network environment, significantly improving the success rate, concealment and destructiveness of the attack, thereby effectively improving the reliability and accuracy of the attack optimization.

[0097] Step 304: Obtain initial decision parameters of the value decision network, and generate an attack optimization model based on the initial decision parameters, state space, action space, and reward model.

[0098] Step 304 is described in detail below.

[0099] In some embodiments, in order to process high-dimensional state space and action space, the present application embodiment selects the deep Q network (DQN) as the algorithm of the reinforcement learning optimization module. DQN is an extension of the traditional Q-learning algorithm, which uses a deep neural network to approximate the Q value function, so that it can process complex input data.

[0100] The core idea of ​​DQN is to estimate the Q value of each state-action (s, a) pair through a deep neural network. The Q value represents the cumulative reward that can be obtained by performing action a in state s. The update process of DQN is shown in the following formula (4).

[0101] (4) in, is the Q value corresponding to the current state st and action at, is the learning rate, which controls how much weight the model places on new information (i.e., the difference between the target value and the current estimate) at each update; It is the immediate reward obtained based on the reward model after executing action at; is a discount factor used to weigh the relationship between immediate rewards and future rewards; is in the next state The maximum Q value of all possible actions under the given condition represents the optimal action chosen by the agent. DQN continuously updates the Q value in this way to find the optimal strategy.

[0102] Therefore, when constructing an attack optimization model, it is necessary to first generate an attack optimization model based on the initial decision parameters (i.e., the Q values ​​corresponding to different actions and different states) of the value decision network (i.e., the above-mentioned deep learning network DQN) and then based on the initial decision parameters, state space, action space, and reward model. Later, in the process of attack optimization of the adversarial system, the generated actions are used to act on the adversarial system to obtain the changed environmental data, combined with the reward feedback obtained by the reward model, to update and improve the initial decision parameters of the value decision network.

[0103] Through the above steps 301 to 304, the idea of ​​reinforcement learning is used to generate state space, action space, attack optimization function and reward model, and build an attack optimization model in combination with initial decision parameters to achieve intelligent optimization of attack strategy. The generation of state space is based on the defense state, attack characteristics and changing environmental parameters of the adversarial system, ensuring that the attack generator can fully perceive the real-time state of the target system; the generation of action space is based on the attack type, attack intensity and attack timing, providing a variety of attack strategy options; the design of attack optimization function and reward model comprehensively considers the success, concealment and destructiveness of the attack, ensuring that the generated attack strategy is not only efficient but also targeted. Finally, by combining initial decision parameters, state space, action space and reward model, the attack optimization model can adaptively generate the optimal attack strategy in a complex network environment, significantly improving the success rate, concealment and destructiveness of the attack, and realizing the efficiency and intelligence of network attack and defense confrontation.

[0104] In actual attack optimization, after obtaining the current system environment data of the adversarial system, the attack generator uses the attack optimization model obtained by the reinforcement learning algorithm to generate an initial attack strategy for the system environment data. Then, the initial attack strategy is used in the adversarial system, and then the environment perception module is used to obtain the changed environment data of the adversarial system after the initial attack strategy is applied.

[0105] Step 103: Input the changed environment data into the attack discriminator for data processing to obtain an initial attack discrimination result corresponding to the initial attack strategy.

[0106] The following is a detailed description of step 103.

[0107] In some embodiments, after obtaining the changed environmental data of the adversarial system after being affected by the initial attack strategy, the changed environmental data is input into the attack discriminator in the adversarial network (CGAN) for data processing and analysis to obtain an initial attack discrimination result corresponding to the attack effect of the initial attack strategy.

[0108] The attack discriminator is responsible for evaluating the attack method output by the generator to ensure that the generated attack strategy can effectively break through the defense of the target system and achieve the expected effect. Unlike the traditional discriminator design, the present invention will use the Transformer model to realize the evaluation of the attack effect. The Transformer model is based on the self-attention mechanism and has the ability to efficiently process long sequence data. It can capture long-range dependencies in the input data, which makes it particularly good at processing complex attack data (such as network traffic, system logs, etc.).

[0109] The following will further describe how to perform data processing and analysis on the changing environment data in the attack discriminator.

[0110] Reference Figure 5 , inputting the changed environment data into the attack discriminator for data processing to obtain an initial attack discrimination result corresponding to the initial attack strategy, including the following steps 501 to 505.

[0111] Step 501: In the attack discriminator, the change environment data is formatted to obtain sequence change environment data.

[0112] Step 502: Perform attention processing and feedforward processing on the sequence-changing environmental data in turn to obtain environmental processing data.

[0113] Step 503: Perform result classification and discrimination processing on the environmental processing data to obtain a successful attack result corresponding to the initial attack strategy.

[0114] Step 504: Perform result regression discrimination processing on the environmental processing data to obtain an attack performance prediction result corresponding to the initial attack strategy.

[0115] Step 505: Based on the attack success result and the attack performance prediction result, an initial attack identification result is obtained.

[0116] Steps 501 to 505 are described in detail below.

[0117] In the attack discriminator, the obtained change environment data mainly comes from the response data of the adversarial system, which includes: network traffic data, that is, the traffic characteristics after the attack is launched, such as the size of the packet, the transmission rate, the IP source and destination, etc.; system log data, including abnormal events, error messages, access records, etc. in the system log; attack process data, the steps of the attack method, the attack path, the tools used, etc. The attack discriminator converts this data into a format suitable for input to the Transformer model (usually serialized data) to obtain the sequence change environment data and processes it as input.

[0118] Next, the input sequence change environment data is embedded and transformed into a vector with fixed dimensions. For discrete data such as network traffic and logs, "positional encoding" can be used to retain the order information in the data. Transformer weights each input position through the self-attention mechanism to learn the global dependency of the input sequence.

[0119] Then, the embedded transformed sequence-changing environmental data is sequentially processed with attention and feedforward to obtain environmental processing data that can be used for classification and regression processing.

[0120] In some embodiments, the self-attention mechanism and multi-head attention mechanism of Transformer are used to perform attention processing on the sequence-changing environment data after embedding transformation, and the feedforward neural network in Transformer is used for feedforward processing.

[0121] Among them, the self-attention mechanism of Transformer is used to capture long-distance dependencies in the input data. In the attack effect discrimination task, the effect of the attack is not only affected by local features, but also related to long-term factors such as the global state of the system and network traffic. Therefore, through the self-attention mechanism, the discriminator can comprehensively evaluate the relationship between multiple input features.

[0122] In addition, Transformer uses a multi-head attention mechanism to calculate multiple different attention weights in parallel, so that the relationship between input data can be learned in different subspaces. Through this mechanism, Transformer can capture richer patterns and features in the data.

[0123] In the Transformer feedforward neural network, each layer of the self-attention module is followed by a feedforward neural network to further process the attention-weighted features. The feedforward neural network contains two fully connected layers, and the activation function is generally ReLU.

[0124] Finally, the Transformer model outputs the results of the two main tasks through a classification head and a regression head. Therefore, after the self-attention processing, multi-head attention processing, and feedforward processing of the Transformer to obtain the environment processing data, the environment processing data is subjected to result classification and discrimination processing to obtain the attack success result corresponding to the initial attack strategy, and the attack discriminator outputs whether the attack is effective. At the same time, the environment processing data is subjected to result regression and discrimination processing to obtain the attack performance prediction result corresponding to the initial attack strategy, and the attack discriminator outputs the success rate and concealment of the attack.

[0125] In addition, refer to Figure 6 After obtaining the initial attack identification result based on the attack success result and the attack performance prediction result, the attack optimization method of the adversarial system further includes the following steps 601 to 603.

[0126] Step 601: Using the cross entropy function, a classification loss value is calculated based on the true label corresponding to the initial attack strategy and the attack performance prediction result.

[0127] Step 602: Using the mean square error function, a regression loss value is calculated based on the true label and the attack performance prediction result.

[0128] Step 603: Update the attack discriminator based on the classification loss value and the regression loss value.

[0129] Steps 601 to 603 are described in detail below.

[0130] In some embodiments, the classification head in the Transformer model is optimized and updated using a cross entropy function, that is, based on the true label corresponding to the initial attack strategy. The attack performance prediction results output by the classification head (identifies the predicted attack effectiveness probability), and the classification loss value is calculated as shown in the following formula (5).

[0131] (5) Correspondingly, the regression head in the Transformer model is optimized and updated using the mean square error function, that is, based on the true signature and the attack performance prediction results, the regression loss value is calculated as shown in the following formula (6).

[0132] (6) The initial attack strategy corresponds to is the true label, It is the predicted value output by the regression head, usually indicating the attack success rate or stealth score.

[0133] Then, the classification loss value and regression loss value obtained by this initial attack strategy are used to update the classification head and regression head in the Transformer model in the attack discriminator.

[0134] In some embodiments, the attack discriminator also uses a back-propagation algorithm to optimize the parameters of the Transformer model. Since the Transformer model involves a large amount of calculations, the Adam optimizer is usually used during training to accelerate convergence and reduce training time. The optimization formula of the Transformer model adopted in the embodiment of the present application is shown in the following formula (7).

[0135] (7) in, is the model parameter at the current moment, is the learning rate, is the gradient of the loss function.

[0136] Through the above steps 501 to 505, and steps 601 to 603, by converting the changing environment data into sequence data and performing attention processing, feedforward processing, classification discrimination and regression discrimination, a comprehensive evaluation of the attack effect and a dynamic update of the attack discriminator are achieved. The format conversion and attention processing of the sequence changing environment data ensure that the model can capture the long-distance dependencies in the environment data and improve the analysis ability of the discriminator; the classification discrimination and regression discrimination respectively provide a qualitative evaluation of the success or failure of the attack and a quantitative prediction of the attack performance, making the discrimination results more comprehensive and accurate. By calculating the classification loss value and the regression loss value through the cross entropy function and the mean square error function, the attack discriminator can self-optimize according to the difference between the true label and the predicted result, and continuously improve its discrimination ability. This closed-loop feedback mechanism ensures that the attack discriminator can adapt to the changes of the target system in real time and provide accurate attack effect evaluation, thereby providing reliable feedback for the optimization of the attack generator, and ultimately improving the effectiveness and intelligence of the attack strategy.

[0137] During the training process, the attack generator and the attack discriminator are jointly optimized through adversarial training. The goal of the attack generator is to generate virtual attacks that are difficult to be distinguished by the attack discriminator, and the goal of the attack discriminator is to distinguish the difference between the generated virtual attacks and the real attack data. The loss function of the attack generator uses the cross entropy loss function, as shown in the following formula (8).

[0138] (8) Among them, D(G(z|c)) is the score of the attack discriminator for the generated attack method, and G(z|c) is the virtual attack generated by the attack generator.

[0139] The loss function of the attack discriminator is shown in the following formula (9).

[0140] (9) Among them, x is the real attack method, and D(x|c) is the attack discriminator’s prediction of the real attack.

[0141] After the attack method is generated and evaluated by the discriminator, the attack generator will be adjusted based on the feedback from the discriminator. Through reinforcement learning, the attack generator can gradually optimize its attack strategy to improve the success rate and concealment of the attack.

[0142] Step 104: Obtain the action state value based on the initial attack discrimination result and the system environment data, obtain the initial action reward based on the initial attack discrimination result and the reward model, update the attack generator model based on the initial action reward and the action state value, and use the updated attack generator to generate an optimized attack strategy corresponding to the changed environment data.

[0143] Step 104 is described in detail below.

[0144] In some embodiments, after the attack discriminator evaluates the changed environmental data generated after the adversarial system is subjected to the initial attack strategy and obtains the initial attack discrimination result for characterizing the attack characteristics of the initial attack strategy, the attack optimization model associated with the attack generator is updated and optimized using the update framework of deep reinforcement learning, as described below.

[0145] Reference Figure 7 , obtain the action state value according to the initial attack discrimination result and the system environment data, obtain the initial action reward based on the initial attack discrimination result and the reward model, and update the attack generator model according to the initial action reward and the action state value, including the following steps 701 to 703.

[0146] Step 701: Obtain the action state value corresponding to the initial attack judgment result and the system environment data in the value decision network, and obtain the maximum predicted action state value corresponding to the changed environment data in the value decision network.

[0147] Step 702: Obtain an initial action reward based on the initial attack discrimination result and the reward model.

[0148] Step 703: Update the initial decision parameters corresponding to the initial attack discrimination result and the system environment data based on the initial action reward, the action state value, and the maximum predicted action state value.

[0149] Steps 701 to 703 are described in detail below.

[0150] In some embodiments, based on the value decision network in the attack optimization model, the state action value (i.e. ), and determine the maximum predicted action state value corresponding to the changing environment data In addition, the initial action reward is obtained based on the initial attack discrimination result and the reward model , then based on the initial action reward , action state value and the maximum predicted action state value , using the DQN update function shown in the above formula (4), the initial attack discrimination results and the initial decision parameters corresponding to the system environment data are updated .

[0151] Afterwards, the attack generator uses the updated attack optimization model to generate an optimized attack strategy corresponding to the changed environmental data, and uses the optimized attack strategy to act on the adversary to iteratively optimize the attack optimization model, attack generator, and attack discriminator, and then continuously optimize and improve the generated attack.

[0152] It can be understood that the training process of the reinforcement learning optimization module is divided into the following steps.

[0153] (1) Initialization: Initialize the parameters of the value decision network and set hyperparameters such as the discount factor γ, learning rate α, and reward function weight.

[0154] (2) Training loop: Based on the current state st, the attack generator selects an action at. After executing the action, the immediate reward rt and the new state st+1 are calculated. The Q value function is updated using DQN. Based on the new Q value, the generator updates its strategy.

[0155] (3) Strategy update: The attack generator adjusts its attack strategy according to the constantly changing Q-value function during the training process to achieve higher success rate, concealment, and adaptability.

[0156] (4) Convergence and Evaluation: The training process continues until the attack generator converges to a stable attack strategy that can demonstrate excellent performance in various defense situations.

[0157] Through the above steps 701 to 703, by obtaining the action state value and the maximum predicted action state value, and combining the initial attack discrimination result and the reward model to generate the initial action reward, the dynamic update of the initial decision parameters is realized. The acquisition of the action state value and the maximum predicted action state value ensures that the attack generator can optimize the decision based on the current state and future potential state, and improves the foresight and adaptability of the attack strategy; the generation of the initial action reward is based on the attack discrimination result and the reward model, and comprehensively considers the success, concealment and destructiveness of the attack, ensuring the rationality and comprehensiveness of the reward mechanism. By combining the initial action reward, the action state value and the maximum predicted action state value, the attack generator can optimize the initial decision parameters, so that the attack strategy can be adaptively adjusted in a complex environment, significantly improving the success rate, concealment and destructiveness of the attack.

[0158] Reference Figure 8 , is a schematic flow chart of an attack optimization process for a countermeasure system provided in an embodiment of the present application. Figure 8As shown in , first, the environmental perception module collects environmental data of the adversarial system (such as network topology, vulnerability information, traffic patterns, etc.) to provide real-time data support for the attack generator; the attack generator generates an initial attack strategy based on the environmental data and launches an attack on the adversarial system; the attack discriminator evaluates the attack effect and generates attack success results and performance prediction results; the environmental feedback module feeds back the changed environmental data after the attack to the attack generator and discriminator to form a closed-loop optimization mechanism; the feedback training module optimizes the attack strategy through reinforcement learning to improve the success rate, concealment and destructiveness of the attack; finally, the attack optimization model continuously adjusts itself in a dynamic environment to generate the optimal attack strategy. This process realizes the intelligence of attack generation, evaluation and optimization, and significantly improves the efficiency and effectiveness of network attack and defense confrontation.

[0159] The attack optimization method, device, electronic device and storage medium of the adversarial system proposed in the embodiment of the present application include: first, obtaining the current initial system environment data in the adversarial system, the initial system environment data including network topology information, vulnerability scanning information and traffic pattern information, performing data cleaning, normalization processing and feature screening processing on the initial system environment data in turn to obtain system environment feature data, performing environment conversion processing on the system environment feature data to obtain system environment data matching the input data of the attack generator; then, the attack generator includes an attack optimization model, which generates a state space based on the defense state information of the adversarial system, the attack characteristics of the attack generator and the changed environment parameters of the corresponding system after being attacked, generates an action space based on the attack type, attack intensity and attack timing of the attack generator, obtains an attack success parameter according to the changed environment parameters, generates an attack success reward function, generates an attack concealment reward function according to the attack concealment parameter obtained from the changed environment parameters, generates an attack destruction reward function according to the attack destruction parameter obtained from the changed environment parameters, generates an attack failure penalty function according to the attack failure parameter obtained from the changed environment parameters, accumulates the attack success reward function, the attack concealment reward function and the attack destruction reward function, and Subtract the attack failure penalty function to obtain the attack optimization function, and generate a reward model based on the attack optimization function to obtain the initial decision parameters of the value decision network, and generate an attack optimization model based on the initial decision parameters, state space, action space and reward model, and input the system environment data into the attack generator for data processing to obtain the initial attack strategy, and obtain the change environment data of the adversarial system after the initial attack strategy is applied; next, in the attack discriminator, convert the change environment data into a format to obtain the sequence change environment data, and perform attention processing and feedforward processing on the sequence change environment data in turn to obtain the environment processing Data, classify and discriminate the environmental processing data to obtain the successful attack result corresponding to the initial attack strategy, perform regression and discriminate the environmental processing data to obtain the attack performance prediction result corresponding to the initial attack strategy, obtain the initial attack discrimination result based on the successful attack result and the attack performance prediction result, use the cross entropy function to calculate the classification loss value based on the true label corresponding to the initial attack strategy and the attack performance prediction result, use the mean square error function to calculate the regression loss value based on the true label and the attack performance prediction result, and update the attack discriminator based on the classification loss value and the regression loss value;Finally, the action state value corresponding to the initial attack discrimination result and the system environment data in the value decision network is obtained, and the maximum predicted action state value corresponding to the changed environment data in the value decision network is obtained. The initial action reward is obtained based on the initial attack discrimination result and the reward model. The initial decision parameters corresponding to the initial attack discrimination result and the system environment data are updated based on the initial action reward, the action state value and the maximum predicted action state value, and the updated attack generator is used to generate an optimized attack strategy corresponding to the changed environment data. ;

[0160] The embodiment of the present application combines adversarial network technology and reinforcement learning technology, uses an attack generator combined with reinforcement learning technology to generate an initial attack strategy corresponding to the real-time system environment data of the adversarial system, and uses an attack discriminator to evaluate and discriminate the changed environment data of the adversarial system after the initial attack strategy acts on it to obtain an initial attack discrimination result, and then uses the initial attack discrimination result to optimize and improve the attack generator, thereby dynamically adapting to changes in the target system, generating diversified attack strategies, and continuously improving the effectiveness and concealment of the attack through a closed-loop optimization mechanism, so as to overcome the problems of the existing attack generation technology lacking environmental adaptability, the singleness of the attack method, and the incompleteness of the evaluation mechanism, and provide a more efficient, intelligent, and dynamically optimized network attack and defense confrontation model, which can better adaptively generate targeted attacks in a complex dynamic network environment. In addition, the environment perception module is used to accurately obtain the current network topology information, vulnerability scanning information and traffic pattern information in the adversarial system, and then cleans, normalizes, features screens and environmental transformation processes to ensure the accuracy, consistency and efficiency of the obtained system environment data, and provides high-quality input data for the attack generator, so that it is convenient for the subsequent attack generator to generate attack strategies, improve the pertinence and effectiveness of the attack strategy, and optimize the utilization of computing resources, making the attack generation process more intelligent and efficient, thereby improving the reliability and accuracy of attack optimization; and by generating reward functions for attack success, concealment, destructiveness and failure penalties according to changing environmental parameters, and accumulating these functions to obtain the attack optimization function, a multi-dimensional evaluation and optimization of the attack strategy is achieved. The design of this multi-reward mechanism enables the attack generator to not only focus on the success rate of the attack, but also comprehensively consider the concealment and destructiveness of the attack, ensuring that the generated attack strategy can effectively bypass the detection mechanism and cause greater damage while breaking through the defense of the target system. At the same time, the introduction of the attack failure penalty function avoids the generation of invalid or inefficient attack strategies, and improves the efficiency and intelligence of the attack generator. In addition, by using the idea of ​​reinforcement learning, the attack optimization model is constructed by generating state space, action space, attack optimization function and reward model, and combining the initial decision parameters to realize the intelligent optimization of the attack strategy. The generation of the state space is based on the defense state, attack characteristics and changing environmental parameters of the adversarial system, ensuring that the attack generator can fully perceive the real-time state of the target system; the generation of the action space is based on the attack type, attack intensity and attack timing, providing a variety of attack strategy options; the design of the attack optimization function and reward model comprehensively considers the success, concealment and destructiveness of the attack, ensuring that the generated attack strategy is not only efficient but also targeted.Finally, by combining the initial decision parameters, state space, action space and reward model, the attack optimization model can adaptively generate the optimal attack strategy in a complex network environment; in addition, by converting the changing environment data into sequence data and performing attention processing, feedforward processing, classification discrimination and regression discrimination, a comprehensive evaluation of the attack effect and a dynamic update of the attack discriminator are achieved. The format conversion and attention processing of the sequence-changing environment data ensure that the model can capture the long-distance dependencies in the environment data and improve the analysis ability of the discriminator; classification discrimination and regression discrimination provide qualitative evaluation of the success of the attack and quantitative prediction of the attack performance, respectively, making the discrimination results more comprehensive and accurate. By calculating the classification loss value and regression loss value through the cross entropy function and the mean square error function, the attack discriminator can self-optimize according to the difference between the true label and the predicted result, and continuously improve its discrimination ability; and by obtaining the action state value and the maximum predicted action state value, and combining the initial attack discrimination result and the reward model to generate the initial action reward, the dynamic update of the initial decision parameters is achieved. The acquisition of action state value and maximum predicted action state value ensures that the attack generator can optimize decisions based on the current state and future potential state, improving the foresight and adaptability of the attack strategy; the generation of initial action rewards is based on the attack discrimination results and reward model, taking into account the success, concealment and destructiveness of the attack, ensuring the rationality and comprehensiveness of the reward mechanism. By combining the initial action reward, action state value and maximum predicted action state value, the attack generator can optimize the initial decision parameters, so that the attack strategy can be adaptively adjusted in a complex environment, significantly improving the success rate, concealment and destructiveness of the attack.

[0161] The embodiment of the present application also provides an attack optimization device for a confrontation system, which can implement the attack optimization method for the confrontation system. Fig. 9 , the device 900 comprises: An acquisition module 910 is used to acquire system environment data in the confrontation system; The attack generation module 920 is used to input the system environment data into the attack generator for data processing, obtain the initial attack strategy, and obtain the changed environment data of the adversarial system after the initial attack strategy is applied; The attack identification module 930 is used to input the changed environment data into the attack identification device for data processing to obtain an initial attack identification result corresponding to the initial attack strategy; The attack optimization module 940 is used to obtain the action state value based on the initial attack discrimination result and the system environment data, obtain the initial action reward based on the initial attack discrimination result and the reward model, update the attack generator model according to the initial action reward and the action state value, and use the updated attack generator to generate an optimized attack strategy corresponding to the changed environment data.

[0162] In some embodiments, the attack generation module 920 is further configured to: Generate a state space based on the defense state information of the adversarial system, the attack characteristics of the attack generator, and the changed environmental parameters of the corresponding system after being attacked; Generate an action space based on the attack type, attack strength, and attack timing of the attack generator; Generate an attack optimization function based on the changing environment parameters, and generate a reward model based on the attack optimization function; The initial decision parameters of the value decision network are obtained, and an attack optimization model is generated based on the initial decision parameters, state space, action space, and reward model.

[0163] In some embodiments, the attack generation module 920 is further configured to: Obtain attack success parameters according to the changing environment parameters and generate an attack success reward function; Generate an attack concealment reward function based on the attack concealment parameters obtained from the changing environment parameters; Generate an attack and destruction reward function based on the attack and destruction parameters obtained from the changing environmental parameters; Generate an attack failure penalty function based on the attack failure parameters obtained from the changing environmental parameters; The attack success reward function, attack concealment reward function, and attack destruction reward function are accumulated, and the attack failure penalty function is subtracted to obtain the attack optimization function.

[0164] In some embodiments, the attack optimization module 940 is further configured to: Obtaining the action state value corresponding to the initial attack discrimination result and the system environment data in the value decision network, and obtaining the maximum predicted action state value corresponding to the changed environment data in the value decision network; Get the initial action reward based on the initial attack discrimination result and reward model; Based on the initial action reward, action state value and maximum predicted action state value, the initial attack discrimination results and the initial decision parameters corresponding to the system environment data are updated.

[0165] In some embodiments, the attack determination module 930 is further configured to: In the attack discriminator, the format of the change environment data is converted to obtain the sequence change environment data; Perform attention processing and feedforward processing on the sequence-changing environmental data in turn to obtain environmental processing data; The environmental processing data is classified and judged to obtain the successful attack result corresponding to the initial attack strategy; Perform result regression and discrimination processing on the environmental processing data to obtain the attack performance prediction result corresponding to the initial attack strategy; Based on the attack success results and attack performance prediction results, the initial attack discrimination results are obtained.

[0166] In some embodiments, the attack determination module 930 is further configured to: Using the cross entropy function, the classification loss value is calculated based on the true label corresponding to the initial attack strategy and the attack performance prediction result; Using the mean square error function, the regression loss value is calculated based on the true label and the attack performance prediction results; The attack discriminator is updated based on the classification loss value and the regression loss value.

[0167] In some embodiments, the acquisition module 910 is further configured to: Obtaining the current initial system environment data in the adversarial system, the initial system environment data including network topology information, vulnerability scanning information, and traffic pattern information; The initial system environment data is sequentially cleaned, normalized, and feature screened to obtain system environment feature data; Perform environmental transformation processing on the system environment feature data to obtain system environment data that matches the input data of the attack generator.

[0168] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, the specific implementation method of the attack optimization device of the confrontation system is basically the same as the specific implementation method of the attack optimization method of the confrontation system, and will not be repeated here.

[0169] In the embodiment of the present application, the attack optimization device of the adversarial system combines the adversarial network technology and the reinforcement learning technology, uses the attack generator combined with the reinforcement learning technology to generate an initial attack strategy corresponding to the real-time system environment data of the adversarial system, and uses the attack discriminator to evaluate and discriminate the changed environment data of the adversarial system after the initial attack strategy acts on it to obtain the initial attack discrimination result, and then uses the initial attack discrimination result to optimize and improve the attack generator, so as to dynamically adapt to the changes of the target system, generate diversified attack strategies, and continuously improve the effectiveness and concealment of the attack through a closed-loop optimization mechanism, so as to overcome the problems of the existing attack generation technology lacking environmental adaptability, the singleness of the attack method and the incompleteness of the evaluation mechanism, and provide a more efficient, intelligent and dynamically optimized network attack and defense confrontation model, which can better adapt to the complex dynamic network environment. In addition, the environment perception module is used to accurately obtain the current network topology information, vulnerability scanning information and traffic pattern information in the adversarial system, and then cleans, normalizes, features the data and transforms the environment to ensure the accuracy, consistency and efficiency of the obtained system environment data, and provides high-quality input data for the attack generator, so that it is convenient for the subsequent attack generator to generate attack strategies, improve the pertinence and effectiveness of the attack strategy, and optimize the utilization of computing resources, making the attack generation process more intelligent and efficient, thereby improving the reliability and accuracy of attack optimization; and by generating reward functions for attack success, concealment, destructiveness and failure penalties according to changing environmental parameters, and accumulating these functions to obtain the attack optimization function, the multi-dimensional evaluation and optimization of the attack strategy is realized. The design of this multi-reward mechanism enables the attack generator to not only focus on the success rate of the attack, but also comprehensively consider the concealment and destructiveness of the attack, ensuring that the generated attack strategy can effectively bypass the detection mechanism and cause greater damage while breaking through the defense of the target system. At the same time, the introduction of the attack failure penalty function avoids the generation of invalid or inefficient attack strategies, and improves the efficiency and intelligence of the attack generator. In addition, by using the idea of ​​reinforcement learning, the attack optimization model is constructed by generating state space, action space, attack optimization function and reward model, and combining the initial decision parameters to realize the intelligent optimization of the attack strategy. The generation of the state space is based on the defense state, attack characteristics and changing environmental parameters of the adversarial system, ensuring that the attack generator can fully perceive the real-time state of the target system; the generation of the action space is based on the attack type, attack intensity and attack timing, providing a variety of attack strategy options; the design of the attack optimization function and reward model comprehensively considers the success, concealment and destructiveness of the attack, ensuring that the generated attack strategy is not only efficient but also targeted.Finally, by combining the initial decision parameters, state space, action space and reward model, the attack optimization model can adaptively generate the optimal attack strategy in a complex network environment; in addition, by converting the changing environment data into sequence data and performing attention processing, feedforward processing, classification discrimination and regression discrimination, a comprehensive evaluation of the attack effect and a dynamic update of the attack discriminator are achieved. The format conversion and attention processing of the sequence-changing environment data ensure that the model can capture the long-distance dependencies in the environment data and improve the analysis ability of the discriminator; classification discrimination and regression discrimination provide qualitative evaluation of the success of the attack and quantitative prediction of the attack performance, respectively, making the discrimination results more comprehensive and accurate. By calculating the classification loss value and regression loss value through the cross entropy function and the mean square error function, the attack discriminator can self-optimize according to the difference between the true label and the predicted result, and continuously improve its discrimination ability; and by obtaining the action state value and the maximum predicted action state value, and combining the initial attack discrimination result and the reward model to generate the initial action reward, the dynamic update of the initial decision parameters is achieved. The acquisition of action state value and maximum predicted action state value ensures that the attack generator can optimize decisions based on the current state and future potential state, improving the foresight and adaptability of the attack strategy; the generation of initial action rewards is based on the attack discrimination results and reward model, taking into account the success, concealment and destructiveness of the attack, ensuring the rationality and comprehensiveness of the reward mechanism. By combining the initial action reward, action state value and maximum predicted action state value, the attack generator can optimize the initial decision parameters, so that the attack strategy can be adaptively adjusted in a complex environment, significantly improving the success rate, concealment and destructiveness of the attack.

[0170] The present application also provides an electronic device, including: at least one memory; at least one processor; at least one program; The program is stored in the memory, and the processor executes the at least one program to implement the attack optimization method of the above-mentioned countermeasure system implemented in this application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0171] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes: The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 1002 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device or RAM (Random Access Memory). The memory 1002 can store operating systems and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1002, and the processor 1001 calls and executes the attack optimization method of the countermeasure system in the embodiments of this application; Input / output interface 1003, used to implement information input and output; The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.); A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 ); The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0172] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the attack optimization method for the above-mentioned countermeasure system is implemented.

[0173] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0174] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0175] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0176] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0178] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0179] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0181] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0183] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0184] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for optimizing an attack against a system, characterized in that: The method comprises: Acquiring system environment data in the confrontation system; Inputting the system environment data into the attack generator for data processing to obtain an initial attack strategy, and obtaining the changed environment data of the adversarial system after the initial attack strategy is applied; Inputting the changed environment data into an attack discriminator for data processing to obtain an initial attack discrimination result corresponding to the initial attack strategy; An action state value is obtained according to the initial attack discrimination result and the system environment data, an initial action reward is obtained based on the initial attack discrimination result and a reward model, a model of the attack generator is updated according to the initial action reward and the action state value, and an optimized attack strategy corresponding to the changed environment data is generated using the updated attack generator.

2. The attack optimization method for the confrontation system according to claim 1, characterized in that: The attack generator includes an attack optimization model, and the steps of constructing the attack optimization model include: Generate a state space based on the defense state information of the adversarial system, the attack characteristics of the attack generator, and the changed environmental parameters of the corresponding system after being attacked; Generate an action space based on the attack type, attack intensity, and attack timing of the attack generator; generating an attack optimization function based on the changing environmental parameters, and generating a reward model based on the attack optimization function; Initial decision parameters of the value decision network are obtained, and the attack optimization model is generated based on the initial decision parameters, the state space, the action space, and the reward model.

3. The attack optimization method for the confrontation system according to claim 2, characterized in that: The generating of the attack optimization function based on the changing environment parameters comprises: Obtaining an attack success parameter according to the changed environmental parameter, and generating an attack success reward function; Generating an attack concealment reward function according to the attack concealment parameter obtained from the changing environment parameter; Generating an attack and destruction reward function according to the attack and destruction parameters obtained from the changing environmental parameters; generating an attack failure penalty function according to the attack failure parameters obtained from the changing environmental parameters; The attack success reward function, the attack concealment reward function, and the attack destruction reward function are accumulated, and the attack failure penalty function is subtracted to obtain the attack optimization function.

4. The attack optimization method for the confrontation system according to claim 2, characterized in that: The step of obtaining an action state value according to the initial attack discrimination result and the system environment data, obtaining an initial action reward based on the initial attack discrimination result and a reward model, and updating a model of the attack generator according to the initial action reward and the action state value includes: Obtaining the action state value corresponding to the initial attack discrimination result and the system environment data in the value decision network, and obtaining the maximum predicted action state value corresponding to the changed environment data in the value decision network; Obtaining an initial action reward based on the initial attack discrimination result and the reward model; The initial attack discrimination result and the initial decision parameter corresponding to the system environment data are updated based on the initial action reward, the action state value and the maximum predicted action state value.

5. The attack optimization method for the confrontation system according to claim 1, characterized in that: The step of inputting the changed environment data into an attack discriminator for data processing to obtain an initial attack discriminant result corresponding to the initial attack strategy includes: In the attack discriminator, the change environment data is formatted to obtain sequence change environment data; Performing attention processing and feedforward processing on the sequence-changing environmental data in sequence to obtain environmental processing data; Performing result classification and discrimination processing on the environmental processing data to obtain a successful attack result corresponding to the initial attack strategy; Performing result regression and discrimination processing on the environmental processing data to obtain an attack performance prediction result corresponding to the initial attack strategy; The initial attack identification result is obtained based on the attack success result and the attack performance prediction result.

6. The attack optimization method for the confrontation system according to claim 5, characterized in that: After obtaining the initial attack identification result based on the attack success result and the attack performance prediction result, the method further includes: Using a cross entropy function, based on the true label corresponding to the initial attack strategy and the attack performance prediction result, a classification loss value is calculated; Using a mean square error function, based on the true label and the attack performance prediction result, a regression loss value is calculated; The attack discriminator is updated based on the classification loss value and the regression loss value.

7. The attack optimization method for the confrontation system according to claim 1, characterized in that: The obtaining of system environment data in the confrontation system includes: Acquire current initial system environment data in the adversarial system, wherein the initial system environment data includes network topology information, vulnerability scanning information, and traffic pattern information; The initial system environment data is sequentially cleaned, normalized, and characterized to obtain system environment characteristic data; Performing environment conversion processing on the system environment feature data to obtain the system environment data that matches the input data of the attack generator.

8. An attack optimization device for a countermeasure system, characterized in that: The device comprises: An acquisition module, used for acquiring system environment data in the confrontation system; An attack generation module, used for inputting the system environment data into an attack generator for data processing, obtaining an initial attack strategy, and obtaining the changed environment data of the adversarial system after the initial attack strategy acts; An attack discrimination module, used for inputting the changed environment data into an attack discriminator for data processing to obtain an initial attack discrimination result corresponding to the initial attack strategy; An attack optimization module is used to obtain an action state value based on the initial attack discrimination result and the system environment data, obtain an initial action reward based on the initial attack discrimination result and a reward model, update the attack generator model based on the initial action reward and the action state value, and use the updated attack generator to generate an optimized attack strategy corresponding to the changed environment data.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the attack optimization method for the countermeasure system described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the attack optimization method for the countermeasure system described in any one of claims 1 to 7 is implemented.

Images