System security boundary evaluation method and system based on confrontation test

Abstract

The invention discloses a system security boundary evaluation method and system based on an adversarial test, and belongs to the field of system security boundary evaluation, the method adopts an improved Latin hypercube sampling method provided by the invention to design adversarial test working conditions, and forms an adversarial test working condition set; according to the confrontation test working condition set, confrontation test experiments under different confrontation test working conditions are carried out; and performing security boundary evaluation analysis according to the test result of the confrontation test dependent variable. According to the method provided by the invention, the security boundary of the target system can be evaluated, and a reference direction is provided for realizing security boundary optimization of the target system.

Description

Technical Field

[0001] The present invention belongs to the field of system security boundary evaluation, and more specifically, relates to a system security boundary evaluation method and system based on adversarial testing. Background Art

[0002] A network range is a highly realistic network experiment platform that constructs a network environment highly similar to a real-world system by replicating complex network architectures and various software and hardware environments, providing a virtual space with controllable boundaries for security personnel. Conducting adversarial testing on a target system in a network range environment can make attack means more realistic, and the evaluation process is highly dynamic, enabling more direct observation of the actual performance of the system when facing malicious attacks, helping security personnel timely discover potential vulnerabilities in the system, and enabling security personnel to comprehensively evaluate the security of the target system; the network range, with its flexible scale strength regulation and isolation measures, helps evaluators overall control the overall risk of the evaluation process. To carry out adversarial testing relying on the network range, it is necessary to first construct a range environment close to the real scenario and set various attack scenarios and rules; subsequently, security personnel attack the target system, and security personnel use various attack means to simulate real threats while the target system implements defense strategies. During the testing process, attack behaviors, defense effects, and system responses are recorded in detail.

[0003] Constrained by the time cost, computing cost, and cognitive constraints of adversarial testing, at present, adversarial testing is more applied to the verification of the security boundary conclusion of the target system, that is, after obtaining the security boundary conclusion of the target system through security evaluation means such as security baselines and risk matrices, a small-sample adversarial test is implemented to verify whether it meets the standard. Since the accuracy of the security boundary obtained by static evaluation means is relatively rough, even if adversarial testing is applied for verification, the advantages of the dynamic process and true conclusion of adversarial testing cannot be fully utilized, and it is impossible to explore the security boundary of the target system to the bottom and obtain the true security level of the system. However, blindly conducting large-sample adversarial testing in a network range has the following three drawbacks: first, it will cause waste of computing, time, and human resources; second, it will lead to the lack of effectiveness of test results, and the evaluation conclusion of the system security boundary is not interpretable; third, it will bring additional security risks and may overly damage the target system.

[0004] Therefore, how to reasonably conduct large-sample adversarial testing to evaluate the security boundary of the target system is an urgent problem to be solved currently. Summary of the Invention

[0005] In view of the above deficiencies or improvement requirements of the prior art, the present invention provides a method and system for evaluating the security boundary of a system based on adversarial testing, thereby solving the problems of poor analysis accuracy and inability to accurately identify the security boundary of the target system existing in the existing security boundary evaluation means combining static evaluation and adversarial testing verification, as well as the problems of waste of computing, time and human resources, insufficient effectiveness of test results, and additional security risks of the target system existing in blindly evaluating the security boundary through large-sample adversarial testing.

[0006] To achieve the above object, according to the first aspect of the present invention, there is provided a method for evaluating the security boundary of a system based on adversarial testing, including:

[0007] S1, setting the number of adversarial tests N for the target system test , the total number of independent variables NUM for the adversarial test test-x , the set D of independent variables for the adversarial test test-x and the set D of dependent variables for the adversarial test test-y ;

[0008] Among them, D test-x includes the name of the independent variable, the type of the independent variable, the subset of discrete independent variable values, and the inverse cumulative distribution function of the continuous independent variable; the type of the independent variable includes discrete and continuous, and the subset of discrete independent variable values includes the value sets of each discrete independent variable; D test-x includes the name of the dependent variable and the type of the dependent variable, and the dependent variable is the security index of the target system;

[0009] S2, generating random sequences corresponding to the respective independent variables within the standardized parameter space [0, 1) NUMtest-x , and forming an adversarial test condition standard sample matrix X test-x ×N test in size; mapping the adversarial test condition standard sample points corresponding to the discrete independent variables in X standard to the subset of discrete independent variable values to obtain the corresponding adversarial test condition actual sample points; substituting the adversarial test condition standard sample points corresponding to the continuous independent variables in X standard into their inverse cumulative distribution functions to obtain the corresponding adversarial test condition actual sample points; standard S3, performing an adversarial test on the target system according to the adversarial test condition actual sample matrix X actual , and extracting the values of the respective dependent variables of the adversarial test from the test results for evaluating the security boundary of the target system;

[0010]

[0011] Among them, X actual is composed of the actual sample points of the adversarial test conditions of each discrete independent variable and each continuous independent variable.

[0012] According to the second aspect of the present invention, there is provided an electronic device, comprising: a computer-readable storage medium and a processor;

[0013] The computer-readable storage medium is used to store executable instructions;

[0014] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method as described in the first aspect.

[0015] According to the third aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to execute the method as described in the first aspect.

[0016] According to the fourth aspect of the present invention, there is provided a computer program product comprising a computer program or instructions which, when executed by a processor, implement the method as described in the first aspect.

[0017] Generally speaking, compared with the prior art, the above technical solution conceived by the present invention can achieve the following beneficial effects:

[0018] The method provided by the present invention generates random sequences corresponding to each independent variable used in the adversarial test respectively within the standardized parameter space, forms a standard sample matrix for the adversarial test working conditions, and maps the standard sample points corresponding to each independent variable in the above matrix to actual sample points according to the type of the independent variable to obtain an actual sample matrix for the adversarial test working conditions, so as to perform a large-sample adversarial working condition test on the target system according to the actual sample matrix for the adversarial test working conditions; by reasonably designing the adversarial test working conditions, it is possible to cover multi-variable combinations with a relatively small number of samples. Compared with blindly conducting a large-sample adversarial test, the present method can reduce useless repeated calculations, thereby reducing time and computing power costs, and can help security personnel reasonably implement adversarial test tasks in the network range at a relatively low cost, realize probing the security boundary of the target system, and provide a reference direction for optimizing the security boundary of the target system; considering that in addition to the large number of independent variables in the adversarial test, the variable types are also diverse: for example, it usually includes continuous variables such as network environment delay and discrete variables such as attack types, etc., so the design of the adversarial test working conditions needs to consider the sampling tasks of both continuous variables and discrete variables at the same time. While the typical Latin hypercube is mostly used for sampling tasks of a single variable type, using an improved Latin hypercube sampling method to implement the design of the adversarial test working conditions can achieve collaborative sampling of mixed variables, improve the sampling efficiency, and avoid combination omissions caused by the fragmentation of variable types in the traditional method. Description of the Drawings

[0019] Figure 1Flowchart of the system security boundary evaluation method based on adversarial testing provided by an embodiment of the present invention;

[0020] Figure 2 Flowchart of the construction of the adversarial testing working condition set provided by an embodiment of the present invention;

[0021] Figure 3 Flowchart of the processing of adversarial testing results provided by an embodiment of the present invention;

[0022] Figure 4 Flowchart of the security boundary evaluation provided by an embodiment of the present invention;

[0023] Figure 5 Security boundary envelope diagram of the dependent variable "security event_system scheduling delay" provided by an embodiment of the present invention;

[0024] Figure 6 Schematic diagram of the system structure of the system security boundary evaluation based on adversarial testing provided by an embodiment of the present invention. Detailed implementation manners

[0025] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0026] An embodiment of the present invention provides a system security boundary evaluation method based on adversarial testing, including:

[0027] S1, setting the number of adversarial tests N of the target system test , the total number of independent variables NUM of the adversarial test test-x , the set D of independent variables of the adversarial test test-x and the set D of dependent variables of the adversarial test test-y ;

[0028] Among them, D test-x includes the independent variable name, independent variable type, subset of discrete independent variable values and the inverse cumulative distribution function of the continuous independent variable; the independent variable type includes discrete and continuous types, and the subset of discrete independent variable values includes the value sets of each discrete independent variable; D test-x includes the dependent variable name and dependent variable type, and the dependent variable is the security index of the target system.

[0029] In step S1, security personnel sort out and clarify the mission scenario for implementing the adversarial test using the network range, including the number of adversarial tests N test , the total number of independent variables NUM of the adversarial testtest-x , the total number of dependent variables NUM in the adversarial test test-y , the set of independent variables D in the adversarial test test-x , the set of dependent variables D in the adversarial test test-y .

[0030] It can be understood that the system is a general concept, such as industrial control systems, manufacturing execution systems, enterprise resource planning systems, supervisory control and data acquisition systems, etc.

[0031] The number of adversarial tests N test is an integer variable, which is the number of times security personnel conduct adversarial tests on the network range relied on to evaluate the security boundary of the target system.

[0032] The total number of independent variables NUM in the adversarial test test-x is an integer variable, which is the total number of variables actively set and manipulated by security personnel when conducting adversarial tests on the network range relied on to evaluate the security boundary of the target system.

[0033] The total number of dependent variables NUM in the adversarial test test-y is an integer variable, which is the total number of target system security index variables that security personnel are concerned about.

[0034] The set of independent variables D in the adversarial test test-x , which is the details of the active settings and manipulations of the network range by security personnel to evaluate the security boundary of the target system. The set of independent variables in the adversarial test is shown in formula (1):

[0035] D test-x = {X-ID, X-type, X-value, X-ICDF} (1)

[0036] In formula (1), the set of independent variables D in the adversarial test test-x is a 4-tuple, including the independent variable name X-ID, independent variable type X-type, subset of independent variable values X-value, and inverse cumulative distribution function of the independent variable X-ICDF.

[0037] In formula (1), the independent variable name X-ID is used by the system security boundary evaluation system to identify the independent variables involved in the evaluation, and the independent variable name needs to ensure uniqueness.

[0038] In formula (1), the independent variable type X-type represents the variable attributes of each independent variable in the set of independent variables in the adversarial test, including 2 types: continuous variables and discrete variables.

[0039] In formula (1), the independent variable value subset X - value defines the value set of each discrete variable in the set of independent variables for the adversarial test, and guides the adversarial test working condition design module to extract the values of the discrete independent variables from it. For continuous variables, the independent variable value subset X - value is not set. Therefore, the independent variable value subset X - value is also called the discrete independent variable value subset.

[0040] In formula (1), the independent variable inverse cumulative distribution function X - ICDF is a function that maps the given probability values of different independent variables to the corresponding quantiles, so as to realize the mapping from the standard sample points of the test working conditions to the actual sample points of the test working conditions. For discrete variables, the independent variable inverse cumulative distribution function X - ICDF is not set. Therefore, the independent variable inverse cumulative distribution function X - ICDF is also called the continuous independent variable inverse cumulative distribution function X - ICDF, which is used to map the standard sample points of the test working conditions corresponding to the continuous independent variables to the actual value range of the actual sample points of the test working conditions, so as to obtain the values of the actual sample points of the test working conditions corresponding to the continuous independent variables.

[0041] Set D of dependent variables for the adversarial test test-y , which is the target system security index concerned by security personnel. The set of independent variables for the adversarial test can be as shown in formula (2):

[0042] D test-y ={Y - ID, Y - type} (2)

[0043] In formula (2), the set D of dependent variables for the adversarial test test-y is a 2 - tuple, including the dependent variable name Y - ID and the dependent variable type Y - type.

[0044] In formula (2), the dependent variable name Y - ID is used to identify the dependent variables involved in the evaluation, and the dependent variable name needs to ensure uniqueness.

[0045] In formula (2), the dependent variable type Y - type represents the variable attributes of each dependent variable in the set of dependent variables for the adversarial test, including continuous variables and discrete variables.

[0046] S2. Generate random sequences corresponding to their respective independent variables within the standardized parameter space to form an adversarial test working condition standard sample matrix X of size NUM test-x ×N test ; For each discrete independent variable, use the equal - width binning method to map the values of the standard sample points of the adversarial test working conditions corresponding to it in X standard to the discrete independent variable value subset, and obtain the values of the actual sample points of the adversarial test working conditions corresponding to each discrete independent variable. For each continuous independent variable, for X standard standardThe value of the corresponding standard sample point of the adversarial test condition in it is used as the given probability value and substituted into its inverse cumulative distribution function to obtain the value of the actual sample point of the adversarial test condition corresponding to each continuous independent variable.

[0047] In step S2, the improved Latin hypercube sampling method proposed by the present invention is used to implement the design of the adversarial test condition. The core of the improved Latin hypercube sampling method is: in the standardized parameter space After generating the standard sample matrix, different distribution mapping methods are used to achieve the hierarchical uniformity of continuous variables and discrete variables, thereby making up for the limitation that the classical Latin hypercube sampling method cannot sample in the mixed variable space of continuous and discrete variables at the same time.

[0048] The design process of the adversarial test condition is as Figure 3 shown, and it includes a total of 2 steps: S21 standard sample matrix generation and S22 actual condition interval mapping. S21 standard sample matrix generation includes sub-steps of S211 independent permutation design, S212 multi-dimensional sample generation, and S213 sample matrix combination; S22 actual condition interval mapping includes sub-steps of S221 variable type judgment, S222 equal-width binning mapping, S223 inverse transformation method mapping, and S224 adversarial test condition set integration.

[0049] Sub-step S211 independent permutation design: This step is to ensure the uniform distribution of samples in the standardized parameter space. In the standardized parameter space NUMtest-x is the dimension of this standardized parameter space. For each adversarial test independent variable k (k = 1, 2,... NUM test-x ), a random sequence is generated (that is, random sequences corresponding to each adversarial test independent variable are generated respectively, and a total of NUM test-x random sequences are generated, and the length of each random sequence is N test ). The random sequence is shown in formula (3):

[0050] π k =[π k (1), π k (2),... π k (N test )] (3)

[0051] Sub-step S212 multi-dimensional sample generation: It is implemented by accumulating random offsets on the basis of the random sequence. This step is an optional step. In order to further improve the randomness of the random sequence, the k-th adversarial test independent variable coordinate of the i-th adversarial test condition sample point is shown in formula (4):

[0052]

[0053] In formula (4), the random offset Subject to the standard uniform distribution.

[0054] S213 Sample matrix combination sub-step: This step can generate the standard sample matrix for the test conditions. Each row of this matrix represents the standard sample of a simulation condition, and each column represents the standard sample of an independent variable in each simulation condition. The standard sample matrix is shown in formula (5):

[0055]

[0056] S221 Variable type judgment sub-step: By reading the set D of independent variables for the adversarial test test-x , successively execute the corresponding distribution mapping method according to the independent variable type X-type. For the k-th independent variable of the i-th adversarial test condition sample point, if its independent variable type X-type k is a discrete variable, then execute the equal-width binning mapping sub-step; if it is a continuous variable, then execute the inverse transformation method mapping sub-step.

[0057] S222 Equal-width binning mapping sub-step: First, it is necessary to encode the k-th subset of continuous independent variable values X-value k , count the number of elements in the subset X-value k of this discrete variable, that is, num(X-value k ), divide the interval [0, 1) into num(X-value k ) continuous interval blocks, and assign a separate interval block to each element in the subset X-value k set. Map according to the interval block where the coordinates of the standard sample matrix fall, and determine the actual condition of the k-th independent variable of the i-th adversarial test condition sample point as the element corresponding to the interval block, as shown in formula (6):

[0058]

[0059] In formula (6), is the actual condition of the k-th independent variable of the i-th adversarial test condition sample point.

[0060] It should be noted that other binning methods can also be used in S222, such as equal-frequency binning method, quantile binning method, etc. To ensure balanced sampling, the present invention preferably uses the equal-width binning method.

[0061] S223 Inverse transformation method mapping sub-step: First, read the inverse cumulative distribution function X-ICDF of the independent variable of this continuous variable k , and use the coordinates Bring in the independent variable inverse cumulative distribution function X-ICDF k The actual working condition of the k-th adversarial test independent variable of the i-th adversarial test working condition sample point can be obtained as shown in formula (7):

[0062]

[0063] By sequentially completing the mapping transformation of the adversarial test independent variables for each adversarial test working condition sample point, an actual working condition sample matrix can be obtained as shown in formula (8):

[0064]

[0065] That is, X actual It consists of the actual sample points of the adversarial test working conditions of each discrete independent variable and each continuous independent variable. It can be understood that X actual The elements in are in one-to-one correspondence with the elements in X standard They are of the same size, both NUM test-x ×N test .

[0066] It should be noted that considering that the inverse cumulative distribution functions of some continuous independent variables are unknown. For example, the inverse cumulative distribution function of the independent variable is non-explicit or its cumulative distribution function is too complex, then the historical data of the continuous independent variable in the target system is fitted to obtain its inverse cumulative distribution function.

[0067] Specifically, the continuous independent variable inverse cumulative distribution function X-ICDF can be obtained by formula expression, numerical method solution or table lookup method:[[]]END]]

[0068] (1) If the cumulative distribution function of the independent variable has a clear analytical expression and the expression has an inverse function, then the inverse cumulative distribution function can be directly derived through formula expression. For example, the independent variable of "network environment - network delay" is a uniform distribution function in the interval of 10 - 500ms, and its independent variable inverse cumulative distribution function X-ICDF = 10 + 490x;

[0069] (2) When the inverse cumulative distribution function of the independent variable is non-explicit or the distribution of its complex function is too complex, numerical methods such as rational approximation or Newton's method can be used to solve and obtain it. Rational approximation approximates the inverse cumulative distribution function within a specific interval to minimize the error by constructing a rational function (quotient of polynomials) and optimizing the coefficients based on the least squares or Chebyshev criterion; while the solution process of Newton's method starts from an initial guess value, iteratively adjusts the numerical value, and gradually approaches the target quantile according to the gap between the current cumulative probability and the target probability and the change trend of the probability density until the accuracy requirement is met. For example, the independent variable "External Attack - Attack Path Complexity" follows a Zipf distribution with a value range of [1, 100], and its inverse cumulative distribution function is non-explicit. The distribution parameters cannot be directly deduced through theoretical formulas, and the inverse cumulative distribution function can be solved by fitting and calculating using numerical methods based on historical data;

[0070] (3) In the process of engineering applications, when the accuracy requirement for the distribution is not high, the table lookup method is a common means to obtain the values of the inverse cumulative distribution function. For common probability distributions such as the normal distribution, t-distribution, and chi-square distribution, the values of the inverse cumulative distribution function corresponding to different probability values are calculated in advance and made into a table. In actual applications, only by looking up the table according to the given probability value can an approximate result be obtained. For example, the independent variable "External Attack - Attack Duration" follows a normal distribution with a mean of 3 hours and a standard deviation of 1 hour. At this time, the Z-value table can be used for querying, and the negative part of the Z-value table can be truncated and processed as 0.

[0071] S224 Anti-Test Condition Set Integration Sub-step: This step is an optimization step. To ensure that the designed anti-test conditions are easy to identify, the actual condition sample matrix X actual is integrated with the independent variable name X-ID into the anti-test condition set X profiles , as shown in formula (9).

[0072]

[0073] The anti-test condition set generated by the design is sent to the information processing module and the network range module through the application programming interface.

[0074] S3. Conduct an anti-test on the target system according to the actual sample matrix X actual of the anti-test conditions, and extract the values of each anti-test dependent variable from the test results for the security boundary evaluation of the target system;

[0075] Step S3 includes three sub-steps: S31 Execute the anti-test, S32 Process the process data, and S33 Implement the security boundary evaluation. Specifically:

[0076] S31 Perform adversarial testing: Using the set of adversarial testing scenarios as input, automatically conduct adversarial testing experiments under different adversarial testing scenarios, and push the process data of the adversarial testing experiments under different adversarial testing scenarios to the information processing module. By parsing the set of adversarial testing scenarios X profiles N test adversarial testing scenario sample points can be obtained, and the adversarial testing is sequentially performed according to the adversarial testing independent variables set for the adversarial testing scenario sample points.

[0077] S32 Process the process data: As Figure 4 shown, it includes the following sub-steps:

[0078] S321, sequentially receive the process data of the adversarial testing experiments under different adversarial testing scenarios;

[0079] S322, refer to the total number of adversarial testing dependent variables and the set of adversarial testing dependent variables sorted out.

[0080] Furthermore, for the convenience of data identification, preferably, it also includes: S333, process and extract all the process data of the adversarial testing experiments, integrate the results after processing and extraction with the set of adversarial testing scenarios to form a set of adversarial testing results, as shown in formula (10):

[0081]

[0082] After the processing is completed, perform a safety boundary evaluation according to Y test

[0083] S33 Implement safety boundary evaluation: As Figure 5 shown, it includes 2 sub-steps: S61 safety boundary evaluation analysis and S62 safety boundary envelope visualization.

[0084] S61 Safety boundary evaluation analysis: Use statistical means to sequentially analyze and calculate the statistical characteristics of the distribution of each adversarial testing dependent variable in the set of adversarial testing results: The statistical characteristics include mean, standard deviation, key point values (minimum value, 25% value, 50% value, 75% value, maximum value).

[0085] S62 Safety boundary envelope visualization: Use a violin plot to visually display the safety boundary envelope of the target system, providing a reference direction for optimizing the safety boundary of the target system.

[0086] Thus, the safety boundary evaluation of the target system is completed.

[0087] Next, taking the target system as a certain industrial production scheduling system as an example, the method provided by the present invention will be further described.

[0088] ​S1. Security personnel sort out and clarify the mission scenario for conducting confrontation tests using the network range, including the number of confrontation tests N test 、the total number of independent variables NUM in the confrontation test test-x 、the total number of dependent variables NUM in the confrontation test test-y 、the set of independent variables D in the confrontation test test-x 、the set of dependent variables D in the confrontation test test-y 。

[0089] In the security boundary evaluation of the industrial production scheduling system this time, the number of confrontation tests N test is set to 30,000 times.

[0090] Security personnel sorted out 50 independent variables from 5 dimensions: network environment, system configuration, production tasks, external attacks, and data characteristics. The total number of independent variables NUM in the confrontation test test-x is set to 50.

[0091] Security personnel sorted out 20 dependent variables from 5 dimensions: security events, system impacts, vulnerability responses, data security, and comprehensive evaluations. The total number of dependent variables NUM in the confrontation test test-y is set to 20.

[0092] The name X-ID of the independent variable in this test is named according to the format of "unified prefix_core description_unit identifier (if any)". For example, the independent variable "network environment-network latency" is named Net_Latency_ms, and the independent variable "system configuration-operating system version" is named Sys_OSVersion.

[0093] There are 19 continuous variables and 31 discrete variables in the independent variable type X-type of this test.

[0094] The subset of independent variable values X-value limits the value set of each discrete variable in the set of independent variables for the confrontation test. For continuous variables, the subset of independent variable values X-value is not set. For example, the subset of independent variable values X-value of "system configuration-operating system version" = [NeoKylin, Tongxin UOS, Ubuntu22, Windows Server2016].

[0095] The inverse cumulative distribution function of the independent variable X-ICDF is a function that maps the given probability values of different independent variables to the corresponding quantiles, and is obtained using formula expression, rational approximation, Newton iteration method, or look-up table method. For example, the independent variable of "network environment-network latency" is a uniform distribution function in the interval of 10 to 500 ms, and its inverse cumulative distribution function of the independent variable X-ICDF = 10 + 490x.

[0096] In this experiment, the dependent variable name Y-ID is named according to the format of "unified prefix _ core description _ unit identifier (if any)". For example, the dependent variable "Security event _ number of data leaks" is named SecEvent_DataLeak_Count, and the dependent variable "Security event _ system scheduling delay" is named SecEvent_SchedulingLatency_ms.

[0097] There are 10 continuous variables and 10 discrete variables in the dependent variable type Y-type of this experiment.

[0098] S2. Design of operating conditions

[0099] In addition to the large number of independent variables in the confrontation test, the variable types are also diverse: including continuous variables such as network environment delay and discrete variables such as attack types. Therefore, the design of the confrontation test operating conditions needs to consider the sampling tasks of both continuous variables and discrete variables at the same time. The typical Latin hypercube is mostly used for sampling tasks of a single variable type, and there are few Latin hypercube sampling methods that consider multiple variable types at the same time. The improved Latin hypercube sampling method provided by the present invention is used to implement the design of the confrontation test operating conditions. The design process of the confrontation test operating conditions is as Figure 3 shown, which includes two major parts: S21 generation of the standard sample matrix and S22 mapping of the actual operating condition interval. The generation of the S21 standard sample matrix includes sub-steps of S211 independent permutation design, S212 multi-dimensional sample generation, and S213 sample matrix combination; the mapping of the S22 actual operating condition interval includes sub-steps of S221 variable type judgment, S222 equal-width binning mapping, S223 inverse transformation method mapping, and S224 integration of the confrontation test operating condition set.

[0100] Sub-step S211 of independent permutation design: In order to ensure the uniform distribution of samples in the standardized parameter space, in the standardized parameter space [0,1) 50 , for each independent variable k (k = 1, 2,... 50) of the confrontation test, a random permutation is generated, and the random permutation is shown in formula (11):

[0101] π k =[π k (1), π k (2),... π k (30000)] (11)

[0102] Sub-step S212 of multi-dimensional sample generation: It is realized by accumulating random offsets on the basis of the random permutation. The coordinate of the k-th independent variable of the i-th confrontation test operating condition sample point is shown in formula (12):

[0103]

[0104] In formula (12), the random offset obeys the standard uniform distribution.

[0105] The S213 sample matrix combination sub-step can realize the generation of the standard sample matrix, and the standard sample matrix is shown in formula (13):

[0106]

[0107] S221 variable type judgment sub-step: By reading the set D of independent variables for the adversarial test test-x , successively execute the corresponding distribution mapping method according to the independent variable type X-type. For the k-th independent variable of the i-th adversarial test condition sample point, if its independent variable type X-type k is a discrete variable, then execute the equal-width binning mapping sub-step; if it is a continuous variable, then execute the inverse transformation method mapping sub-step.

[0108] S222 equal-width binning mapping sub-step: First, it is necessary to encode the subset X-value of the independent variable values k . Taking the independent variable "system configuration - operating system version" as an example, count the number of elements num(X-value k ) of the subset X-value of this discrete variable is 4. Divide the interval [0, 1) into 4 consecutive interval blocks, and assign each element in the subset X-value k set a separate interval block. Map according to the interval block where the coordinates k of the standard sample matrix fall, and determine the actual condition of the k-th independent variable of the i-th adversarial test condition sample point corresponding to the element of the interval block, as shown in formula (14): In formula (14),

[0109]

[0110] is the actual condition of the k-th independent variable of the i-th adversarial test condition sample point.

[0111] The S223 inverse transformation method mapping sub-step first reads the inverse cumulative distribution function X-ICDF of the independent variable of this continuous variable k . Taking the independent variable "network environment - network latency" as an example, substitute the coordinates of the standard sample matrix into the inverse cumulative distribution function X-ICDF k to obtain the actual condition of the k-th independent variable of the i-th adversarial test condition sample point, as shown in formula (15):

[0112]

[0113] Complete the mapping transformation of the independent variables for the adversarial test for each sample point of the adversarial test working condition in sequence, and the actual working condition sample matrix can be obtained, as shown in formula (16):

[0114]

[0115] The integrator sub-step of the S224 anti-test working condition set integrates the actual working condition sample matrix X actual with the independent variable name X-ID into the adversarial test working condition set X profiles , as shown in formula (17).

[0116]

[0117] S3. According to the actual sample matrix X of the adversarial test working condition actual conduct an adversarial test on the target system, and extract the values of each adversarial test dependent variable from the test results for the safety boundary evaluation of the target system, including:

[0118] S31: Execute the adversarial test

[0119] Taking the adversarial test working condition set as the input, automatically conduct adversarial test experiments under different adversarial test working conditions. By parsing the adversarial test working condition set X profiles 30,000 sample points of the adversarial test working condition can be obtained, and the adversarial test is carried out in sequence according to the independent variables of the adversarial test working condition sample points.

[0120] S32: Process the process data, including:

[0121] S321 Receive the adversarial test experiment data: Receive the adversarial test experiment process data under different adversarial test working conditions in sequence.

[0122] S322 Extract the results of the adversarial test dependent variables: Process and extract all the adversarial test experiment process data with reference to the total number of sorted adversarial test dependent variables and the set of adversarial test dependent variables.

[0123] The integrator sub-step of the adversarial test result set: Integrate the processed and extracted results with the adversarial test working condition set pushed by the adversarial test working condition design module to form the adversarial test result set, as shown in formula (18):

[0124]

[0125] S33: Implement the safety boundary evaluation:

[0126] S331 Safety boundary evaluation and analysis: Use statistical means to conduct safety boundary evaluation and analysis, and sequentially analyze and calculate the statistical characteristics of the distribution of each dependent variable in the set of adversarial test results: The statistical characteristics include the mean, standard deviation, and key point values (minimum value, 25% value, 50% value, 75% value, maximum value). Taking the dependent variable "Safety event_system scheduling delay" as an example, the analysis results of its statistical characteristics are: Mean: 505.35 ms, Standard deviation: 112.51 ms, Key point values (Minimum value: 215.60 ms, 25% value: 411.05 ms, 50% value: 520.18 ms, 75% value: 603.60 ms, Maximum value: 702.54 ms). If the statistical characteristics of the safety boundary do not meet the design indicators, it is considered that the safety boundary does not meet the standard.

[0127] S332 Safety boundary envelope visualization: Use a violin plot to visually display the safety boundary envelope of the target system. Taking the dependent variable "Safety event_system scheduling delay" as an example, its safety boundary envelope is as Figure 6 shown. The evaluation of the safety boundary is achieved through adversarial testing, providing a reference direction for optimizing the safety boundary of the target system.

[0128] The following describes the system safety boundary evaluation system based on adversarial testing provided by the embodiments of the present invention. The system safety boundary evaluation system based on adversarial testing described below can be mutually corresponding and referenced with the system safety boundary evaluation method based on adversarial testing described above.

[0129] The embodiments of the present invention provide a system safety boundary evaluation system based on adversarial testing, including:

[0130] A setting module for setting the number of adversarial tests N of the target system test , the total number of independent variables NUM of the adversarial test test-x , the set of independent variables D of the adversarial test test-x and the set of dependent variables D of the adversarial test test-y ;

[0131] Among them, D test-x includes the independent variable name, independent variable type, subset of discrete independent variable values, and inverse cumulative distribution function of the continuous independent variable; the independent variable type includes discrete and continuous, and the subset of discrete independent variable values includes the value sets of each discrete independent variable; D test-x includes the dependent variable name and dependent variable type, and the dependent variable is the safety index of the target system;

[0132] An adversarial test condition design module for generating random sequences corresponding to each independent variable within the standardized parameter space to form a size of NUM test-x ×Ntest Standard sample matrix X of the adversarial test working condition standard ; Map the standard sample points of the adversarial test working conditions corresponding to the discrete independent variables in X standard to the subset of discrete independent variable values to obtain the corresponding actual sample points of the adversarial test working conditions; Map the standard sample points of the adversarial test working conditions corresponding to the continuous independent variables in X standard to their inverse cumulative distribution functions to obtain the corresponding actual sample points of the adversarial test working conditions.

[0133] The adversarial test working condition design module designs the adversarial test working conditions by using the improved Latin hypercube sampling method proposed by the present invention, generates representative sample points of the adversarial test working conditions in the independent variable parameter space, forms an adversarial test working condition set, and improves the effectiveness of the test results of the adversarial test.

[0134] The network range module is used to perform an adversarial test on the target system according to the actual sample matrix X of the adversarial test working conditions actual ; where X actual is composed of the actual sample points of the adversarial test working conditions of each discrete independent variable and each continuous independent variable.

[0135] The network range module is used to automatically conduct adversarial test experiments under different adversarial test working conditions in the network range.

[0136] The information processing module is used to extract the values of each adversarial test dependent variable from the test results.

[0137] The information processing module processes and extracts the adversarial test dependent variables concerned by security personnel from the process data of the adversarial test experiment, and integrates them with the adversarial test working condition set to form an adversarial test result set for conducting security boundary evaluation.

[0138] The security boundary evaluation module is used to conduct a security boundary evaluation of the target system.

[0139] The security boundary evaluation module uses statistical means to conduct a security boundary evaluation analysis, and can also use a violin plot to visually display the security boundary envelope of the target system, providing a reference direction for optimizing the security boundary of the target system.

[0140] Each is connected in the form of an application programming interface, and the connection and layout relationship are as Figure 6 shown.

[0141] An embodiment of the present invention provides an electronic device, including: a computer-readable storage medium and a processor;

[0142] The computer-readable storage medium is used to store executable instructions;

[0143] The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the method according to any one of the foregoing embodiments.

[0144] An embodiment of the present invention provides a computer-readable storage medium storing computer instructions for causing a processor to execute the method according to any one of the foregoing embodiments.

[0145] Those skilled in the art can easily understand that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for evaluating the security boundary of a system based on adversarial testing, characterized in that Including: S1, Set the number of adversarial tests N for the target system test , the total number of independent variables NUM for the adversarial test test-x , the set of independent variables D for the adversarial test test-x and the set of dependent variables D for the adversarial test test-y ; Among them, D test-x includes the independent variable name, independent variable type, subset of discrete independent variable values, and inverse cumulative distribution function of the continuous independent variable; the independent variable type includes discrete and continuous types, and the subset of discrete independent variable values includes the value sets of each discrete independent variable; D test-x includes the dependent variable name and dependent variable type, and the dependent variable is the security index of the target system; S2. Generate random sequences corresponding to respective independent variables within the standardized parameter space to form an adversarial test condition standard sample matrix X of size NUM test-x ×N test ; Map the adversarial test condition standard sample points corresponding to the discrete independent variables in X standard to the subset of discrete independent variable values to obtain their corresponding adversarial test condition actual sample points; Substitute the adversarial test condition standard sample points corresponding to the continuous independent variables in X standard into their inverse cumulative distribution functions to obtain their corresponding adversarial test condition actual sample points; standard ​ S3. According to the actual sample matrix X of the adversarial test condition actual Conduct an adversarial test on the target system, and extract the values of each adversarial test dependent variable from the test results for the security boundary evaluation of the target system; Among them, X actual It consists of the actual sample points of the confrontation test conditions for each discrete independent variable and each continuous independent variable.

2. The method according to claim 1, wherein If the inverse cumulative distribution function of a certain continuous independent variable is unknown, the historical data of the continuous independent variable in the target system is fitted to obtain its inverse cumulative distribution function.

3. The method according to claim 1, wherein In step S2, in the standardized parameter space after generating random sequences corresponding to respective independent variables and before forming a standard sample matrix of size NUM test-x ×N test , further includes: Adding a random offset to each random sequence; wherein, the added random offset follows a standard uniform distribution.

4. The method according to claim 1, characterized in that In step S3, according to the actual sample matrix X of the adversarial test condition actual Before performing the adversarial test on the target system, it further includes: Integrate the actual working condition sample matrix X actual with the independent variable name X-ID; Before performing the security boundary evaluation of the target system, it further includes: Integrate the values X of each countermeasure test dependent variable actual into the countermeasure test result set.

5. A system security boundary evaluation system based on adversarial testing, characterized in that, Including: A setting module for setting the number of adversarial tests N of the target system test , the total number of independent variables NUM of the adversarial test test-x , the set D of independent variables of the adversarial test test-x and the set D of dependent variables of the adversarial test test-y ; Among them, D test-x includes the independent variable name, independent variable type, subset of discrete independent variable values, and inverse cumulative distribution function of the continuous independent variable; the independent variable type includes discrete and continuous types, and the subset of discrete independent variable values includes the value sets of each discrete independent variable; D test-x includes the dependent variable name and dependent variable type, and the dependent variable is the security index of the target system; Adversarial test condition design module, which is used to generate random sequences corresponding to respective independent variables within the standardized parameter space to form an adversarial test condition standard sample matrix X of size NUM test-x ×N test ; map the adversarial test condition standard sample points corresponding to the discrete independent variables in X standard to the subset of discrete independent variable values to obtain the corresponding adversarial test condition actual sample points; substitute the adversarial test condition standard sample points corresponding to the continuous independent variables in X standard into their inverse cumulative distribution functions to obtain the corresponding adversarial test condition actual sample points; standard ​ A network range module for conducting adversarial tests on a target system according to the actual sample matrix X of the adversarial test working conditions actual wherein X actual is composed of the actual sample points of the adversarial test working conditions of each discrete independent variable and each continuous independent variable; An information processing module for extracting the values of each countermeasure test dependent variable from the test results; A security boundary evaluation module for performing the security boundary evaluation of the target system.

6. An electronic device, characterized in that, Including: A computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the processor to execute the method according to any one of claims 1-4.

Images