An adaptive search and evaluation method for test cases of high-dimensional black-box intelligent systems

Abstract

The present invention discloses a method for adaptive search and evaluation of test cases of a high-dimensional black-box intelligent system, including: generating an initial test design case, obtaining the output response of each initial test design case through a black-box system; locally adaptively generating a point-added test design case based on the initial test design case and its output response, obtaining the output response of each point-added test design case through a black-box system; screening test design cases whose output responses are within a preset range of interest from the initial test design case and the point-added test design case; performing sequential intelligent test design on the screened test case set based on an evolutionary algorithm to obtain an expanded test case set and its output response set; and hierarchically clustering the set based on the test factor space of the set. The present invention improves the adaptive test design capability and test case generation capability for high-dimensional black-box intelligent system testing, and is conducive to selecting representative test design cases.

Description

Technical Field

[0001] The present invention belongs to the technical field of system testing, and in particular relates to an adaptive search and evaluation method for test cases of a high-dimensional black box intelligent system. Background Art

[0002] In recent years, the rapid progress of science and technology has promoted the widespread application of high-dimensional black-box intelligent systems in many fields. The input variables of high-dimensional black-box intelligent systems have high-dimensional characteristics, and the behavioral characteristics of the system can only be explored and analyzed through input and output. The intelligent capabilities of such systems are often learned from data, so there are "data loopholes", and due to the lack of transparency and explainability, their performance and credibility are often questioned, which seriously hinders their application and promotion in key fields. In order to improve the credibility and reliability of black-box intelligent systems, extensive and in-depth testing is required to troubleshoot possible logical errors, functional defects and performance bottlenecks in the system. However, due to the combinatorial explosion characteristics of high-dimensional input space, traditional testing methods face huge challenges in dealing with high-dimensional black-box intelligent systems. How to optimize the design of test cases within an acceptable time, to ensure that the main functional processes of the black box are fully verified, and to take into account its potential high-risk vulnerabilities, has become the core problem faced by the current black-box testing field. In other words, there is an urgent need for a design scheme for test cases of high-dimensional black-box intelligent systems to improve the adaptive test design capability and test case generation capability for high-dimensional black-box intelligent system testing. Summary of the invention

[0003] In order to solve the above problems existing in the prior art, the present invention provides a method for adaptive search and evaluation of test cases of a high-dimensional black-box intelligent system.

[0004] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0005] The present invention provides a high-dimensional black box intelligent system test case adaptive search and evaluation method, comprising:

[0006] Based on the preset test factor space, generating a first preset number of initial experimental design cases, and obtaining an output response of each initial experimental design case through a black box system;

[0007] Based on the preset number of test design cases with added points, the first preset number of initial test design cases and the output response of each initial test design case, locally and adaptively generate multiple test design cases with added points, and obtain the output response of each test design case with added points through the black box system;

[0008] From the set consisting of the first preset number of initial test design cases and the plurality of added test design cases, test design cases whose output responses are within a preset range of interest are selected to obtain a test case set. And the test case set The output response set ;

[0009] Based on the evolutionary algorithm, the test case set Performing sequential intelligent experimental design to search for more experimental design cases whose output responses are within the preset range of interest, and obtaining an expanded set of experimental cases and the expanded test case set The output response set ;

[0010] Based on the expanded test case set The test factor space and the output response set , for the expanded test case set Perform hierarchical clustering to obtain multiple clustered use case sets.

[0011] In some embodiments, locally adaptively generating multiple added-point test design cases based on the preset number of added-point test design cases, the first preset number of initial test design cases, and the output response of each initial test design case, and obtaining the output response of each added-point test design case through the black box system, includes:

[0012] The first The value intervals of the dimensions are evenly divided into subintervals, and get an initial tensor , where the initial tensor correspond sub-interval combinations, the preset test factor space is composed of multiple variables, and the multiple variables are multiple dimensions of the preset test factor space, represents the total number of dimensions in the preset test factor space, is a positive integer, The value range is 1 to ;

[0013] According to the values ​​of the variables in each initial experimental design case, determine the initial experimental design case that falls within each sub-interval combination;

[0014] Calculate the severity of the change of the output response of the initial test design case falling within each sub-interval combination, and obtain the severity of the change value of each sub-interval combination;

[0015] Based on the change severity value of each sub-interval combination and the preset number of test design cases for adding points, the number of test design cases for adding points required for each sub-interval combination is generated;

[0016] According to the number of point-added test design cases required for each sub-interval combination, a corresponding number of point-added test design cases for each sub-interval combination is generated;

[0017] The output response of each added-point test design case generated within each sub-interval combination is obtained through the black box system.

[0018] In some embodiments, The expression for the number of test cases required for the combination of subintervals is as follows:

[0019] ;

[0020] in, Indicates The number of test cases required for the combination of sub-intervals. , Represents the initial tensor The total number of corresponding sub-interval combinations, Indicates the The change intensity value of the sub-interval combination is Indicates The change intensity value of the sub-interval combination is Indicates the number of cases for the preset point-added test design.

[0021] In some embodiments, calculating the severity of the change in the output response of the initial experimental design case falling within each sub-interval combination to obtain the severity of the change value of each sub-interval combination includes:

[0022] The standard deviation of the output response of the initial experimental design case falling within each sub-interval combination is calculated as the severity of the change in the output response of the initial experimental design case falling within each sub-interval combination, and the severity of the change value of each sub-interval combination is obtained.

[0023] In some embodiments, the test case set is analyzed based on the evolutionary algorithm. Performing sequential intelligent experimental design to search for more experimental design cases whose output responses are within the preset range of interest, and obtaining an expanded set of experimental cases and the expanded test case set The output response set ,include:

[0024] Generate an initial population;

[0025] In each iteration, based on the set of test cases obtained in the previous iteration And the set of test cases obtained in the previous iteration The output response set , and the population obtained in the previous iteration, select the target individual from the population obtained in the previous iteration, and calculate the output response of the target individual; wherein, in the first iteration, the population obtained in the previous iteration is the initial population, wherein, Represents the last time, On behalf of this;

[0026] Add the target individual to the set of test cases obtained in the previous iteration In this iteration, we get the set of test cases obtained. , and adding the output response of the target individual to the set of test cases obtained in the previous iteration The output response set In the above example, we get the set of test cases obtained in this iteration. The output response set , and determine the population obtained in this iteration based on the population obtained in the previous iteration, so as to continue the next iteration until the iteration termination condition is reached, and the expanded test case set is obtained. and the expanded test case set The output response set .

[0027] In some embodiments, the test case set obtained based on the previous iteration And the set of test cases obtained in the previous iteration The output response set , and the population obtained in the previous iteration, screening the target individual from the population obtained in the previous iteration, and calculating the output response of the target individual, including:

[0028] The test case set obtained in the previous iteration The output response set Arrange the elements in descending order to get the sequence , calculate the sequence The maximum interval between adjacent output responses ;

[0029] Calculate the output response of each individual in the population obtained in the previous iteration, and add the output response of one individual to the sequence according to its size each time Then calculate the maximum interval between adjacent output responses of the newly obtained sequence , and after each calculation, the output response added each time is taken out from the newly obtained sequence to obtain the maximum interval corresponding to each individual ;

[0030] The set of test cases obtained from the previous iteration Find the k nearest neighbors of each individual and calculate the average distance between each individual and the k nearest neighbors ;

[0031] According to the maximum interval , the maximum interval corresponding to each individual , and the average distance between each individual and its k nearest neighbors , calculate the fitness of each individual ;

[0032] The fitness of the population obtained in the previous iteration The largest individual is taken as the target individual, and the output response of the target individual is calculated.

[0033] In some embodiments, the fitness of each individual The expression is as follows:

[0034] ;

[0035] in, is the preset coefficient.

[0036] In some embodiments, the iteration termination condition includes: the number of iterations reaches a preset number or the fitness of each individual are all less than the preset threshold.

[0037] In some embodiments, the expanded test case set The variables in the experimental design cases in the abovementioned set of expanded experimental cases The test factor space; the test case set based on the expanded The test factor space and the output response set , for the expanded test case set Perform hierarchical clustering to obtain multiple clustered use case sets, including:

[0038] Using the set output response interval, according to the output response set , for the expanded test case set Divide and obtain a variety of test design use case sets;

[0039] According to the values ​​of the variables in the experimental design use cases in each experimental design use case set, each experimental design use case set is clustered based on density to obtain multiple experimental design use case sets under each experimental design use case set.

[0040] In some embodiments, generating a first preset number of initial experimental design cases based on a preset test factor space, and obtaining an output response of each initial experimental design case through a black box system includes:

[0041] Read in the value range and value type of each variable in the preset test factor space, wherein the preset test factor space is composed of multiple variables, and the multiple variables are multiple dimensions of the preset test factor space;

[0042] According to the read-in value range and value type, and the first preset number, generating the first preset number of initial experimental design cases by using the experimental design case generation method;

[0043] Each initial experimental design case is input into the black box system to obtain an output response for each initial experimental design case.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The adaptive search and evaluation method for high-dimensional black-box intelligent system test cases proposed in the present invention greatly improves the adaptive test design capability and test case generation capability for high-dimensional black-box intelligent system testing by exploring the input-output spatial response distribution of the black-box system while adjusting the test design direction. Moreover, the present invention combines testing, optimization and sample analysis, focuses on the exploration, feedback and adjustment of the test space of the high-dimensional black-box intelligent system by the test case design, strengthens the adaptive exploration of the test factor space and its output response distribution, and can be better applied to application scenarios that need to test the distribution of complex systems; in addition, since the present invention supports sequential intelligent sampling search of test design cases in the area of ​​interest, it is particularly suitable for local and key area test testing of black-box systems. In addition, the present invention facilitates the selection of representative test design cases that take into account the distribution of the test factor space and the output response space through hierarchical clustering of test design cases.

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of a method for adaptive search and evaluation of test cases of a high-dimensional black-box intelligent system provided by an embodiment of the present invention;

[0048] Figure 2is a distribution diagram of a first preset number of initial experimental design use cases in a three-dimensional space provided by an embodiment of the present invention;

[0049] Figure 3 The embodiment of the present invention provides Figure 2 After the first preset number of initial test design cases are locally adaptively generated into a plurality of added-point test design cases, a distribution diagram of all the generated added-point test design cases in three-dimensional space;

[0050] Figure 4 The embodiment of the present invention provides an evolutionary algorithm for Figure 2 and Figure 3 After sequential intelligent test design is performed on the test case set in the 3D space, the distribution diagram of the expanded test case set is obtained;

[0051] Figure 5 is the value of the variable in the test design case provided by the embodiment of the present invention, Figure 4 The two-dimensional distribution diagram of several clusters obtained after density-based clustering of the experimental design cases whose output responses in the experimental case set are within the set output response interval of [5,7] is shown;

[0052] Figure 6 is the value of the variable in the test design case provided by the embodiment of the present invention, Figure 4 The three-dimensional distribution diagram of several cluster clusters obtained after density-based clustering of this type of experimental design cases in the experimental case set whose output responses are within the set output response interval of [5,7] is shown. DETAILED DESCRIPTION

[0053] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0054] Figure 1 is a flow chart of a method for adaptive search and evaluation of a high-dimensional black-box intelligent system test case provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0055] S101. Generate a first preset number of initial experimental design cases based on a preset test factor space, and obtain an output response of each initial experimental design case through a black box system.

[0056] S102. Based on a preset number of added-point test design cases, a first preset number of initial test design cases and an output response of each initial test design case, locally and adaptively generate multiple added-point test design cases, and obtain the output response of each added-point test design case through a black box system.

[0057] S103, from a set consisting of a first preset number of initial experimental design cases and a plurality of added experimental design cases, screen out experimental design cases whose output responses are within a preset range of interest, to obtain an experimental case set and the output response set of the test case set .

[0058] S104. Experimental case set based on evolutionary algorithm Perform sequential intelligent experimental design to search for more experimental design cases whose output responses are within the preset range of interest, and obtain an expanded set of experimental cases And the expanded set of test cases The output response set .

[0059] S105, based on the expanded test case set The test factor space and output response set of , for the expanded set of test cases Perform hierarchical clustering to obtain multiple clustered use case sets.

[0060] The adaptive search and evaluation method for high-dimensional black-box intelligent system test cases proposed in the present invention greatly improves the adaptive test design capability and test case generation capability for high-dimensional black-box intelligent system testing by exploring the input-output spatial response distribution of the black-box system while adjusting the test design direction. Moreover, the present invention combines testing, optimization and sample analysis, focuses on the exploration, feedback and adjustment of the test space of the high-dimensional black-box intelligent system by the test case design, strengthens the adaptive exploration of the test factor space and its output response distribution, and can be better applied to application scenarios that need to test the distribution of complex systems; in addition, since the present invention supports sequential intelligent sampling search of test design cases in the area of ​​interest, it is particularly suitable for local and key area test testing of black-box systems. In addition, the present invention facilitates the selection of representative test design cases that take into account the distribution of the test factor space and the output response space through hierarchical clustering of test design cases.

[0061] In some embodiments, the above S101 is implemented by the following steps:

[0062] S1011. Read in the value range and value type of each variable in the preset test factor space, wherein the preset test factor space is composed of multiple variables, and the multiple variables are multiple dimensions of the preset test factor space.

[0063] Here, the preset test factor space represents the set of factors that affect the black box system. The preset test factor space can be represented by The variables consist of is a positive integer, and The specific value of can be set according to actual needs, and each variable is a dimension of the preset test factor. Each variable has a preset value range and a value type, and the value range and value type are also set according to actual needs. For example, the value type can be integer, double, binary, etc.

[0064] S1012. Generate a first preset number of initial experimental design cases through an experimental design case generation method according to the read-in value range and value type, as well as the first preset number.

[0065] Here, the specific value of the first preset number can be set according to actual needs; and the first preset number of initial experimental design cases can be generated by using methods such as uniform experimental design or orthogonal experimental design, wherein each initial experimental design case can include variables, and each variable has a value.

[0066] S1013. Input each initial experimental design use case into the black box system to obtain an output response for each initial experimental design use case.

[0067] For example, Figure 2 When the preset test factor space is a two-dimensional space and the black box system is a Levy03 Reverse function, a distribution diagram of the first preset number of initial experimental design cases in the three-dimensional space formed by the two-dimensional preset test factor space and the one-dimensional output response. Figure 2 In the formula, x1 represents one dimension of the two-dimensional preset test factor space, x2 represents the other dimension of the two-dimensional preset test factor space, f(x,y) represents the output response, and, Figure 2 Each red dot in represents an initial experimental design use case. Figure 2 The different colored surfaces in represent the surfaces formed by the initial experimental design cases with different output responses, and the larger the output response of the initial experimental design case, the warmer the color of the formed surface. For example, Figure 2 The output response of the initial experimental design cases that form the yellow surface is larger, and the output response of the initial experimental design cases that form the purple surface is the smallest.

[0068] In some embodiments, the above S102 is implemented by the following steps:

[0069] S1021, set the first The value intervals of the dimensions are evenly divided into intervals, and get an initial tensor , where the initial tensor correspond The preset test factor space is composed of multiple variables, and the multiple variables are multiple dimensions of the preset test factor space. represents the total number of dimensions in the preset test factor space, is a positive integer, The value range is 1 to .

[0070] Here, for the value interval of each dimension, when the dimension is more sensitive, the value interval of the dimension can be divided finer to divide into more sub-intervals, and when the dimension is insensitive, the value interval of the dimension can be divided coarser to divide into fewer sub-intervals; wherein, when the value of a dimension changes slightly, the output response corresponding to the value changes significantly, which means that the dimension is more sensitive, and when the value of a dimension changes slightly, the output response corresponding to the value changes very little, which means that the dimension is insensitive. On this basis, whether a dimension is sensitive can be determined according to a specifically set change threshold, and the present invention is not limited to this.

[0071] Here, the value interval of each dimension of the preset test factor space is evenly divided to obtain multiple sub-intervals, and by combining the obtained sub-intervals, we can get subinterval combinations, and each subinterval combination can be expressed as ,in, Indicates Dimensions A subinterval in a subinterval.

[0072] S1022. Determine the initial experimental design use cases that fall within each sub-interval combination based on the values ​​of the variables in each initial experimental design use case.

[0073] Here, for an initial experimental design case, when the value of each variable included in the initial experimental design case falls into a corresponding sub-interval in a sub-interval combination, it means that the initial experimental design case falls into the sub-interval combination.

[0074] S1023. Calculate the severity of the change in the output response of the initial test design use case that falls within each sub-interval combination, and obtain a severity of change value for each sub-interval combination.

[0075] For example, the standard deviation of the output response of the initial experimental design case falling within each sub-interval combination can be calculated as the degree of change in the output response of the initial experimental design case falling within the sub-interval combination. In this way, the degree of change value of each sub-interval combination can be obtained.

[0076] S1024. Design of test cases based on the severity of change of each sub-interval combination and the number of preset additional test cases , the number of experimental design cases required to generate each subinterval combination.

[0077] For example, The expression for the number of test cases required for the combination of subintervals is as follows:

[0078] ;

[0079] in, Indicates The number of test cases required for the combination of sub-intervals. , Represents the initial tensor The total number of corresponding sub-interval combinations, that is , Indicates The change intensity value of the sub-interval combination is Indicates The change intensity value of the sub-interval combination is Indicates the number of cases for the preset experimental design with added points.

[0080] S1025. Generate a corresponding number of added-point test design cases for each sub-interval combination according to the number of added-point test design cases required for each sub-interval combination.

[0081] Specifically, for each sub-interval combination, when the number of test design cases required for the sub-interval combination is When , each sub-interval in the sub-interval combination can be used as the value range of a corresponding variable in the above-mentioned preset test factor space, and according to the value type of each variable, the method of generating initial test design cases is adopted to generate the corresponding sub-interval combination a test design case and An experimental design case is used as a point-added experimental design case for this sub-interval combination.

[0082] S1026. Obtain the output response of each added-point test design case generated in each sub-interval combination through the black box system.

[0083] In the present invention, the preset range of interest in the above S103 can be expressed as Moreover, the preset range of interest can be set according to actual needs, and the present invention is not limited to this.

[0084] The present invention can achieve the purpose of adding more test cases in the area with drastic changes in the distribution space of the first preset number of initial test design cases through the above S102. For example, Figure 3 Yes Figure 2 After the first preset number of initial test design cases are locally adaptively generated into multiple test design cases with added points, the distribution diagram of all the test design cases with added points generated in three-dimensional space is shown, and Figure 2 Similarly, the x dimension of the three-dimensional space is x1, the y dimension is x2, the z dimension is the output response, and the black box system is the Levy03 Reverse function. Figure 3 Each point in represents a design of experiment case, and, Figure 3 The points of different colors in the figure represent the experimental design cases with output responses in different ranges. Moreover, the larger the output response, the warmer the color of the corresponding point. For example, Figure 3 The output response of the yellow point is larger, and the output response of the purple point is the smallest.

[0085] In some embodiments, the above S104 is implemented by the following steps:

[0086] S1041. Generate an initial population.

[0087] Specifically, the value range and value type of each variable in the preset test factor space can be read in, and a second preset number of experimental design cases can be regenerated according to the read value range and value type; then, each regenerated experimental design case is encoded to obtain a second preset number of codes; each code in the second preset number of codes is taken as an individual to obtain an initial population with a second preset number of individuals. It should be noted that the specific value of the second preset number can be set according to actual needs. It should be noted that the encoding method can be selected according to actual needs. For example, a binary encoding method can be used to encode each experimental design case into a binary code.

[0088] S1042. In each iteration, based on the test case set obtained in the previous iteration And the set of test cases obtained in the previous iteration The output response set , and the population obtained in the previous iteration, select the target individual from the population obtained in the previous iteration, and calculate the output response of the target individual; where, in the first iteration, the population obtained in the previous iteration is the initial population, where, Represents the last time, Representing this time.

[0089] Specifically, S1042 can be implemented through steps S10 to S14:

[0090] S10, the test case set obtained in the previous iteration The output response set Arrange the elements in descending order to get the sequence , calculate the sequence The maximum interval between adjacent output responses .

[0091] Specifically, , Representation sequence The Output response, Representation sequence The Output response, , Representation sequence The total number of output responses in .

[0092] S11. Calculate the output response of each individual in the population obtained in the previous iteration, and add the output response of one individual to the sequence according to its size each time. Then calculate the maximum interval between adjacent output responses of the newly obtained sequence , and after each calculation, the output response added each time is taken out from the newly obtained sequence to obtain the maximum interval corresponding to each individual .

[0093] For example, when calculating the maximum interval corresponding to the second individual in the population obtained in the previous iteration, When , the output response of the second individual is added to the sequence according to the size After that, a new sequence in descending order is obtained, and then the maximum interval between adjacent output responses of the new sequence is calculated. , the maximum interval As the maximum interval corresponding to the second individual , then take the output response of the second individual from the new sequence to get the original sequence After that, the same method is used to calculate the maximum interval corresponding to the next individual in the population obtained in the previous iteration. , so we can get the maximum interval corresponding to each individual in the population obtained in the previous iteration .

[0094] In some embodiments, after obtaining the above test case set and a collection of test cases The output response set Afterwards, the method provided by the present invention further comprises the steps of:

[0095] S20. Use a collection of test cases and a collection of test cases The output response set , train the selected proxy model so that the proxy model learns the set of test cases The input-output mapping relationship is obtained to obtain the trained proxy model; for example, the proxy model can be a 3-layer neural network.

[0096] S21. Compare the output response generation speed of the trained proxy model and the above black box system;

[0097] S22. When the output response generation speed of the black box system is faster than the output response generation speed of the trained proxy model, the black box system is used to generate the output response of each individual in the population obtained in the previous iteration.

[0098] S23. When the output response generation speed of the trained proxy model is better than the output response generation speed of the black box system, the trained proxy model is used to generate the output response of each individual in the population obtained in the previous iteration.

[0099] The maximum interval corresponding to each individual is obtained through the above S11 Then, continue to execute S12:

[0100] S12. A set of test cases obtained from the previous iteration Find the k nearest neighbors of each individual and calculate the average distance between each individual and the k nearest neighbors .

[0101] Here, we can use the k-nearest neighbor algorithm to obtain the set of test cases from the previous iteration according to the value of the variable in each individual. Find the k neighbor cases of each individual, and then calculate the average distance between each individual and its k neighbor cases based on the values ​​of the variables of each individual and its k neighbor cases .

[0102] S13, according to the maximum interval , the maximum interval corresponding to each individual , and the average distance between each individual and its k nearest neighbors , calculate the fitness of each individual .

[0103] Here, the fitness of each individual The expression is as follows:

[0104] ;

[0105] in, is the preset coefficient.

[0106] S14, the fitness of the population obtained in the previous iteration The largest individual is taken as the target individual, and the output response of the target individual is calculated.

[0107] Here, the method for calculating the output response of each target individual is the same as the method for calculating the output response of each individual in the population obtained in the previous iteration.

[0108] S1043. Add the target individual to the set of test cases obtained in the previous iteration. In this iteration, we get the set of test cases obtained. , and add the output response of the target individual to the set of test cases obtained in the previous iteration The output response set In this iteration, we get the set of test cases obtained. The output response set , and determine the population obtained in this iteration based on the population obtained in the previous iteration to continue the next iteration, and so on, until the iteration termination condition is reached, the iteration is stopped, so that the expanded test case set can be obtained And the expanded set of test cases The output response set .

[0109] Here, when determining the population obtained in this iteration based on the population obtained in the previous iteration, the individuals in the population obtained in the previous iteration may be crossed and / or mutated to generate new individuals, thereby obtaining the population obtained in this iteration. For example, when each experimental design case is encoded as a binary code in a binary encoding manner, when crossing two binary codes, the code values ​​of some positions in the two binary codes may be interchanged to generate two new binary codes, and when mutating each binary code, the code values ​​of some positions in the binary code may be changed to generate a new binary code.

[0110] Here, we get the set of test cases obtained in this iteration. , output response set After the population obtained in this iteration, the same principle as above can be used to continue the next iteration until the number of iterations reaches the preset number or the fitness of each individual is less than the preset threshold, indicating that the iteration termination condition is reached, so the iteration can be stopped and the test case set obtained in the last iteration is used as the expanded test case set. , the output response set of the test case set obtained in the last iteration is used as the expanded test case set The output response set . For example, Figure 4 The evolutionary algorithm is used to Figure 2 and Figure 3 A collection of test cases in After sequential intelligent test design, the set of test cases obtained is The distribution diagram in three-dimensional space, and Figure 3 Similarly, the x dimension of the three-dimensional space is x1, the y dimension is x2, the z dimension is the output response, and the black box system is the Levy03 Reverse function. Figure 4 Each point in also represents a design of experiment use case, and, Figure 4 The points of different colors in the same figure represent the experimental design cases with output responses in different ranges. Moreover, the larger the output response, the warmer the color of the corresponding point. For example, Figure 4 The output response of the yellow point is larger, and the output response of the purple point is the smallest.

[0111] In some embodiments, the above S105 can be implemented by the following steps:

[0112] S1051, using the set output response interval, according to the output response set , for the expanded set of test cases Divide and obtain a variety of experimental design use case sets.

[0113] Here, we can use several different output response intervals to expand the test case set The corresponding test design use case sets are divided into several different sets, so as to facilitate the selection of test design use cases with representative output response space distribution. It should be noted that the output response interval used for division can be set according to actual needs.

[0114] S1052. Perform density-based clustering on each experimental design case set according to the values ​​of the variables in the experimental design case in each experimental design case set to obtain multiple experimental design case sets under each experimental design case set; in this way, multiple clustered case sets are obtained through clustering.

[0115] Here, a density-based clustering method can be used to cluster each set of experimental design use cases, thereby obtaining multiple sets of experimental design use cases under each set of experimental design use cases, thereby facilitating the selection of representative experimental design use cases that take into account the distribution of both the test factor space and the output response space.

[0116] For example, Figure 5 and Figure 6 are the values ​​of the variables in the experimental design cases. Figure 4The test case set shown After performing density-based clustering on this type of experimental design case whose output response is within the set output response interval [5,7], the two-dimensional and three-dimensional distribution diagrams of several cluster clusters are obtained, and, Figure 5 and Figure 6 In the above example, feature 1 represents the variable x1 in the x dimension, and feature 2 represents the variable x2 in the y dimension. Figure 6 The z dimension in is the output response; Figure 5 and Figure 6 In the figure, each point represents an experimental design use case, points of different colors represent different clusters to which they belong, and different clusters represent different sets of experimental design use cases.

[0117] The high-dimensional black-box intelligent system test case adaptive search and evaluation method proposed in the present invention has the following advantages:

[0118] 1) The present invention effectively combines traditional experimental design methods, adaptive point-adding design methods, evolutionary search methods and agent models to gradually explore the test factor space and output response space, dynamically adjust the experimental design case generation strategy, and continuously add filling experimental design cases, thereby alleviating the problems of redundant testing and combination explosion.

[0119] 2) The present invention proposes a local adaptive point-adding design method based on the response sensitivity of the use case output, which can increase the number of experimental design cases in the area of ​​​​rapid response fluctuations in the test factor space and enhance the control of the input and output distribution of the black box system.

[0120] 3) The present invention designs a fitness function that takes into account the fillingness of the test factor space and the output response space, which can promote the evolutionary algorithm to strengthen the design of test cases in the sparse area of ​​test factor space and output response space.

[0121] 4) The present invention proposes a hierarchical clustering method based on the output response space and the test factor space, which can screen out representative experimental design use cases from a large number of experimental design use cases, and can more comprehensively cover key areas and high-risk scenarios with limited test resources.

[0122] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0123] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0124] In the specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality of components. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good effects.

[0125] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A method for adaptive search and evaluation of test cases for high-dimensional black-box intelligent systems, characterized in that: include: Based on the preset test factor space, generating a first preset number of initial experimental design cases, and obtaining an output response of each initial experimental design case through a black box system; Based on the preset number of test design cases with added points, the first preset number of initial test design cases and the output response of each initial test design case, locally and adaptively generate multiple test design cases with added points, and obtain the output response of each test design case with added points through the black box system; From the set consisting of the first preset number of initial test design cases and the plurality of added test design cases, test design cases whose output responses are within a preset range of interest are selected to obtain a test case set. And the test case set The output response set ; Based on the evolutionary algorithm, the test case set Performing sequential intelligent experimental design to search for more experimental design cases whose output responses are within the preset range of interest, and obtaining an expanded set of experimental cases and the expanded test case set The output response set ; Based on the expanded test case set The test factor space and the output response set , for the expanded test case set Perform hierarchical clustering to obtain multiple clustered use case sets.

2. The high-dimensional black-box intelligent system test case adaptive search and evaluation method according to claim 1 is characterized in that: The method of locally adaptively generating a plurality of added-point test design cases based on the preset number of added-point test design cases, the first preset number of initial test design cases, and the output response of each initial test design case, and obtaining the output response of each added-point test design case through the black box system, includes: The first The value intervals of the dimensions are evenly divided into subintervals, and get an initial tensor , where the initial tensor correspond sub-interval combinations, the preset test factor space is composed of multiple variables, and the multiple variables are multiple dimensions of the preset test factor space, represents the total number of dimensions in the preset test factor space, is a positive integer, The value range is 1 to ; According to the values ​​of the variables in each initial experimental design case, determine the initial experimental design case that falls within each sub-interval combination; Calculate the severity of the change of the output response of the initial test design case falling within each sub-interval combination, and obtain the severity of the change value of each sub-interval combination; Based on the change severity value of each sub-interval combination and the preset number of test design cases with added points, the number of test design cases with added points required for each sub-interval combination is generated; According to the number of point-added test design cases required for each sub-interval combination, a corresponding number of point-added test design cases for each sub-interval combination is generated; The output response of each added-point test design case of each sub-interval combination is obtained through the black box system.

3. The high-dimensional black-box intelligent system test case adaptive search and evaluation method according to claim 2 is characterized in that: No. The expression for the number of test cases required for the combination of subintervals is as follows: ； in, Indicates The number of test cases required for the combination of sub-intervals. , Represents the initial tensor The total number of corresponding sub-interval combinations, Indicates the The change intensity value of the sub-interval combination is Indicates The change intensity value of the sub-interval combination is Indicates the number of cases for the preset point-added test design.

4. The method for adaptive search and evaluation of test cases for high-dimensional black-box intelligent systems according to claim 2 is characterized in that: The calculation of the severity of the change of the output response of the initial test design case falling within each sub-interval combination to obtain the severity of the change value of each sub-interval combination includes: The standard deviation of the output response of the initial experimental design case falling within each sub-interval combination is calculated as the severity of the change in the output response of the initial experimental design case falling within each sub-interval combination, and the severity of the change value of each sub-interval combination is obtained.

5. The high-dimensional black-box intelligent system test case adaptive search and evaluation method according to claim 1 is characterized in that: The test case set is analyzed based on the evolutionary algorithm Performing sequential intelligent experimental design to search for more experimental design cases whose output responses are within the preset range of interest, and obtaining an expanded set of experimental cases and the expanded test case set The output response set ,include: Generate an initial population; In each iteration, based on the set of test cases obtained in the previous iteration And the set of test cases obtained in the previous iteration The output response set , and the population obtained in the previous iteration, select the target individual from the population obtained in the previous iteration, and calculate the output response of the target individual; wherein, in the first iteration, the population obtained in the previous iteration is the initial population, wherein, Represents the last time, On behalf of this; Add the target individual to the set of test cases obtained in the previous iteration In this iteration, we get the set of test cases obtained. , and adding the output response of the target individual to the set of test cases obtained in the previous iteration The output response set In the above example, we get the set of test cases obtained in this iteration. The output response set , and determine the population obtained in this iteration based on the population obtained in the previous iteration, so as to continue the next iteration until the iteration termination condition is reached, and the expanded test case set is obtained. and the expanded test case set The output response set .

6. The method for adaptive search and evaluation of test cases for high-dimensional black-box intelligent systems according to claim 5 is characterized in that: The test case set based on the previous iteration And the set of test cases obtained in the previous iteration The output response set , and the population obtained in the previous iteration, screening the target individual from the population obtained in the previous iteration, and calculating the output response of the target individual, including: The test case set obtained in the previous iteration The output response set Arrange the elements in descending order to get the sequence , calculate the sequence The maximum interval between adjacent output responses ; Calculate the output response of each individual in the population obtained in the previous iteration, and add the output response of one individual to the sequence according to its size each time Then calculate the maximum interval between adjacent output responses of the newly obtained sequence , and after each calculation, the output response added each time is taken out from the newly obtained sequence to obtain the maximum interval corresponding to each individual ; The set of test cases obtained from the previous iteration Find the k nearest neighbors of each individual and calculate the average distance between each individual and the k nearest neighbors ; According to the maximum interval , the maximum interval corresponding to each individual , and the average distance between each individual and its k nearest neighbors , calculate the fitness of each individual ; The fitness of the population obtained in the previous iteration The largest individual is taken as the target individual, and the output response of the target individual is calculated.

7. The method for adaptive search and evaluation of test cases for high-dimensional black-box intelligent systems according to claim 6 is characterized in that: The fitness of each individual The expression is as follows: ； in, is the preset coefficient.

8. The high-dimensional black-box intelligent system test case adaptive search and evaluation method according to claim 6 is characterized in that: The iteration termination conditions include: the number of iterations reaches a preset number or the fitness of each individual are all less than the preset threshold.

9. The high-dimensional black-box intelligent system test case adaptive search and evaluation method according to claim 1 is characterized in that: The expanded test case set The variables in the experimental design cases in the abovementioned set of expanded experimental cases The test factor space; the test case set based on the expanded The test factor space and the output response set , for the expanded test case set Perform hierarchical clustering to obtain multiple clustered use case sets, including: Using the set output response interval, according to the output response set , for the expanded test case set Divide and obtain a variety of test design use case sets; According to the values ​​of the variables in the experimental design use cases in each experimental design use case set, each experimental design use case set is clustered based on density to obtain multiple experimental design use case sets under each experimental design use case set.

10. The high-dimensional black-box intelligent system test case adaptive search and evaluation method according to claim 1 is characterized in that: The step of generating a first preset number of initial test design cases based on a preset test factor space, and obtaining an output response of each initial test design case through a black box system includes: Read in the value range and value type of each variable in the preset test factor space, wherein the preset test factor space is composed of multiple variables, and the multiple variables are multiple dimensions of the preset test factor space; According to the read-in value range and value type, and the first preset number, generating the first preset number of initial experimental design cases by using the experimental design case generation method; Each initial experimental design case is input into the black box system to obtain an output response for each initial experimental design case.

Images