A performance testing method and device of an intelligent algorithm platform

Abstract

The application relates to a performance test method and device of an intelligent algorithm platform, and belongs to the field of software testing. The method comprises the following steps: constructing a text sequence dataset, and generating corresponding adversarial sequence datasets based on each text sequence by using an improved genetic algorithm adversarial sample generation method; respectively running a text classification model to be trained on a benchmark intelligent algorithm platform and an intelligent algorithm platform to be tested to obtain a benchmark intelligent algorithm platform performance index value and a performance test result of the intelligent algorithm platform to be tested; using a text classification model trained on the intelligent algorithm platform to be tested to classify and predict the adversarial sequence datasets to obtain an adversarial prediction result; and obtaining a performance evaluation result of the intelligent algorithm platform to be tested based on the benchmark intelligent algorithm platform performance index value, the performance test result of the intelligent algorithm platform to be tested and the adversarial prediction result. The method solves the problem that existing tools cannot evaluate the adversarial robustness of an intelligent algorithm model trained by a platform.

Description

Technical Field

[0001] This invention belongs to the field of software testing, and in particular relates to a performance testing method and apparatus for an intelligent algorithm platform. Background Technology

[0002] With the maturity of technologies such as big data, machine learning, and neural network models, various intelligent algorithm models have been widely applied to tasks such as news recommendation, facial recognition, and fault detection. These tasks typically require the operation of an intelligent algorithm platform. An intelligent algorithm platform is a comprehensive software or hardware platform that integrates multiple intelligent algorithms and related tools, designed to provide powerful computing capabilities, rich data processing capabilities, and easy-to-use interfaces to support various intelligent tasks and applications. The application of intelligent algorithm models usually includes two stages: the model training stage and the model inference and prediction stage. In the model training stage, data preprocessing is first performed, loading, cleaning, and extracting features from the raw data to provide high-quality input for the model. Then, the model's parameters and weights are optimized using labeled samples. After model training is complete, the inference stage begins. The intelligent algorithm platform loads the pre-trained model parameters and prepares for prediction. For new input samples, the model utilizes the learned features and knowledge to quickly provide prediction results. The inference stage is the key stage for applying intelligent algorithm models in real-world scenarios; it can respond to user needs in real time and provide accurate predictions and decision support in a short period of time.

[0003] Currently, both domestic and international academic and industrial communities have proposed tools such as TPCx-AI and MLPerf AI Benchmark to evaluate the training and inference performance of intelligent algorithm platforms (software and hardware). Existing performance testing methods and tools for intelligent algorithm platforms have two main problems: First, the data labels for model training and testing are manually labeled, which is highly subjective and results in a lack of test predictions. Furthermore, current performance benchmark tools do not consider the impact of random initialization on model scoring, making it difficult to quantitatively compare and evaluate the correctness of models across different scenarios. Second, current testing tools do not support methods for generating intelligent adversarial examples in black-box scenarios, making it difficult to evaluate the adversarial robustness of the intelligent algorithm models trained on the platform; they can only support time performance testing for training or inference. Summary of the Invention

[0004] Based on the above analysis, the present invention aims to provide a performance testing method and tool for intelligent algorithm platforms, which is used to evaluate the performance of the intelligent platform algorithm under test from multiple aspects such as data loading, model training, model inference and model adversarial robustness.

[0005] On the one hand, the present invention provides a performance testing method for an intelligent algorithm platform, specifically including the following steps: A text sequence dataset is constructed, and an adversarial example generation method based on each text sequence is used to generate corresponding adversarial sequences, resulting in an adversarial sequence dataset. Based on the text sequence dataset, the text classification model to be trained was run on the benchmark intelligent algorithm platform and the intelligent algorithm platform to be tested, respectively, to obtain the performance index value of the benchmark intelligent algorithm platform and the performance test results of the intelligent algorithm platform to be tested. The text classification model trained on the intelligent algorithm platform under test is used to classify and predict the adversarial sequence dataset to obtain the adversarial prediction result. The performance evaluation result of the intelligent algorithm platform under test is obtained based on the performance index value of the benchmark intelligent algorithm platform, the performance test result of the intelligent algorithm platform under test, and the adversarial prediction result.

[0006] Furthermore, the adversarial example generation method using the improved genetic algorithm to generate corresponding adversarial sequences based on each of the text sequences includes performing the following operations on each of the text sequences: By replacing words at certain positions in the text sequence with synonyms, an initial adversarial sequence is obtained, which satisfies the same classification decision as the text sequence. At each replacement position, try to replace the synonym with the original word or retain the synonym, so that the intermediate adversarial sequence obtained after traversing all replacement positions and replacing or retaining the synonym satisfies the adversarial criterion; Based on an improved genetic algorithm, the synonyms in the intermediate adversarial sequence are replaced back with the original words or other synonyms to obtain a corresponding adversarial sequence that is semantically similar to the text sequence.

[0007] Furthermore, based on an improved genetic algorithm, the synonyms in the intermediate adversarial sequence are replaced back with the original words or other synonyms to obtain corresponding adversarial sequences that are semantically similar to the text sequence, including: Step s1: Mutate multiple words in the intermediate adversarial sequence to generate multiple candidate sequences. The candidate sequences satisfy the adversarial criterion. Generate an initial population based on the multiple candidate sequences. Step s2: Calculate the similarity values ​​between the multiple candidate sequences in the initial population and the text sequence, and determine the optimal sequence based on all similarity values; Step s3: Perform genetic iteration based on the initial population and the optimal sequence. In each iteration: determine whether to end the genetic iteration based on the number of genetic iterations. If yes, record the optimal sequence as the corresponding adversarial sequence and end the algorithm. If no, perform genetic crossover and mutation on individuals in the population, and update the optimal sequence and the number of genetic iterations based on the crossover and mutation results.

[0008] Furthermore, the genetic crossover of individuals in the population includes uniform crossover and block crossover, wherein uniform crossover is represented as: ; ; ; in, and Let represent the candidate parents for genetic crossover in the population during the k-th round of genetic iteration, i.e., two sequences in the population. The subscript n represents the number of words in the two sequences, and the superscripts 1 and 2 represent one of the two sequences respectively. This represents the sequence obtained by uniform crossover. This indicates that the words in the two sequences are interleaved; The block intersection is represented as: ; in, The sequence obtained by block intersection; This indicates that for the word at position i, the words before and after it are interleaved in a block, and the following conditions are met: .

[0009] Furthermore, based on the text sequence dataset, the text classification model to be trained is run on both a benchmark intelligent algorithm platform and a test intelligent algorithm platform to obtain the performance index values ​​of the benchmark intelligent algorithm platform and the performance test results of the test intelligent algorithm platform, including: The text classification model to be trained was run on the benchmark intelligent algorithm platform and the intelligent algorithm platform to be tested, respectively. The model was used to load the text sequence dataset to obtain the data loading time of the benchmark platform and the data loading time of the platform to be tested. The text classification model to be trained is trained on the benchmark intelligent algorithm platform and the intelligent algorithm platform to be tested, respectively, based on the text sequence dataset. The trained first classification model, the training time of the benchmark platform, the trained second classification model, and the training time of the platform to be tested are obtained. The first classification model and the second classification model are used to perform classification prediction on the text sequence dataset to obtain the prediction accuracy and prediction time of the benchmark platform and the prediction accuracy and prediction time of the platform under test.

[0010] Furthermore, the step of using the text classification model trained on the intelligent algorithm platform under test to classify and predict the adversarial sequence dataset, and obtaining the adversarial prediction result, includes: The second classification model is used on the intelligent algorithm platform under test to perform classification prediction on the adversarial sequence dataset to obtain adversarial prediction results. Based on the adversarial prediction results, the adversarial sample prediction accuracy of the platform under test is calculated.

[0011] Furthermore, evaluating the performance of the intelligent algorithm platform under test based on the benchmark intelligent algorithm platform performance index values, the performance test results of the intelligent algorithm platform under test, and the adversarial prediction results includes: The data loading metric value is calculated based on the data loading time of the benchmark platform and the data loading time of the platform under test. Calculate training performance metrics based on the training time of the benchmark platform and the training time of the platform under test; Calculate inference performance metrics based on the prediction time of the benchmark platform and the prediction time of the platform under test; The inference accuracy metric is calculated based on the preset prediction accuracy threshold and the prediction accuracy of the platform under test. Calculate the adversarial robustness metric value based on the prediction accuracy of the platform under test and the prediction accuracy of adversarial examples; The performance of the intelligent algorithm platform under test is evaluated based on computational data loading metrics, training performance metrics, inference performance metrics, inference accuracy metrics, and adversarial robustness metrics.

[0012] Furthermore, the method for calculating the robustness metric value is as follows: ; in, Indicates a measure of robustness against adversarial forces; This indicates the preset prediction accuracy threshold; This indicates the accuracy of the model in predicting adversarial examples.

[0013] On the other hand, the present invention also provides a performance testing tool for an intelligent algorithm platform, comprising: The data construction module is used to construct a text sequence dataset and generate corresponding adversarial sequences based on each text sequence using an improved genetic algorithm adversarial example generation method, thereby obtaining an adversarial sequence dataset. The model performance testing module is used to run the text classification model to be trained on a benchmark intelligent algorithm platform and a test intelligent algorithm platform based on the text sequence dataset, respectively, to obtain the performance index value of the benchmark intelligent algorithm platform and the performance test result of the test intelligent algorithm platform; and to use the text classification model trained on the test intelligent algorithm platform to perform classification prediction on the adversarial sequence dataset to obtain the adversarial prediction result. The evaluation module is used to evaluate the performance of the intelligent algorithm platform under test based on the performance index values ​​of the benchmark intelligent algorithm platform, the performance test results of the intelligent algorithm platform under test, and the adversarial prediction results.

[0014] Furthermore, the construction of the text sequence dataset includes automatically constructing sequence data based on binary state Markov chains and obtaining training, validation, and test sets for performance testing based on existing classification data.

[0015] The present invention can achieve at least one of the following beneficial effects: By constructing a binary state sequence classification test set based on Markov chains, the constructed dataset can obtain theoretically optimal prediction results through state transition probabilities, which can be used to determine whether the intelligent algorithm classification model can obtain the optimal solution. Furthermore, by fixing the randomization seed for different classification models to reduce the impact of different model parameter randomization factors on model convergence, the performance of the intelligent algorithm platform can be tested fairly.

[0016] By using an improved genetic algorithm to generate adversarial examples, adversarial sequences are generated based on the text sequences to obtain an adversarial sequence dataset. Based on the adversarial sequence dataset, the adversarial prediction results of the model on the intelligent algorithm platform are obtained, thus solving the problem that existing tools cannot evaluate the adversarial robustness of the intelligent algorithm model trained on the platform.

[0017] By improving the genetic algorithm, the adversarial sample generation method of this invention can be applied not only to natural language text, but also to binary state sequence classification. By comprehensively considering average crossover and block crossover in the improved genetic algorithm, the search and optimization efficiency of the genetic algorithm is higher and more accurate, thereby making the generated adversarial sequence more similar to the original sequence and the attack success rate higher.

[0018] Other features and advantages of the invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained from what is particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0019] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Figure 1 This is a flowchart of the performance testing method of the present invention. Detailed Implementation

[0020] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0021] Method Implementation Examples A specific embodiment of the present invention discloses a performance testing method for an intelligent algorithm platform, which specifically includes the following steps: Step S01: Construct a text sequence dataset, and use an improved genetic algorithm-based adversarial example generation method to generate corresponding adversarial sequences based on each text sequence to obtain an adversarial sequence dataset; Step S02: Based on the text sequence dataset, run the text classification model to be trained on the benchmark intelligent algorithm platform and the intelligent algorithm platform to be tested, respectively, to obtain the performance index value of the benchmark intelligent algorithm platform and the performance test results of the intelligent algorithm platform to be tested. Step S03: On the intelligent algorithm platform to be tested, use the text classification model trained on the platform to classify and predict the adversarial sequence dataset to obtain the adversarial prediction result; Step S04: Based on the performance index values ​​of the benchmark intelligent algorithm platform, the performance test results of the intelligent algorithm platform under test, and the adversarial prediction results, obtain the performance evaluation results of the intelligent algorithm platform under test.

[0022] In this embodiment, when constructing performance test data for an intelligent algorithm platform, a text sequence dataset is constructed, including sequence data based on binary state Markov chains. An adversarial sequence dataset is generated using an improved genetic algorithm adversarial example generation method. Based on the text sequence dataset and the adversarial sequence dataset, the performance of the intelligent algorithm platform under test is tested, and the performance evaluation results of the training, inference, and adversarial robustness of the algorithm of the intelligent platform under test can be obtained.

[0023] Specifically, the intelligent algorithm platform refers to a comprehensive software or hardware platform that integrates multiple intelligent algorithms and related tools, aiming to provide powerful computing capabilities, rich data processing capabilities, and easy-to-use interfaces to support various intelligent model applications. In this embodiment, the benchmark intelligent algorithm platform refers to an intelligent algorithm platform constructed using pre-defined software and hardware that has the same functions as the intelligent algorithm platform under test (i.e., both can support various intelligent algorithm model applications) but different performance (there are performance differences when training various intelligent algorithm models and when performing data prediction based on the models). For example, a 4-way rack server with an x86 architecture, two Intel(R) Xeon(R) Gold 6148 CPUs @2.40GHz, 384GB of memory, four Tesla V100-SXM2-32GB GPUs, and an Ubuntu 22.04.3 LTS operating system can be used to build the benchmark intelligent algorithm platform.

[0024] Specifically, in step S01, constructing the text sequence dataset includes automatically constructing sequence data based on binary state Markov chains and obtaining training, validation, and test sets for performance testing based on existing classification data. Existing classification data refers to text data with classification labels used to participate in the training of the text classification model.

[0025] Furthermore, constructing sequence data based on a binary state Markov chain includes: given two states, 0 and 1, and the corresponding state transition probabilities (0->0, 0->1, 1->1, 1->0). The initial sequence is randomly generated with 0 or 1. The next state element depends only on the current state and the state transition probability until a sequence of the expected length (Len) is generated.

[0026] For example, the method for generating Markov Sequence-50 data with a sequence length of 50 is as follows: Select The sequence label generated by the parameters is 0. An example automatically generated by the program based on state transition probabilities is (0,0,0,0,0,0,0,0,0,1,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,0,0,1,1,0,0,0,0,1,1,0,1,0,0,0,0,0,0,0,0), where the last element of the sequence is the label. Select... The sequence label generated by the parameters α=0.3,β=0.6 is 1. The example generated by the program based on the state transition probability is (0,0,0,1,0,0,0,0,0,1,0,0,1,1,1,1,0,1,1,0,0,1,0,0,0,0,0,1,1,0,0,0,0,0,0,1,0,0,1,0,1,0,0,1,1,0,0,0,0,1). The last element of this sequence is the label.

[0027] Preferably, sequences are generated with expected lengths Len of 50, 100, and 150, respectively. The binary-state Markov chain sequence training set, validation set, and test set for performance testing consist of 5000 sequences, all composed of 0s and 1s. It should be noted that automatically constructing sequence data based on binary-state Markov chains reduces the cost of manual data labeling and, by leveraging state transition probabilities, obtains theoretically optimal prediction results. Furthermore, the most even linguistic result is used to determine whether the neural network model used for classification can obtain the optimal solution.

[0028] Furthermore, training, validation, and test sets for performance testing were obtained based on existing classification data, including the Weibo2018 and TNews datasets acquired through open-source methods. The Weibo2018 dataset is a short text sentiment classification dataset for Sina Weibo, with labels including both positive and negative comments. A total of 9,000 training samples, 1,000 validation samples, and 500 test samples were constructed, all in text data format. The TNews dataset contains news texts with 15 topic tags: tech, finance, entertainment, world, car, culture, sports, military, education, game, travel, agriculture, house, story, and stock. A total of 533.6 million training samples, 10,000 validation samples, and 10,000 test samples were constructed, all in text data format.

[0029] Specifically, in step S01, the adversarial example generation method using the improved genetic algorithm to generate corresponding adversarial sequences based on each of the text sequences includes performing the following operations on each of the text sequences: S011. Replace words at some positions in the text sequence with synonyms to obtain an initial adversarial sequence, wherein the initial adversarial sequence satisfies the same classification decision as the text sequence; specifically, the replacement position is determined by whether the same classification decision is satisfied before and after the replacement. S012. At each position to be replaced, try to replace the synonym with the original word or keep the synonym, so that the intermediate adversarial sequence obtained after traversing all positions to replace or keep the synonym satisfies the adversarial criterion. S013. Based on the improved genetic algorithm, the synonyms in the intermediate adversarial sequence are replaced back with the original words or other synonyms to obtain a corresponding adversarial sequence that is semantically similar to the text sequence.

[0030] Furthermore, in this embodiment, the text sequence mentioned in S011 can be a sentence (data from the Weibo2018 and TNews datasets) or sequence data (a sequence of binary state Markov chains); the corresponding synonyms are synonyms or approximate sequences.

[0031] For example, selecting the text sequence 20% of the position indices are used to form the set of positions to be replaced, Indices. For position i∈Indices: For the Weibo2018 and TNews datasets, select the 50 most semantically similar synonyms from the thesaurus, and randomly select one of these synonyms to replace the original word. The replacement will ensure that the part-of-speech tag at the position after replacement matches the part-of-speech tag at the same position in the original sequence; otherwise, the synonym will be discarded and other synonyms will be selected for replacement until the tag consistency condition is met. For the MarkovSequence-Len dataset, the numbers at positions i-1 and i are replaced together. If the numbers are 00 or 11, they are replaced with the approximate sequence 01 or 01. If the numbers are 01 or 10, they are replaced with the approximate sequence 00 or 11. After the replacement, the transition probability is calculated to determine whether the model's classification decision has changed. If it has changed, the replacement of the numbers at the two positions is abandoned. If it has not changed, the replacement of the numbers at the two positions is successful.

[0032] Specifically, the initial adversarial sequence is represented as .in Indicates the position during initialization Word The synonym of being replaced As can be seen from the above replacement process, the initial adversarial sequence is the same as the text sequence classification decision.

[0033] Furthermore, in S012, at each replacement position, the synonym is sequentially replaced with the original word or the synonym is retained, such that the intermediate adversarial sequence obtained after traversing all replacement positions and performing replacements or retaining the synonym satisfies the adversarial criteria, including: Specifically, given an initial adversarial sample For each position i replaced by a synonym (or approximate sequence) (i-1 and i for approximate sequences), use the corresponding original word. Replace it to obtain a temporary adversarial sequence And calculate the similarity between the temporary adversarial sequence and the original sentence. If the adversarial criterion is met, that is Then the element Add to the candidate set `Scores`. Sort the set `Scores` in descending row order based on the similarity scores of the corresponding temporary adversarial sequences. Iterate through the set `Scores`, and for each element... ,against The position in the middle They all use Replace Obtaining the intermediate adversarial sequence , During traversal, retain the values ​​that result in the replacement. Conform to the adversarial principle All Abandoning the replacement yields Not in line with the standard of confrontation ; Specifically, the final intermediate adversarial sequence Approaching the classification decision boundary, but at this point... With the text sequence The similarity is low, and further optimization is needed to make it closer to the text sequence. The similarity.

[0034] Furthermore, in S013, based on an improved genetic algorithm, the synonyms in the intermediate adversarial sequence are replaced back with the original words or other synonyms to obtain corresponding adversarial sequences that are semantically similar to the text sequence, including: Step s1: Mutate multiple words in the intermediate adversarial sequence to generate multiple candidate sequences. The candidate sequences satisfy the adversarial criterion. Generate an initial population based on the multiple candidate sequences. Step s2: Calculate the similarity values ​​between the multiple candidate sequences in the initial population and the text sequence, and determine the optimal sequence based on all similarity values; Step s3: Perform genetic iteration based on the initial population and the optimal sequence. In each iteration: determine whether to end the genetic iteration based on the number of genetic iterations. If yes, record the optimal sequence as the corresponding adversarial sequence and end the algorithm. If no, perform genetic crossover and mutation on individuals in the population, and update the optimal sequence and the number of genetic iterations based on the crossover and mutation results.

[0035] Specifically, in step s1, the positions corresponding to all different words in the intermediate adversarial sequence that are in the text sequence (i.e., the positions corresponding to all synonyms of the original words that were replaced in steps S011 and S012) are saved to the set pos.

[0036] Specifically, for a word at position idx∈pos, a mutation operation is performed on the word, changing the position of the word... Replace with the original word If the intermediate adversarial sequence still conforms to the adversarial criteria after replacement, then... As the final word for that location, and transfer that location information from Delete it; otherwise, find other synonyms of the word at that position and replace them sequentially. If the adversarial criterion is still met after each replacement, add the replaced intermediate adversarial sequence to the candidate set at that position, resulting in a candidate set containing multiple replaced intermediate adversarial sequences. From the candidate set at that position, select the one with the highest similarity to the text sequence and greater than [a certain value]. The sequence was added to the initial population. .

[0037] Iterate through all positions idx∈pos to obtain the initial population. Represented as: ; ; in, Indicates population The m-th sequence, Indicates the initial population The number of; express Based on The sequence obtained by word mutation at position idx.

[0038] Specifically, in step s2, the similarity value between each candidate sequence in the initial population and the text sequence is calculated, and the candidate sequence with the highest similarity value is selected as the optimal sequence in the initial population, and recorded as follows: .

[0039] Specifically, in step s3, each iteration includes steps s3a-s3b.

[0040] Specifically, the number of population iterations is recorded as k, and the population in each iteration is represented as the population. (k∈[0,N-1], N is the number of iterations, preferably N is 100), for the initial population, k=0.

[0041] Step s3a: Specifically, during iteration, determine if k is greater than or equal to N-1. If so, terminate the algorithm. optimal sequence That is, the corresponding adversarial sequence that is semantically similar to the text sequence; if not, proceed with the iteration.

[0042] Step s3b: Perform genetic crossover and mutation on individuals in the population, and update the optimal sequence and the number of genetic iterations based on the crossover and mutation results.

[0043] Specifically, according to probability From population Select one pair of candidate parents: ; ; ; in, Represents the y-th sequence The probability of being selected. This indicates the similarity between the y-th sequence and the text sequence; and Let f(x) represent the candidate parents selected in probability order, and satisfy the following conditions: (greater than) ); Furthermore, C-1 pairs of candidate parents are randomly selected according to probability, where C≤ .

[0044] Furthermore, for each candidate parent pair in C-1, a crossover operation is performed to obtain candidate sequences for population iteration. Genetic crossover of individuals in the population includes uniform crossover and block crossover.

[0045] Specifically, for k∈[0,N / 2-1] population iterations, a uniform crossover operation is performed on each pair of candidate parents to generate a new population; for k∈[N / 2-1,N-1] population iterations, a bidirectional block crossover operation is performed on each pair of candidate parents to generate a new population.

[0046] Specifically, uniform crossover is represented as: ; ; ; in, and Let represent the candidate parents for genetic crossover in the population during the k-th round of genetic iteration, i.e., two sequences in the population. The subscript n represents the number of words in the two sequences, and the superscripts 1 and 2 represent one of the two sequences respectively. This represents the sequence obtained by uniform crossover. This indicates that the words in the two sequences are interleaved.

[0047] Furthermore, when the sequence obtained by uniform crossover has Greater than If the sequence is not found to be a candidate sequence, then the sequence is considered as a candidate sequence. For C-1 pairs of candidate parents, a maximum of C-1 candidate sequences can be obtained.

[0048] Specifically, block crosses are represented as: ; in, This represents the sequence obtained by fast crossover; This indicates that for the word at position i, the words before and after it are interleaved in a block, and the following conditions are met: .

[0049] Furthermore, when the sequence obtained by block crossover has Greater than If the sequence is not found to be a candidate sequence, then the sequence is considered as a candidate sequence. For C-1 pairs of candidate parents, a maximum of C-1 candidate sequences can be obtained.

[0050] Furthermore, based on population In Generate the next generation population from C-1 candidate sequences. .

[0051] Specifically, population In Add directly .

[0052] Specifically, we sequentially examine whether each of the C-1 candidate sequences needs to be mutated. If so, we mutate the candidate sequence and then add it. If not, directly add the candidate sequence. .

[0053] Specifically, during the test, for each candidate sequence, randomly select from... Choose a location ,if exist The original words have not yet been replaced, so it is necessary to... In position Mutation operation is performed at the location to obtain join in It should be noted that for each position, a maximum of [number] operations can be performed. To prevent the genetic algorithm from getting trapped in local optima, mutations are performed repeatedly. If the number of mutations at a certain position reaches 20, no further mutations are performed, and the algorithm is directly terminated. Join and move that location from Delete from the set.

[0054] Specifically, Represented as: ; Represented as: .

[0055] Furthermore, update the iteration count, let k = k + 1, and return to s3a to enter the next iteration.

[0056] Specifically, in step S02, based on the text sequence dataset, the text classification model to be trained is run on both the benchmark intelligent algorithm platform and the intelligent algorithm platform under test, respectively, to obtain the performance index values ​​of the benchmark intelligent algorithm platform and the performance test results of the intelligent algorithm platform under test, including: S021. Run the text classification model to be trained on the benchmark intelligent algorithm platform and the intelligent algorithm platform to be tested respectively, use the model to load the text sequence dataset, and obtain the data loading time of the benchmark platform and the data loading time of the platform to be tested. S022. Train the text classification model to be trained on the benchmark intelligent algorithm platform and the intelligent algorithm platform to be tested respectively based on the text sequence dataset to obtain the trained first classification model, the training time of the benchmark platform, the trained second classification model, and the training time of the platform to be tested. S023. Use the first classification model and the second classification model to perform classification prediction on the text sequence dataset respectively, and obtain the prediction accuracy and prediction time of the benchmark platform and the prediction accuracy and prediction time of the platform under test.

[0057] Optionally, in step S021, the text classification model to be trained includes WordLSTM, WordCNN, and BERT models, which are constructed based on intelligent algorithms such as LSTM, CNN, and BERT, respectively. It should be noted that these three models cover three different types of neural network structures, contain different neural network operators, and have different parameter counts. Running these three models on a benchmark intelligent algorithm platform and the intelligent algorithm platform under test, respectively, and obtaining the corresponding performance test results can objectively evaluate the performance of the intelligent algorithm platform under test.

[0058] Furthermore, in step S022, the text classification model to be trained is trained on the benchmark intelligent algorithm platform and the intelligent algorithm platform to be tested, respectively, based on the text sequence dataset.

[0059] Furthermore, in step S022, the benchmark intelligent algorithm platform and the intelligent algorithm platform under test first preprocess the text sequence dataset to obtain the data loading time of the benchmark platform and the data loading time of the platform under test.

[0060] Specifically, the preprocessing operation includes: Tokenization of text sequences: For sequence data of binary state Markov chains, each '0' or '1' character is a word; for Weibo2018 data, geographical locations, usernames, and various symbols need to be removed, and the Jieba tokenization tool is used for tokenization. A set of words is constructed based on all the words obtained after word segmentation: a set of words. ,in Indicator, Indicates the frequency of a single occurrence; Generate a text sequence of equal length based on the set: based on Generate a text sequence of equal length from the text sequence, so that the text classification model to be trained can be trained and perform classification prediction based on the text sequence of equal length.

[0061] Furthermore, in step S022, the text classification model to be trained is trained on both the benchmark intelligent algorithm platform and the intelligent algorithm platform under test based on the preprocessed text sequence dataset, respectively, to obtain the trained first classification model, the training time of the benchmark platform, and the trained second classification model and the training time of the platform under test. The first classification model is trained on the benchmark intelligent algorithm platform, and the second classification model is trained on the intelligent algorithm platform under test.

[0062] Specifically, the text classification model is trained using the training and validation sets from the text sequence dataset. Training terminates when the maximum number of iterations is reached. It should be noted that, to ensure fair testing across different platforms, the random number generator in the model is set to a fixed value. This ensures that the parameters generated by random initialization remain consistent, guaranteeing that the model can achieve roughly the same optimal result across different platforms.

[0063] Furthermore, in step S023, the first classification model and the second classification model obtained in step S022 are used to classify and predict the test set data of the text sequence dataset, respectively, to obtain the prediction accuracy and prediction time of the benchmark platform and the prediction accuracy and prediction time of the platform under test.

[0064] The prediction accuracy refers to the accuracy obtained by comparing the labels generated in the prediction stage with the true labels. Preferably, this embodiment uses the F1-Score as the accuracy. The F1 Score is a statistical indicator used to measure the precision of a binary classification model, which takes into account both the precision and recall of the classification model.

[0065] Specifically, in step S03, the second classification model is used on the intelligent algorithm platform under test to perform classification prediction on the adversarial sequence dataset, obtaining the adversarial prediction result. Based on the adversarial prediction result, the prediction accuracy of the adversarial samples on the platform under test is calculated. Further, before prediction, the adversarial sequence dataset needs to be preprocessed, and the preprocessing process is the same as the preprocessing process in step S022.

[0066] Specifically, in step S04, a data loading metric is calculated based on the data loading time of the benchmark platform and the data loading time of the platform under test. The calculation method is as follows: ; in, Indicates the time taken to load data on the benchmark platform; and These represent the start and end times of data loading on the platform under test, respectively. This indicates the time taken to load data on the platform under test; This indicates the value of the data loading metric.

[0067] Furthermore, training performance metrics are calculated based on the training time of the benchmark platform and the training time of the platform under test. The calculation method is as follows: ; in, Indicates the training time of the benchmark platform; and These represent the start and end times of training the model on the platform under test, respectively. This indicates the training time of the platform under test; This represents the training performance metric value.

[0068] Furthermore, inference performance metrics are calculated based on the prediction time of the benchmark platform and the prediction time of the platform under test. The calculation method is as follows: ; in, Indicates the prediction time of the benchmark platform; and These represent the start and end times of the inference and prediction process of the platform under test model, respectively. This indicates the prediction time for the platform under test; This represents the inference performance metric value.

[0069] Furthermore, the inference accuracy metric is calculated based on the preset prediction accuracy threshold and the prediction accuracy of the platform under test. It should be noted that for the binary state Markov chain sequence dataset, since the test sequences are obtained based on probability transitions, the theoretically optimal classification accuracy (84.4%) can be calculated. Therefore, the corresponding preset prediction accuracy threshold can be determined based on the theoretically optimal classification accuracy. However, for the Weibo2018 and TNews datasets, since the test datasets rely on manual labeling, it is difficult to obtain the theoretically optimal classification accuracy. Therefore, the corresponding preset prediction accuracy threshold can be determined based on the prediction accuracy of the benchmark platform.

[0070] Furthermore, based on the preset prediction accuracy threshold and the prediction accuracy of the platform under test, the inference accuracy metric is calculated. The calculation method is as follows: ; in, This represents the value of the inference accuracy metric. This indicates the preset prediction accuracy threshold; This represents the prediction accuracy of the platform under test based on the test set in the text sequence dataset.

[0071] Furthermore, based on the prediction accuracy of the platform under test and the prediction accuracy of adversarial examples, an adversarial robustness metric is calculated. The calculation method is as follows: ; in, This represents the value of the robustness metric against adversarial forces. Indicates the accuracy of adversarial sample prediction; This represents the prediction accuracy based on the test set in the text sequence dataset.

[0072] Specifically, in step S04, the performance of the intelligent algorithm platform under test is evaluated based on the calculated data loading metrics, training performance metrics, inference performance metrics, inference accuracy metrics, and adversarial robustness metrics.

[0073] Specifically, data loading metric values The larger the value, the better the data loading performance of the intelligent algorithm platform under test. For example, in one embodiment, when training the WordLSTM model, the total time taken to load the TNews data from the text sequence dataset on the benchmark intelligent algorithm platform was 91.9 seconds. If the time taken for the WordLSTM model of the benchmark intelligent algorithm platform to load the TNews data is less than 91.9 seconds, then... This indicates that the loading performance of the quasi-intelligent algorithm platform under test is better than that of the benchmark intelligent algorithm platform.

[0074] Similarly, training performance metrics The larger the value, the better the training performance of the intelligent algorithm platform under test. For example, in one embodiment, training the WordLSTM model based on the training set of the text sequence dataset took a total of 814 seconds. If the training time of the WordLSTM model of the quasi-intelligent algorithm platform under test is less than 814 seconds, then... This indicates that the training performance of the tested quasi-intelligent algorithm platform is better than that of the benchmark intelligent algorithm platform.

[0075] Similarly, inference performance metrics The larger the value, the better the inference performance of the intelligent algorithm platform under test. In one embodiment, the WordLSTM model of the first classification model is used for prediction inference on the TNews test set of the text sequence dataset, and the total time taken is 2.6 seconds. If the prediction time of the WordLSTM model of the second classification model of the quasi-intelligent algorithm platform under test is less than 2.6 seconds, then... This indicates that the reasoning performance of the tested intelligent algorithm platform is better than that of the benchmark intelligent algorithm platform.

[0076] Specifically, the inference accuracy metric value The larger the value, the better the prediction accuracy of the intelligent algorithm platform under test. This indicates that the platform can achieve the model's metrics, when This indicates that the model has not reached the baseline performance. For example, using the WordLSTM model of the first classification model for prediction inference on the TNews test set of the text sequence dataset, the classification accuracy is 52.1%. If the prediction accuracy of the WordLSTM model of the second classification model of the quasi-intelligent algorithm platform to be tested is greater than 52.1%, it indicates that the prediction accuracy of the quasi-intelligent algorithm platform to be tested is better than that of the baseline intelligent algorithm platform.

[0077] Specifically, the robustness metric value , A larger value indicates higher adversarial robustness of the model trained by the precision intelligent algorithm platform; for the intelligent algorithm platform under test, the calculated... It can also be compared with the robustness metric of the benchmark intelligent algorithm platform. Comparison, if This indicates that the tested intelligent algorithm platform exhibits better adversarial robustness than the benchmark intelligent algorithm platform. For example, in one embodiment, the WordLSTM model of the first classification model of the benchmark intelligent algorithm platform achieves a prediction accuracy of 33.2% on the TNews test set in adversarial sequences, a significant decrease compared to the classification accuracy of 52.1% on the text sequence dataset. If the intelligent algorithm platform to be tested This indicates that the robustness of the tested intelligent algorithm platform is superior to that of the benchmark intelligent algorithm platform.

[0078] This embodiment discloses a performance test of an intelligent algorithm platform. It generates adversarial sequences based on the text sequences using an improved genetic algorithm-based adversarial example generation method, obtaining an adversarial sequence dataset. Based on this dataset, it obtains the adversarial prediction results of the model on the intelligent algorithm platform, solving the problem that existing tools cannot evaluate the adversarial robustness of intelligent algorithm models trained on the platform. The improved genetic algorithm-based adversarial example generation method in this embodiment is applicable not only to natural language text but also to binary state sequence classification. By comprehensively considering average crossover and block crossover in the improved genetic algorithm, the search and optimization efficiency of the genetic algorithm is higher and more accurate, resulting in higher similarity between the generated adversarial sequences and the original sequences, and a higher attack success rate.

[0079] This embodiment also constructs a binary state sequence classification test set based on Markov chains, so that the constructed dataset can obtain the theoretically optimal prediction results by means of state transition probabilities, which can be used to determine whether the classification model can obtain the optimal solution; and by fixing the randomization seed for different classification models, the influence of different model parameter randomization factors on model convergence is reduced, thereby enabling fair performance testing of the intelligent algorithm platform.

[0080] Tool Examples Another specific embodiment of the present invention discloses a performance testing tool for an intelligent algorithm platform, including a data construction module, a model performance testing module, and an evaluation module.

[0081] Specifically, the data construction module is used to construct a text sequence dataset and generate corresponding adversarial sequences based on each text sequence using an improved genetic algorithm adversarial example generation method, thereby obtaining an adversarial sequence dataset.

[0082] Specifically, the model performance testing module is used to run the text classification model to be trained on both the benchmark intelligent algorithm platform and the intelligent algorithm platform to be tested, based on the text sequence dataset, to obtain the performance index values ​​of the benchmark intelligent algorithm platform and the performance test results of the intelligent algorithm platform to be tested; and to use the text classification model trained on the intelligent algorithm platform to be tested to perform classification prediction on the adversarial sequence dataset, to obtain the adversarial prediction results.

[0083] Specifically, the evaluation module is used to evaluate the performance of the intelligent algorithm platform under test based on the performance index values ​​of the benchmark intelligent algorithm platform, the performance test results of the intelligent algorithm platform under test, and the adversarial prediction results.

[0084] Furthermore, the construction of the text sequence dataset includes automatically constructing sequence data based on binary state Markov chains and obtaining training, validation, and test sets for performance testing based on existing classification data.

[0085] Compared to existing technologies, the performance testing tool for an intelligent algorithm platform provided in this embodiment has essentially the same beneficial effects as those provided in the method embodiment, and will not be elaborated upon here.

[0086] It should be noted that the above embodiments are based on the same inventive concept, and the parts not described repeatedly can be used as references to each other.

[0087] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A performance testing method for an intelligent algorithm platform, characterized in that, Includes the following steps: A text sequence dataset is constructed, and an adversarial example generation method based on an improved genetic algorithm is used to generate corresponding adversarial sequences based on each text sequence, resulting in an adversarial sequence dataset. The adversarial example generation method based on each text sequence using the improved genetic algorithm includes performing the following operations on each text sequence: replacing words at some positions in the text sequence with synonyms to obtain an initial adversarial sequence, wherein the initial adversarial sequence satisfies the same classification decision as the text sequence; and sequentially attempting to replace the synonyms with the original words or retain the synonyms at each replaced position, such that the intermediate adversarial sequence obtained after traversing all replaced positions for replacement or retention of the synonyms satisfies the adversarial criteria. An improved genetic algorithm is used to replace the synonyms in the intermediate adversarial sequence back to the original words or other synonyms, resulting in a corresponding adversarial sequence that is semantically similar to the text sequence; wherein, given an initial adversarial sample For each position i replaced by a synonym, use the corresponding original word. Replace it to obtain a temporary adversarial sequence The adversarial criterion is expressed as ; Based on the text sequence dataset, the text classification model to be trained was run on both a benchmark intelligent algorithm platform and the intelligent algorithm platform under test, respectively, to obtain the performance index values ​​of the benchmark intelligent algorithm platform and the performance test results of the intelligent algorithm platform under test. Here, the intelligent algorithm platform refers to a comprehensive software or hardware platform integrating multiple intelligent algorithms and related tools; the benchmark intelligent algorithm platform refers to an intelligent algorithm platform with the same function but different performance as the intelligent algorithm platform under test, constructed using pre-defined software and hardware. The test results include: running the text classification model to be trained on both the benchmark intelligent algorithm platform and the intelligent algorithm platform under test, loading the text sequence dataset using the model, and obtaining the data loading time of the benchmark platform and the data loading time of the platform under test; training the text classification model to be trained on both the benchmark intelligent algorithm platform and the intelligent algorithm platform under test based on the text sequence dataset, and obtaining the trained first classification model, the training time of the benchmark platform, the trained second classification model, and the training time of the platform under test; and using the first classification model and the second classification model to perform classification prediction on the text sequence dataset, and obtaining the prediction accuracy and prediction time of the benchmark platform, the prediction accuracy and prediction time of the platform under test. The process of classifying and predicting the adversarial sequence dataset using the text classification model trained on the intelligent algorithm platform under test to obtain adversarial prediction results includes: classifying and predicting the adversarial sequence dataset using the second classification model on the intelligent algorithm platform under test to obtain adversarial prediction results, and calculating the adversarial sample prediction accuracy of the platform under test based on the adversarial prediction results. The performance evaluation result of the intelligent algorithm platform under test is obtained based on the performance index value of the benchmark intelligent algorithm platform, the performance test result of the intelligent algorithm platform under test, and the adversarial prediction result.

2. The performance testing method according to claim 1, characterized in that, Based on an improved genetic algorithm, the synonyms in the intermediate adversarial sequence are replaced back with the original words or other synonyms to obtain corresponding adversarial sequences that are semantically similar to the text sequence, including: Step s1: Mutate multiple words in the intermediate adversarial sequence to generate multiple candidate sequences. The candidate sequences satisfy the adversarial criterion. Generate an initial population based on the multiple candidate sequences. Step s2: Calculate the similarity values ​​between the multiple candidate sequences in the initial population and the text sequence, and determine the optimal sequence based on all similarity values; Step s3: Perform genetic iteration based on the initial population and the optimal sequence. In each iteration: determine whether to end the genetic iteration based on the number of genetic iterations. If yes, record the optimal sequence as the corresponding adversarial sequence and end the algorithm. If no, perform genetic crossover and mutation on individuals in the population, and update the optimal sequence and the number of genetic iterations based on the crossover and mutation results.

3. The performance testing method according to claim 2, characterized in that, The genetic crossover of individuals in the population includes uniform crossover and block crossover, wherein uniform crossover is represented as follows: ； ； ； in, and Let represent the candidate parents for genetic crossover in the population during the k-th round of genetic iteration, i.e., two sequences in the population. The subscript n represents the number of words in the two sequences, and the superscripts 1 and 2 represent one of the two sequences respectively. This represents the sequence obtained by uniform crossover. This indicates that the words in the two sequences are interleaved; The block intersection is represented as: ； in, The sequence obtained by block intersection; This indicates that for the word at position i, the words before and after it are interleaved in a block, and the following conditions are met: .

4. The performance testing method according to claim 3, characterized in that, The performance evaluation results of the intelligent algorithm platform under test, based on the performance index values ​​of the benchmark intelligent algorithm platform, the performance test results of the intelligent algorithm platform under test, and the adversarial prediction results, include: The data loading metric value is calculated based on the data loading time of the benchmark platform and the data loading time of the platform under test. Calculate training performance metrics based on the training time of the benchmark platform and the training time of the platform under test; Calculate inference performance metrics based on the prediction time of the benchmark platform and the prediction time of the platform under test; The inference accuracy metric is calculated based on the preset prediction accuracy threshold and the prediction accuracy of the platform under test. Calculate the adversarial robustness metric value based on the prediction accuracy of the platform under test and the prediction accuracy of adversarial examples; The performance of the intelligent algorithm platform under test is evaluated based on computational data loading metrics, training performance metrics, inference performance metrics, inference accuracy metrics, and adversarial robustness metrics.

5. The performance testing method according to claim 4, characterized in that, The method for calculating the robustness metric value is as follows: ； in, Indicates a measure of robustness against adversarial forces; This indicates the preset prediction accuracy threshold; This indicates the accuracy of the model in predicting adversarial examples.

6. A performance testing device for an intelligent algorithm platform, characterized in that, include A data construction module is used to construct a text sequence dataset and generate corresponding adversarial sequences based on each text sequence using an improved genetic algorithm-based adversarial example generation method, resulting in an adversarial sequence dataset. The method involves performing the following operations on each text sequence: replacing words at certain positions in the text sequence with synonyms to obtain an initial adversarial sequence, which satisfies the same classification decision as the text sequence; and sequentially attempting to replace the synonyms with the original words or retain the synonyms at each replaced position, such that the intermediate adversarial sequence obtained after traversing all replaced positions and performing replacements or retaining the synonyms satisfies the adversarial criteria. An improved genetic algorithm is used to replace the synonyms in the intermediate adversarial sequence back to the original words or other synonyms, resulting in a corresponding adversarial sequence that is semantically similar to the text sequence; wherein, given an initial adversarial sample For each position i replaced by a synonym, use the corresponding original word. Replace it to obtain a temporary adversarial sequence The principle of confrontation is ; The model performance testing module is used to run the text classification model to be trained on both a benchmark intelligent algorithm platform and a test intelligent algorithm platform based on the text sequence dataset, to obtain the performance index values ​​of the benchmark intelligent algorithm platform and the performance test results of the test intelligent algorithm platform. The intelligent algorithm platform refers to a comprehensive software or hardware platform integrating multiple intelligent algorithms and related tools; the benchmark intelligent algorithm platform refers to an intelligent algorithm platform with the same function but different performance as the test intelligent algorithm platform, constructed using pre-defined software and hardware. The process of running the text classification model to be trained on both the benchmark and test intelligent algorithm platforms based on the text sequence dataset to obtain the performance index values ​​of the benchmark intelligent algorithm platform and the performance test results of the test intelligent algorithm platform includes: running the text classification model to be trained on both the benchmark and test intelligent algorithm platforms, loading the text sequence dataset using the model, and obtaining the benchmark platform data. Loading time and data loading time of the platform under test; training the text classification model on the benchmark intelligent algorithm platform and the intelligent algorithm platform under test based on the text sequence dataset, respectively, to obtain the trained first classification model, the training time of the benchmark platform, the training time of the trained second classification model, and the training time of the platform under test; using the first classification model and the second classification model to perform classification prediction on the text sequence dataset, respectively, to obtain the prediction accuracy of the benchmark platform, the prediction time of the benchmark platform, the prediction accuracy of the platform under test, and the prediction time of the platform under test; and using the text classification model trained on the intelligent algorithm platform under test to perform classification prediction on the adversarial sequence dataset, to obtain the adversarial prediction result, including: using the second classification model on the intelligent algorithm platform under test to perform classification prediction on the adversarial sequence dataset to obtain the adversarial prediction result, and calculating the adversarial sample prediction accuracy of the platform under test based on the adversarial prediction result. The evaluation module is used to evaluate the performance of the intelligent algorithm platform under test based on the performance index values ​​of the benchmark intelligent algorithm platform, the performance test results of the intelligent algorithm platform under test, and the adversarial prediction results.

7. The performance testing apparatus according to claim 6, characterized in that, The construction of the text sequence dataset includes automatically constructing sequence data based on binary state Markov chains and obtaining training, validation, and test sets for performance testing based on existing classification data.

Images