A robustness testing method for intelligent models based on empirical mode decomposition

Abstract

The present invention relates to a method for testing the robustness of an intelligent model based on empirical mode decomposition, and belongs to the technical field of robustness determination of artificial intelligence models. The main technical scheme includes constructing an original test data set based on the intelligent model to be tested; expanding the original test data set based on empirical mode decomposition; calculating the recognition accuracy and loss of the test data set at different decomposition levels; and calculating the intelligent model recognition accuracy change index and recognition loss change index. The method of the present invention uses empirical mode decomposition to expand and collect the original test data set, and by calculating the changing trends of the model recognition accuracy and loss at different decomposition levels, it can objectively evaluate the robustness of the intelligent model.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence model robustness determination, and specifically relates to an intelligent model robustness testing method based on empirical mode decomposition. Background Art

[0002] With the advent of the information age, the advantages of artificial intelligence (AI) have become increasingly apparent. Not only can it make decisions efficiently and reliably, but the decision-making process is also unaffected by human emotions. Consequently, AI technology is increasingly being applied across all sectors of society. Simultaneously, AI is widely used in fields such as image recognition, speech recognition, and object detection. However, as AI applications continue to gain traction, its vulnerabilities are gradually being exposed, raising a series of security concerns. The structural complexity and black-box nature of AI models, such as deep neural networks, mean that even slightly perturbed input data can cause them to make erroneous decisions. Faced with numerous threats, AI robustness has become a new research hotspot. Robustness, a transliteration of "robust," also known as robustness or robustness, refers to the ability of a system to maintain certain performance characteristics under certain parameter perturbations (structural or numerical). Therefore, robustness is crucial for ensuring the proper functioning of critical functions in abnormal and dangerous situations. In AI technology, robustness measures the data tolerance of models and algorithms. AI should maintain its learning and decision-making capabilities even in unexpected data input environments, making accurate decisions while accepting a certain degree of error and anomalies.

[0003] Beijing Qihoo Technology Co., Ltd. proposed a method and apparatus for robustness testing applications in its patent application, "Method and Apparatus for Robustness Testing of Applications" (Patent Application Number: CN201810802657.8, Publication Number: CN110737576A). The method includes: when a class of an application is loaded, parsing the variables to be tested within the class; determining a test value that matches each variable to be tested based on preset rules, and transferring the determined test value to each variable to be tested; after the class is loaded, determining the robustness of the application based on the running status of the application. This technical solution can dynamically inject test values ​​to test whether the application can correctly handle exceptions. If the application crashes, it indicates a robustness issue; if the application runs properly, it indicates good robustness. This provides a simple, convenient, and effective method for testing application robustness. However, this method still has shortcomings: it does not provide a quantitative indicator to demonstrate the robustness of intelligent applications.

[0004] Hunan University proposed a method for effectively attacking an unknown model without knowing the AI ​​model and training dataset in its patent application, “A Method and System for Testing the Robustness of Artificial Intelligence Models” (patent application number: CN202011630126.9, publication number: CN112766315A). The method includes the following steps: capturing the shallow shared features of the original dataset and generating corresponding target samples based on the input conditions. The AI ​​model to be tested is used to predict the output of the target sample, which is compared with the official label to obtain the corresponding loss. The target samples generated at the same time are used to train the alternative model and obtain the corresponding output. The output of the same sample under two different models is compared to obtain the comparison loss, which is used to ensure that the alternative model has well learned all the functions of the AI ​​model to be tested. Then, some mature attack algorithms are used to attack the alternative model to obtain corresponding adversarial samples, and these adversarial samples are used to attack the unknown AI model to determine the robustness of the AI ​​model. However, this method still has some shortcomings: the testing process is complicated and requires training of alternative models. When facing complex models, the testing cost of this method will be huge, and it is too dependent on the alternative model. The quality of the training will directly affect the test results. Summary of the Invention

[0005] (1) Technical issues to be resolved

[0006] The technical problem to be solved by the present invention is: how to study the robustness testing method of intelligent models based on empirical mode decomposition, and provide a testing method that supports robustness testing of intelligent models in various application scenarios and calculates relevant indicators.

[0007] (2) Technical solution

[0008] In order to solve the above technical problems, the present invention provides an intelligent model robustness testing method based on empirical mode decomposition, comprising the following steps:

[0009] Step 1: Based on the intelligent model to be tested, collect the original test data set:

[0010] 1.1, Initialize the test data set parameters, including the number of samples N, sample data format, etc.;

[0011] 1.2. Based on the intelligent model to be tested, an original test data set of size N is collected. This refers to the application scenario of the intelligent model to be tested, and an original test data set of size N is collected in various physical environments.

[0012] 1.3, read n test data from the original test data set and preprocess them based on the input format of the intelligent model to be tested;

[0013] Step 2: Expand the original test dataset based on empirical mode decomposition:

[0014] 2.1, based on the empirical mode decomposition method, each test data is decomposed into k eigensignals and 1 residual signal, and (k+1)·n expanded test data are obtained, which means that the i-th sample data x i Decomposed into (k+1) empirical mode decomposition components {x i1 ,x i2 ,x i3 ......,x ik ,ζ}, where x ij Represents the i-th sample data x i The j-th empirical mode decomposition component of , j = 1, 2, 3, .... k; ζ represents the residual signal.

[0015] Step 3 calculates the recognition accuracy and loss of the test data set at different decomposition levels:

[0016] 3.1, Initialize the recognition accuracy vector and the recognition loss vector;

[0017] 3.2, by accumulating different numbers of eigensignals of the same test data, we can obtain k test data sets of different decomposition degrees of size n. This means that we use the (k+1)·n expanded test data obtained in 2.1 to calculate the test data sets Y of different decomposition degrees. l , where l represents the degree of decomposition, l = 1, 2, 3, ..., k, Y l Each sample in Y is calculated according to the formula: l ={y1,y2,y3,...,y n}

[0018]

[0019] Calculated; where y i Represents the test data set Y with a decomposition degree of l l The i-th sample in, l=1,2,3,...,k, i=1,2,3,...,n, x ij represents the jth eigensignal after empirical mode decomposition of the i-th sample in the original test data set; ζ represents the residual signal after empirical mode decomposition of the i-th sample in the original test data set;

[0020] 3.3, select a test data set with different decomposition levels and input it into the intelligent model to be tested to obtain n recognition results;

[0021] 3.4. Calculate the recognition accuracy and recognition loss based on the recognition results, and add them to the recognition accuracy vector and recognition loss vector respectively. This means that the recognition accuracy c / n is obtained by comparing the n recognition results obtained in step 3.3 with the true labels of the samples, where c represents the number of samples that are correctly identified. Then, use the test data set Y with a decomposition degree of l selected in step 3.3 to calculate the recognition accuracy c / n. l , according to the loss function loss(Y l ), calculate the recognition loss; finally, add the recognition accuracy and recognition loss to the recognition accuracy vector and recognition loss vector;

[0022] 3.5, determine whether the number of loops k has been reached. If so, jump to step 4, otherwise return to step 3.3;

[0023] Step 4: Calculate the intelligent model recognition accuracy change index and recognition loss change index:

[0024] 4.1, based on the recognition accuracy vector and the recognition loss vector, the recognition accuracy change index AC and the recognition loss change index LOSS are calculated. This means using the recognition accuracy vector and the recognition loss vector obtained in step 3, according to the formula:

[0025]

[0026]

[0027]

[0028] Calculated; among them, Represents the recognition accuracy vector, a i represent The i-th element in is the recognition accuracy of the test data set with decomposition degree i, i = 1, 2, 3, ..., k, k represents the number of eigenvalues ​​of a sample after empirical mode decomposition; ac origin represents the recognition accuracy of the original test data set; 1 / k represents the average of the subsequent calculation results; AC represents the change in the recognition accuracy of the test data set for every 1 unit change in the decomposition degree; represents the recognition loss vector, b i represent The i-th element in is the recognition loss of the test data set with decomposition degree i, i = 1, 2, 3, ..., k, k represents the number of eigenvalues ​​of a sample after empirical mode decomposition; loss origin represents the recognition loss of the original test dataset; 1 / k represents the average of subsequent calculation results; LOSS represents the degree of change in the recognition loss of the test dataset when the decomposition degree changes by 1 unit.

[0029] Preferably, the intelligent model to be detected is a voice control instruction recognition model.

[0030] Preferably, the loss function used is the cross entropy loss function.

[0031] The present invention also provides a system for implementing the method.

[0032] The present invention also provides an application of the method in the technical field of artificial intelligence model robustness determination.

[0033] (3) Beneficial effects

[0034] As a result of adopting the above technical solution, the present invention has the following advantages compared with the prior art:

[0035] The present invention proposes to collect an original test data set of size N based on the intelligent model to be tested, which means that the algorithm of the present invention supports robustness testing of intelligent models in various application scenarios and has strong versatility.

[0036] The empirical mode decomposition method proposed in the present invention decomposes each test data into k intrinsic signals and 1 residual signal to obtain (k+1)·n expanded test data. This means that the sample data is subjected to empirical mode decomposition to obtain components with different decomposition degrees and different amounts of information, which can effectively improve the adequacy and diversity of the test samples.

[0037] The present invention proposes to obtain k test data sets of different decomposition degrees of size n by accumulating different numbers of intrinsic signals of the same test data. This means that the components obtained after decomposition are combined together by linear accumulation to obtain test data sets with different information contents, which can effectively improve the accuracy and effectiveness of the testing method.

[0038] The present invention proposes to calculate the recognition accuracy and recognition loss based on the recognition results, and add them to the recognition accuracy vector and the recognition loss vector respectively. This means that the recognition accuracy and recognition loss of test data sets with different decomposition degrees are calculated in turn. This can intuitively express the changing trend of the intelligent model recognition situation when test data sets with different decomposition degrees and information content are input.

[0039] The recognition accuracy change index AC and the recognition loss change index LOSS proposed in the present invention are calculated based on the recognition accuracy vector and the recognition loss vector. They refer to the degree of change in the recognition accuracy and loss of the test data set when the decomposition degree changes by 1 unit. This can effectively quantify the robustness of the model and intuitively compare the robustness gap between different models. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1is a flow chart of the method of the present invention;

[0041] Figure 2 The original audio signal spectrogram and the eigensignal spectrogram and residual signal spectrogram obtained after the original audio signal is subjected to empirical mode decomposition;

[0042] Figure 3 Schematic diagram of the robustness test process of the method of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and examples.

[0044] The present invention provides an intelligent model robustness testing method based on empirical mode decomposition for testing the robustness of intelligent models and calculating relevant indicators, which can be applied to intelligent models in different usage scenarios. The overall process of the method is as follows: Figure 1 As shown in the figure, the method uses empirical mode decomposition to expand the original test data set and objectively evaluates the robustness of the intelligent model by calculating the changing trends of the model recognition accuracy and loss at different decomposition levels. The method includes the following steps:

[0045] The model to be detected used in this embodiment is a voice control instruction recognition model, and the loss function used is a cross-entropy loss function.

[0046] Step 1: Collect the original test data set based on the intelligent model to be tested:

[0047] Initialize the test data set parameters, including the number of samples N = 4000 (including eight English voice control commands including down, go, left, off, on, right, stop, and up, with 500 of each voice control command), the audio data format is ".wav", the sampling frequency is 16 kHz, and the duration is less than or equal to 1 second, that is, the length of the read-in audio data sequence is less than or equal to 16000.

[0048] Based on the intelligent model to be tested, a self-collected original test data set with a scale of 4,000 in the form of WAV audio files contains eight English voice control commands, including down, go, left, off, on, right, stop, and up, with 500 commands for each type of voice control command.

[0049] 4,000 test data items are read from the original test dataset and preprocessed based on the input format of the intelligent model to be tested. Preprocessing involves padding the audio data with a duration of less than 1 second with zeros at the end to bring the length to 16,000.

[0050] Step 2: Expand the original test dataset based on empirical mode decomposition:

[0051] Based on the empirical mode decomposition method, each test data is decomposed into 5 eigensignals and 1 residual signal, and 6*4000=24000 expanded test data are obtained.

[0052] Step 3: Calculate the recognition accuracy and loss of the test data set at different decomposition levels:

[0053] Initialize the recognition accuracy vector and the recognition loss vector;

[0054] By accumulating different numbers of eigensignals of the same test data, 5 test data sets with different decomposition degrees of 4000 are obtained. This means that the test data sets Y with different decomposition degrees are calculated using the 24,000 expanded test data obtained in step 2. l , where l represents the degree of decomposition, l = 1, 2, 3, 4, 5, Y l Each sample in Y is calculated according to the formula: l ={y1,y2,y3,...,y 4000}

[0055]

[0056] Calculated; where y i Represents the test data set Y with a decomposition degree of l l The i-th sample in, l=1,2,3,4,5,i=1,2,3,...,4000,x ij represents the jth eigensignal after empirical mode decomposition of the i-th sample in the original test data set; ζ represents the residual signal after empirical mode decomposition of the i-th sample in the original test data set;

[0057] A test data set with different decomposition levels was selected and input into the intelligent model to be tested, and 4,000 recognition results were obtained;

[0058] The 4000 recognition results are compared with the true labels of the samples to obtain the recognition accuracy c / n, where c represents the number of samples that are correctly identified; then the test data set Y with the selected decomposition level of l is used l , according to the cross entropy loss function CrossEntropyLoss(Y l ), calculate the recognition loss; finally, add the recognition accuracy and recognition loss to the recognition accuracy vector and recognition loss vector;

[0059] Determine whether the loop count has reached 5. If so, jump to step 4, otherwise return to step 3;

[0060] Step 4: Calculate the intelligent model recognition accuracy change index and recognition loss change index:

[0061] Using the recognition accuracy vector and recognition loss vector obtained in step 3, calculate the recognition accuracy change index AC and the recognition loss change index LOSS according to the formula:

[0062]

[0063]

[0064]

[0065] Calculated; among them, Represents the recognition accuracy vector, a i represent The i-th element in is the recognition accuracy of the test data set with decomposition degree i, i = 1, 2, 3, 4, 5, 5 represents the number of eigenvalues ​​after empirical mode decomposition of a sample; ac origin represents the recognition accuracy of the original test data set; 1 / 5 represents the average of the subsequent calculation results; AC represents the change in the recognition accuracy of the test data set for each unit change in the decomposition degree; Represents the recognition loss vector, bi represents The i-th element in is the recognition loss of the test data set with decomposition degree i, i = 1, 2, 3, 4, 5, 5 represents the number of eigensignals of a sample after empirical mode decomposition; loss origin represents the recognition loss of the original test dataset; 1 / 5 represents the average of subsequent calculation results; LOSS represents the change in the recognition loss of the test dataset when the decomposition degree changes by 1 unit.

[0066] The effects of the present invention are further described below in conjunction with simulation experiments:

[0067] 1. Simulation experiment conditions:

[0068] The hardware platform of the simulation experiment of the present invention is: Intel Core (TM) i7-6700@3.40GHZ×8, GPU NVIDIA GeForce GTX 1060, 6GB memory.

[0069] The software platforms for the simulation experiment of the present invention are: Windows 10 operating system and Anaconda3.

[0070] The simulation experiment of the present invention adopts a comparative experiment. The intelligent models to be tested are the voice control command recognition model Model-A and Model-B obtained after reinforcement training of Model-A. The original test data set collected is eight types of English voice control commands including down, go, left, off, on, right, stop, and up, with 500 commands of each type.

[0071] 2. Simulation content and results

[0072] The simulation experiment of the present invention uses the method of the present invention to perform robustness tests on model-A and model-B, and calculates the recognition accuracy change index and the recognition loss change index. The results are shown in Table 1.

[0073] Table 1 Robustness test results of Model-A and Model-B

[0074] Model to be tested Recognition accuracy change index AC Identify the loss change indicator LOSS Model-A -8.544％ 2.8134 Model-B -6.660％ 2.1293

[0075] Compared with the second row of Table 1, for the voice control command recognition model Model-A, the calculated recognition accuracy change index is -8.544%, and the recognition loss change index LOSS is 2.8134, which means that for every 1 unit change in the decomposition degree, the recognition accuracy of Model-A decreases by 8.544%, and the recognition loss increases by 2.8134; after reinforcement training of Model-A, the calculated recognition accuracy change index of Model-B is -6.660%, and the recognition loss change index LOSS is 2.1293, which means that for every 1 unit change in the decomposition degree, the recognition accuracy of Model-B decreases by 6.660%, and the recognition loss increases by 2.1293. The experimental results show that Model-B is more robust than Model-A.

[0076] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for testing the robustness of intelligent models based on empirical mode decomposition, characterized in that: The following steps are involved: Step 1: Collect the original test data set based on the intelligent model to be tested: (1a) Initialize the test data set parameters, including the number of samples N and the sample data format; (1b) Based on the intelligent model to be tested, an original test data set of size N is collected; (1c) Reading n test data from the original test data set and preprocessing them based on the input format of the intelligent model to be tested; Step 2: Expand the original test dataset based on empirical mode decomposition: (2a) Based on the empirical mode decomposition method, each test data is decomposed into k eigensignals and 1 residual signal to obtain (k+1)·n expanded test data; Step 3: Calculate the recognition accuracy and recognition loss of the test data set at different decomposition levels: (3a) Initialize the recognition accuracy vector and the recognition loss vector; (3b) By accumulating different numbers of eigensignals of the same test data, k test data sets of different decomposition degrees of size n are obtained; (3c) Select a test data set with different decomposition levels and input it into the intelligent model to be tested to obtain n recognition results; (3d) Calculate the recognition accuracy and recognition loss based on the recognition results, and add them to the recognition accuracy vector and recognition loss vector respectively; (3e) Determine whether the number of cycles k has been reached. If so, jump to step 4; otherwise, return to step (3c); Step 4: Calculate the intelligent model recognition accuracy change index and recognition loss change index: (4a) Based on the recognition accuracy vector and the recognition loss vector, the recognition accuracy change index AC and the recognition loss change index LOSS are calculated.

2. The intelligent model robustness testing method based on empirical mode decomposition according to claim 1 is characterized in that: The step (1b) of collecting an original test data set of size N based on the intelligent model to be tested refers to collecting an original test data set of size N in a variety of physical environments for the application scenarios of the intelligent model to be tested.

3. The intelligent model robustness testing method based on empirical mode decomposition according to claim 1, characterized in that: The empirical mode decomposition method described in step (2a) decomposes each test data into k intrinsic signals and 1 residual signal to obtain (k+1)·n expanded test data, which means that the i-th sample data x i Decomposed into (k+1) empirical mode decomposition components by the empirical mode method where x ij Represents the i-th sample data x in the original test data set i The jth empirical mode decomposition component after empirical mode decomposition, that is, the jth eigensignal, j = 1, 2, 3, .... k; Represents the residual signal of the i-th sample in the original test data set after empirical mode decomposition.

4. The intelligent model robustness testing method based on empirical mode decomposition according to claim 3 is characterized in that: The step (3b) of obtaining k test data sets of different decomposition degrees of n by accumulating different numbers of eigensignals of the same test data refers to calculating the test data sets Y of different decomposition degrees using the (k+1)·n expanded test data obtained in step (2a). l , where l represents the degree of decomposition, l = 1, 2, 3, ..., k, Y l Each sample in is calculated according to the formula: <h2 style=";text-align:left;direction:ltr">Y<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> (y1,y2,y3,...,y)<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr">} Calculated; where y i Represents the test data set Y with a decomposition degree of l l The i-th sample in , l = 1, 2, 3, ..., k, i = 1, 2, 3, ..., n.

5. The intelligent model robustness testing method based on empirical mode decomposition according to claim 4 is characterized in that: The step (3d) in which the recognition accuracy and recognition loss are calculated based on the recognition results and added to the recognition accuracy vector and recognition loss vector respectively, refers to comparing the n recognition results obtained in step (3c) with the true labels of the samples to obtain the recognition accuracy c / n, where c represents the number of samples that are correctly identified; and then using the test data set Y with a decomposition degree of l selected in step (3c) l , according to the loss function loss(Y l ), calculate the recognition loss; finally, add the recognition accuracy and recognition loss to the recognition accuracy vector and recognition loss vector respectively.

6. The intelligent model robustness testing method based on empirical mode decomposition according to claim 5 is characterized in that: The recognition accuracy change index AC and the recognition loss change index LOSS calculated based on the recognition accuracy vector and the recognition loss vector in step (4a) refer to the recognition accuracy vector and the recognition loss vector obtained in step 3, according to the formula: Calculated; among them, Represents the recognition accuracy vector, a i represent The i-th element in is the recognition accuracy of the test data set with decomposition degree i, i = 1, 2, 3, ..., k, k represents the number of eigenvalues ​​of a sample after empirical mode decomposition; ac origin represents the recognition accuracy of the original test data set; 1 / k represents the average of the subsequent calculation results; AC represents the change in the recognition accuracy of the test data set for every 1 unit change in the decomposition degree; represents the recognition loss vector, b i represent The i-th element in is the recognition loss of the test data set with decomposition degree i; loss origin Represents the recognition loss of the original test dataset; LOSS represents the change in the recognition loss of the test dataset when the decomposition degree changes by 1 unit.

7. The intelligent model robustness testing method based on empirical mode decomposition according to claim 1, characterized in that: The intelligent model to be tested is a voice control command recognition model.

8. The intelligent model robustness testing method based on empirical mode decomposition according to claim 5, characterized in that: The loss function used is the cross entropy loss function.

9. A system for implementing the method according to any one of claims 1 to 8.

10. An application of the method according to any one of claims 1 to 8 in the technical field of artificial intelligence model robustness determination.

Images