A speech adversarial sample recognition model training method and system
By designing the Fratio cumulative value filter group to amplify the gap between speech adversarial samples, the problem of slight differences in the prior art is solved, and efficient and fast speech adversarial sample recognition model training is achieved, which improves recognition accuracy.
Patent Information
- Application Number
- CN202210095222.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-26
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-01-26
AI Technical Summary
The prior art is difficult to capture due to small differences when identifying speech adversarial samples, resulting in high cost of model training, slow training speed and low accuracy.
By designing the Fratio cumulative value filter group of audio data, amplify the tiny gap between normal audio and counteract audio, reduce model training costs, and improve training speed and accuracy.
This realizes speech adversarial sample recognition with high recognition accuracy at less training costs, simplifying the model training process.
Smart Images

Figure CN114550704B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of adversarial sample recognition, and in particular to a method and system for training a speech adversarial sample recognition model. Background Art
[0002] With the continuous development of artificial intelligence technology, more and more applications based on artificial intelligence technology have been widely used in life, including voice content recognition, smart assistants, etc. However, this type of method based on artificial intelligence models to handle business needs is vulnerable and easily attacked by voice adversarial samples. Voice adversarial samples are a type of attack that superimposes disturbances that are difficult for the human ear to detect on the original audio, causing errors in the recognition results of the artificial intelligence model.
[0003] Most existing methods for identifying speech adversarial samples use the slight differences between normal samples and adversarial samples in the time domain or frequency domain. However, since slight differences are difficult to capture, directly using deep learning networks is prone to excessive model training costs. Summary of the invention
[0004] In order to solve the above technical problems, the present invention provides a speech adversarial sample recognition model training method and system, which designs a filter group by utilizing the Fratio cumulative value of audio data, thereby amplifying the small gap between normal audio and adversarial audio, thereby reducing the cost of model training, increasing the training speed and improving the accuracy.
[0005] To achieve the above objectives, the first objective of the present invention is to provide a method for training a speech adversarial sample recognition model, comprising:
[0006] Step 1: Obtain an audio data set, where the audio data set includes normal audio data and adversarial audio data;
[0007] Step 2: Design the audio sampling frequency, obtain the spectrum of each audio data in the audio data set in the (0-m) frequency range, and generate spectrum features based on discrete sampling points; m represents the maximum value of the sampling frequency range;
[0008] Step 3: Calculate the Fratio value and Fratio cumulative value of all audio data in different frequency ranges, and obtain a monotonically increasing curve f of the Fratio cumulative value with respect to the sampling frequency;
[0009] Step 4: Design a filter bank based on the monotonically increasing curve f, and perform filtering operations on all audio in the audio data set;
[0010] Step 5: Use the filtered audio dataset to train the speech adversarial sample recognition model.
[0011] Furthermore, the sampling frequency range is 0-16k.
[0012] Furthermore, the calculation formula of the Fratio value is:
[0013]
[0014] in, represents the spectral features of the jth audio in the i-th category, j = 1, 2, ..., N, N represents the number of audios in the i-th category in the audio dataset, i = 1, 2, corresponding to the normal class and the adversarial class respectively; u i represents the mean of the spectral features of N audios in the i-th category, and u represents the mean of the spectral features of all audios.
[0015] Furthermore, the monotonically increasing curve f is specifically:
[0016] Set different sampling frequencies (m1, m2, ..., m) in the frequency range (0-m) i ,…,m n ), get the value in (0-m i ) frequency range, calculate the Fratio value of all audio data in this frequency range, recorded as The Fratio cumulative value is
[0017] Traverse all (0-m i ) frequency range combination, the Fratio cumulative value of all audio data in different frequency ranges is obtained, and a monotonically increasing curve f about frequency is constructed.
[0018] Furthermore, the design method of the filter bank is:
[0019] Divide the range of the monotonically increasing curve f into M equal markers. Each marker corresponds to a frequency point, which is the center frequency point of the filter. A set of filter center point vectors are obtained as follows:
[0020]
[0021] Among them, Max represents the maximum value of the range of the monotonically increasing curve f; (C i ,1) represents the center point of the i-th filter, which is related to the two adjacent points (C i-1 ,0) and (C i+1 ,0) connected, we can get a triangular filter; traverse C i ,i=1,2,…,M, and M triangular filters are obtained.
[0022] Furthermore, the spectral features of all the audio in the audio data set are convolved with the filter bank to implement the filtering operation of the filter bank on the audio.
[0023] Another object of the present invention is to provide a speech adversarial sample recognition system, comprising:
[0024] An audio preprocessing module, which uses the filter group obtained by the above-mentioned speech adversarial sample recognition model training method to perform filtering operations on all audio in the audio data set;
[0025] An adversarial sample recognition module, which is used to take the filtered audio data to be recognized as input and obtain a discrimination score for the audio;
[0026] The recognition result display module is used to compare the discrimination score with the threshold. If it is higher than the threshold, an alarm is issued to indicate that adversarial audio is detected, and a visual display is performed on the system interface.
[0027] Preferably, the audio preprocessing module includes:
[0028] An audio data set acquisition module, which is used to acquire an audio data set to be recognized;
[0029] A frequency domain transformation module is used to transform the audio data in the audio data set to be identified into a frequency domain to obtain a spectrum diagram, and generate spectrum features according to discrete sampling points;
[0030] The filtering module is used to perform convolution calculation on the spectrum features and the filter group to obtain the filtered audio data to be recognized.
[0031] Preferably, the adversarial sample recognition module is a trained speech adversarial sample recognition model.
[0032] Compared with the prior art, the present invention uses Fratio to design filters to amplify the small gap between normal audio and adversarial audio, thereby providing a new method for recognizing adversarial audio. The model training is simple and has higher recognition accuracy at a lower training cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 A flowchart of a speech adversarial sample recognition model training method provided by the present invention.
[0034] Figure 2 A schematic diagram of a filter design method provided by the present invention.
[0035] Figure 3 A deep learning model framework based on ResNet is provided in the present invention.
[0036] Figure 4A flowchart of a speech adversarial sample recognition system provided by the present invention. DETAILED DESCRIPTION
[0037] The technical framework of the invention is described below in conjunction with the accompanying drawings.
[0038] Most existing methods for identifying speech adversarial samples use the slight differences between normal samples and adversarial samples in the time domain or frequency domain. However, since slight differences are difficult to capture, directly using deep learning networks is prone to excessive model training costs.
[0039] In order to solve the problem of insensitivity to the difference between adversarial samples and normal samples in the prior art, an embodiment of the present invention provides a method and system for training a speech adversarial sample recognition model.
[0040] The following is a detailed description of the technical solutions provided by various embodiments of the present invention in conjunction with the accompanying drawings. The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps can be decomposed, while some steps can be combined or partially combined, so the actual execution order may change according to the actual situation.
[0041] A speech adversarial sample recognition model training method, as shown in the attached Figure 1 As shown, the method includes:
[0042] Step S101, obtaining an audio data set, including normal audio data and adversarial audio data; in this embodiment, the obtained normal audio data and adversarial audio data are about 5000 each, where the adversarial audio data label is marked as 1, and the normal audio data label is marked as 0.
[0043] Step S102: Calculate the Fratio value using all the audio contained in the data set.
[0044] Step S103, designing a filter bank according to the Fratio value, and filtering the audio.
[0045] Step S104: input the filtered audio data into a speech adversarial sample recognition model to train the deep learning model.
[0046] like Figure 2 As shown, the process of designing the filter is as follows:
[0047] The present invention designs an M-dimensional filter bank. First, the audio data in the audio data set is transformed in the frequency domain to obtain a spectrum in the range of 0-8k, which is discretized into 64 sampling points to generate spectrum features. In this embodiment, the 0-8k frequency range is divided into 60 equal parts to obtain (0, m1, m2, ..., m i ,…,8k), get thei ) frequency range, calculate the Fratio value of all audio data in this frequency range, recorded as The Fratio cumulative value is The Fratio value can be used to enhance the high-discrimination information between normal audio data and adversarial audio data, and suppress individual or language information in the audio. By introducing the Fratio value, the present invention can make subsequent training more focused on the difference between normal audio and adversarial audio, and reduce the focus on voice content and speaker differences.
[0048] The calculation formula of Fratio value is:
[0049]
[0050] in, represents the spectral features of the jth audio in the i-th category, j = 1, 2, ..., N, N represents the number of audios in the i-th category in the audio dataset, i = 1, 2, corresponding to the normal class and the adversarial class respectively; u i represents the mean of the spectral features of N audios in the i-th category, and u represents the mean of the spectral features of all audios.
[0051] Traverse all (0-m i ) frequency range combination, and obtain the Fratio cumulative value of all audio data in different frequency ranges. Since each cumulative value is will increase with the increase of i, so we can get a monotonically increasing curve f about frequency, such as Figure 2 Frequency (Hz)-Fratio cumulative value curve in.
[0052] Divide the range of the monotonically increasing curve f into M equal markers. Each marker corresponds to a frequency point, which is the center frequency point of the filter. A set of filter center point vectors are obtained as follows:
[0053]
[0054] Among them, Max represents the maximum value of the range of the monotonically increasing curve f; (C i ,1) represents the center point of the i-th filter, which is related to the two adjacent points (C i-1 ,0) and (C i+1 ,0) connected, we can get a triangular filter; traverse C i ,i=1,2,…,M, we get M triangular filters, such as Figure 2 Schematic diagram of the frequency (Hz)-amplitude triangular filter in the figure, with a total of 24 triangular filters.
[0055] In this embodiment, this step finally obtains a set of 64×256-dimensional filters. The spectral features of all audio in the audio data set and the filter group are convolved to obtain preprocessed data with a dimension of (T, 64), which is used for subsequent model training, where T represents the duration of the audio.
[0056] like Figure 3 As shown in the figure, the ResNet deep model is used as the speech adversarial sample recognition model to be trained. It is composed of three convolutional layers and three residual modules alternately. The size of the convolutional layer is (5×5), and the channels are 64, 128 and 256 respectively. In the residual block, two (3×3) convolutional layers are included. After passing through the pooling layer, the feature passes through multiple groups of fully connected layers to obtain the final score. The higher the score, the greater the possibility that it is adversarial audio.
[0057] The present invention also discloses a speech adversarial sample recognition system, Figure 4 The workflow of the speech adversarial sample recognition system is demonstrated, and the speech adversarial sample recognition system includes:
[0058] An audio preprocessing module, which uses the filter group obtained by the above-mentioned speech adversarial sample recognition model training method to perform filtering operations on all audio in the audio data set;
[0059] An adversarial sample recognition module, which is used to take the filtered audio data to be recognized as input and obtain a prediction score for the audio;
[0060] The recognition result display module is used to compare the discrimination score with the threshold. If it is higher than the threshold, an alarm is issued to indicate that an adversarial sample has been detected, and a visual display is performed on the system interface.
[0061] The adversarial sample recognition module is the above-mentioned trained speech adversarial sample recognition model. For example, in the above process, a ResNet deep learning model is obtained by training 5000 normal audios and 5000 adversarial audios.
[0062] The recognition result display module visualizes the recognition result. The calculated prediction score ε is compared with the threshold ∈. If it is higher than the threshold, the recognized result is played by voice and an alarm sound is played; the model prediction result is displayed in text on the interface, and a warning is given that the audio to be tested may be an adversarial sample. The threshold ∈ is set manually, and is set to 0.6 in this embodiment.
[0063] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only illustrative, wherein the audio preprocessing module may or may not be physically separated. In addition, each functional module in the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit, or may be further split. For example, the audio preprocessing module includes:
[0064] An audio data set acquisition module, which is used to acquire an audio data set to be recognized;
[0065] A frequency domain transformation module is used to transform the audio data in the audio data set to be identified into a frequency domain to obtain a spectrum diagram, and generate spectrum features according to discrete sampling points;
[0066] The filtering module is used to perform convolution calculation on the spectrum features and the filter group to obtain the filtered audio data to be recognized.
[0067] The above-mentioned integrated modules or units can be implemented in the form of hardware or in the form of software functional units, and some or all of the modules can be selected according to actual needs to achieve the purpose of the present application.
[0068] It should be understood by those skilled in the art that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may be deformed or modified in any way without departing from the principles.
Claims
1. A method for training a speech adversarial sample recognition model, characterized in that: include: Step 1: Obtain an audio data set, where the audio data set includes normal audio data and adversarial audio data; Step 2: Design the audio sampling frequency, obtain the spectrum of each audio data in the audio data set in the frequency range of 0-m, and generate spectrum features based on discrete sampling points; m represents the maximum value of the sampling frequency range; Step 3: Calculate the Fratio value and Fratio cumulative value of all audio data in different frequency ranges, and obtain a monotonically increasing curve f of the Fratio cumulative value with respect to the sampling frequency; The monotonically increasing curve f is specifically: Set different sampling frequencies m1, m2, ..., m in the frequency range 0-m i , ..., m n , get the value between 0-m i Spectrum of the frequency range, calculate the Fratio value of all audio data in the frequency range, recorded as The Fratio cumulative value is Traverse all 0-m i The frequency range is combined to obtain the Fratio cumulative value of all audio data in different frequency ranges, forming a monotonically increasing curve f about frequency; Step 4: Design a filter bank based on the monotonically increasing curve f, and perform filtering operations on all audio in the audio data set; The design method of the filter bank is: Divide the range of the monotonically increasing curve f into M equal markers. Each marker corresponds to a frequency point, which is the center frequency point of the filter. A set of filter center point vectors are obtained as follows: Among them, Max represents the maximum value of the range of the monotonically increasing curve f; (C i , 1) represents the center point of the i-th filter, which is related to the two adjacent points (C i-1 , 0) and (C i+1 , 0) are connected to obtain a triangular filter; traversing C i , i = 1, 2, ..., M, and M triangular filters are obtained; Step 5: Use the filtered audio dataset to train the speech adversarial sample recognition model.
2. The speech adversarial sample recognition model training method according to claim 1 is characterized in that: The sampling frequency range is 0-16k.
3. The method for training a speech adversarial sample recognition model according to claim 1, characterized in that: The calculation formula of the Fratio value is: in, represents the spectral features of the jth audio in the i-th category, j = 1, 2, ..., N, N represents the number of audios in the i-th category in the audio dataset, i = 1, 2, corresponding to the normal class and the adversarial class respectively; u i represents the mean of the spectral features of N audios in the i-th category, and u represents the mean of the spectral features of all audios.
4. The speech adversarial sample recognition model training method according to claim 1, characterized in that: The spectral features of all audio in the audio data set are convolved with the filter bank to implement the filtering operation of the filter bank on the audio.
5. A speech adversarial sample recognition system, characterized in that: include: An audio preprocessing module, which uses the filter group obtained by the speech adversarial sample recognition model training method according to claim 1 to perform filtering operations on all audio in the audio data set; An adversarial sample recognition module, which is used to take the filtered audio data to be recognized as input and obtain a discrimination score for the audio; The recognition result display module is used to compare the discrimination score with the threshold. If it is higher than the threshold, an alarm is issued to indicate that adversarial audio is detected, and a visual display is performed on the system interface.
6. The speech adversarial sample recognition system according to claim 5, characterized in that: The audio preprocessing module comprises: An audio data set acquisition module, which is used to acquire an audio data set to be recognized; A frequency domain transformation module is used to transform the audio data in the audio data set to be identified into a frequency domain to obtain a spectrum diagram, and generate spectrum features according to discrete sampling points; A filtering module is used to perform convolution calculation on the frequency spectrum feature and the filter group obtained in claim 1 to obtain filtered audio data to be recognized.
7. The speech adversarial sample recognition system according to claim 5, characterized in that: The adversarial sample recognition module is the speech adversarial sample recognition model trained in claim 1.
