End-to-end recording device source identification method, identification system, and computer device

Abstract

The present invention belongs to the technical field of digital audio passive forensics, and discloses an end-to-end recording device source identification method, identification system, and computer equipment. Mel cepstral coefficients in digital audio are extracted as device source features according to time series frames, and then divided into time series feature segments to extract Gaussian mean matrices respectively to obtain time series Gaussian mean matrix features; deep representation learning is performed on the time series Gaussian mean matrix features using a convolutional neural network to extract deep bottleneck features; the time domain characteristics in the deep time series bottleneck features are extracted through a bidirectional long short-term memory neural network to identify and classify the recording device source. The present invention can effectively detect and distinguish a large number of recording device models, and fully considers the particularity of digital audio itself, thereby improving the accuracy and efficiency of the algorithm; when the present invention performs a recording device source task, the required amount of voice data is small, and each audio file only needs a few seconds in length, and there is no need to deliberately use non-voice segments for identification.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital audio passive forensics, and in particular relates to an end-to-end recording device source identification method based on temporal supervector representation learning. Background Art

[0002] At present, in the past decade, with the rapid development and progress of the electronic information industry, mobile devices such as smart phones and tablet computers have become widely popular, and the convenience of audio recording, storage and transmission has been greatly improved. Recording operations can be performed on mobile devices anytime and anywhere, and are no longer restricted by the need to use professional recording equipment.

[0003] When recording using a mobile device such as a smartphone, the recorded audio is saved in the form of a digital audio data file. The digital audio data file obtained through recording carries a lot of information such as voice content, language type, speaker information, speaker emotion, speaker gender, environmental noise, etc., which also contains information related to the recording device. In tasks related to voice content or speaker recognition, the feature information used comes from the biometric features of the speaker's voice, while in the recording device source recognition task, the feature information used comes from the differential features generated during the digital signal transmission process.

[0004] When using a mobile device to record audio, due to differences in the internal circuit structure and components of different recording devices, the voice excitation signal generated during the audio recording process will be transmitted differently in the device circuit. This difference causes different recording devices to leave distinguishing characteristic traces in the audio file during the voice collection process.

[0005] Many researchers have done some work in the field of recording device source identification, including recording device clustering and identification. Before deep learning technology was applied to the field of recording device source identification, it was usually done by extracting spectral features in the audio, such as MFCC, LFCC, and IMFCC, and then using specific machine learning methods to analyze the features for classification.

[0006] In recent years, due to the rapid development of deep learning technology, many researchers have applied deep learning technology to the field of recording device source identification research, and recording device source identification technology has achieved rapid development. Deep neural networks have powerful representation learning capabilities, which can automatically extract highly abstract and complex features from raw data, automatically find the most relevant feature information for classification tasks, and efficiently extract the most discriminative potential internal features. CNN (convolutional neural network) and DNN (deep neural network) have been proven to be effective methods for recording device source identification and extracting spatial correlations in relevant feature domains. With the development and progress of deep learning technology, the development of neural networks is moving towards a trend of higher and higher network layers and more and more complex network structures, because deep complex networks have stronger fitting capabilities than shallow single networks, can extract more features, and can solve more complex problems. Combined networks show obvious advantages over single networks.

[0007] At the same time, from the perspective of features, previous recording device source identification methods mostly used spectral short-time spectrum features in audio, such as MFCC, LFCC and IMFCC. These features are acoustic features extracted directly from the recording data, and contain interference information that is irrelevant to the recording device source identification, including speech content, environmental noise, speaker information, etc. Some researchers have used the GMM (Gaussian mixture model) method to further extract more expressive Gaussian Super Vector (GSV) features, but these features only use the spatial information obtained from the frequency domain characteristics of the data, while ignoring the time domain characteristics in the continuous audio data, and discarding the timing information in the recording device source signal.

[0008] The recent update of recording device source identification methods is closely linked to the development of deep learning technology. Some researchers have used new deep learning network models and ideas to improve recording device source identification technology, and many excellent deep network models have been applied to the field of recording device source identification. Network models using recurrent neural networks have been applied to recording device source identification tasks, but in terms of features, short-term spectrum features such as MFCC are still used, and the ability to represent device source feature information is very limited.

[0009] There is a common problem with these previous methods, that is, the features are not fully utilized. Most methods focus on the analysis of audio spatial information and lack consideration of temporal information, resulting in the loss of some effective information. In addition, most traditional models or deep learning models contain many manually designed parameters, and the quality of features depends on the judgment of human subjective experience, which makes the recording device source identification task have poor transferability.

[0010] Through the above analysis, the problems and defects of the existing technology are as follows:

[0011] (1) Directly using the short-time spectrum features of speech signals as device source features only focuses on the use of spatial information contained in frequency domain features, does not make full use of features, and lacks consideration of temporal information, which will cause the loss of some effective information during the recognition process;

[0012] (2) Traditional GSV features, when extracting device source feature information using the Gaussian mixture model, completely discard the sequence structure in the acoustic features of the previous stage and only contain the spatial information of the device source signal.

[0013] (3) The existing non-end-to-end recording device source recognition model requires manual control of each stage continuously, and there are many parameters that need to be manually adjusted. The complexity is very high when it is applied, which makes it impossible to apply to other scenarios or other audio device source recognition, and the recognition accuracy is low.

[0014] The difficulty of solving the above problems and defects is: how to extract more expressive features from the recording to distinguish the characteristics of the recording device. How to design an efficient neural network model to effectively improve the accuracy of recording device source identification.

[0015] The significance of solving the above problems and defects is:

[0016] In the task of recording device source identification, the present invention simultaneously analyzes the spatial complexity and temporal complexity of the device source signal, and uses a combined network model corresponding to the timing characteristics to demonstrate a stronger fitting characterization capability, fully and efficiently utilizing the recording device source feature information in the audio data, and realizing the due forensic value of limited audio data evidence, providing an executable and efficient solution to the problem of voice data source forensics. Summary of the invention

[0017] In view of the problems existing in the prior art, the present invention provides an end-to-end recording device source identification method based on temporal supervector representation learning.

[0018] The present invention is implemented as follows: an end-to-end recording device source identification method based on time series supervector representation learning, the end-to-end recording device source identification method based on time series supervector representation learning comprises:

[0019] The Mel-frequency cepstral coefficients in digital audio are extracted as device source features according to time series frames, and then divided into time series feature segments to extract Gaussian mean matrices respectively to obtain sequential Gaussian mean matrix (SGMM) features; the convolutional neural network is used to perform deep representation learning on the sequential Gaussian mean matrix features to extract deep bottleneck features; the bidirectional long short-term memory neural network is used to extract the time domain characteristics in the deep sequential bottleneck features to identify and classify the recording device source.

[0020] Furthermore, the end-to-end recording device source identification method based on temporal supervector representation learning comprises the following steps:

[0021] Step 1: extract the Mel-frequency cepstral coefficient (MFCC) features from the source signal of the recording device, and retain the MFCC features of the same number of frames for each audio data;

[0022] Step 2: Use MFCC feature training to establish a Gaussian mixture universal background model, namely the GMM-UBM model, and extract the time series Gaussian mean matrix by the GMM-UBM model after segmenting the MFCC features;

[0023] Step 3: Use the constructed convolutional neural network to perform deep representation learning, extract deep bottleneck features from the time series Gaussian mean matrix, and retain the time series structure of the time series Gaussian mean matrix features;

[0024] Step 4: Perform time series modeling of the device source identification network, build a classification model based on a bidirectional long short-term memory neural network, and use the classification model to identify and classify the recording device source based on the extracted deep bottleneck features.

[0025] Further, the extracting of Mel-frequency cepstral coefficient MFCC features in the source signal of the recording device and retaining the MFCC features of the same number of frames for each audio data includes:

[0026] Preprocessing the source signal of the recording device by frame division and windowing; performing fast Fourier transform on the source signal of the recording device, and performing Mel filtering on the obtained amplitude spectrum signal using a set of Mel-scale triangular filter groups;

[0027] The logarithmic magnitude spectrum of each filter group output is calculated and discrete cosine transform is performed to obtain MFCC features. The MFCC features of the first n frames are retained for each recording data.

[0028] Furthermore, the pre-processing of the source signal of the recording device by framing and windowing includes: taking a frame length of 256 ms, using 1 / 2 of the frame length as a frame shift, and using a Hamming window function for windowing.

[0029] Further, in step 2, the Gaussian mixture universal background model, i.e., GMM-UBM model, is established by using MFCC feature training, and the time series Gaussian mean matrix is ​​extracted by the GMM-UBM model after segmenting the MFCC features, including:

[0030] Using the recording data of various recording devices, a diagonal covariance universal background model (UBM model) with n Gaussian number is trained to obtain the weight coefficient matrix, mean matrix and covariance matrix of n Gaussian mixtures; using the maximum a posteriori estimation algorithm for the feature vector of each device in the training data, the universal background model is adaptively obtained to obtain the GMM model of the specific device source;

[0031] The extracted MFCC features are divided into t continuous feature segments, and the mean matrix of GMM is extracted by the GMM-UBM model. The n mean matrices of each GMM are sequentially connected into a mean matrix with a total length of n × input MFCC dimension, that is, the time-series Gaussian mean matrix feature.

[0032] Further, in step 3, the deep representation learning is performed using the constructed convolutional neural network, the deep bottleneck features are extracted from the time series Gaussian mean matrix, and the time series structure of the time series Gaussian mean matrix features is retained, including:

[0033] Construct a convolutional neural network for deep representation learning, use Gaussian mean matrix features to train and test the network model;

[0034] The deep bottleneck features of the trained convolutional neural network model are extracted and saved in the form of a time series structure of Gaussian mean matrix features.

[0035] Furthermore, the deep representation learning is performed using the constructed convolutional neural network, the deep bottleneck features are extracted from the time series Gaussian mean matrix, and the time series structure of retaining the time series Gaussian mean matrix features also includes:

[0036] Use labeled training sets to conduct supervised training on deep neural networks and build a representation model based on input features; based on the representation model, select the activation value of the appropriate layer as the deep bottleneck feature;

[0037] The specific steps are as follows:

[0038] The obtained one-dimensional feature vector is expanded into a two-dimensional feature map, and the frequency domain feature information is extracted using a convolutional neural network. The obtained SGMM features are input into the constructed convolutional neural network. The input form of each recording device source data sample is 80 feature matrices of 12*16 size arranged in time sequence, where 12 represents the MFCC feature dimension and 16 represents the Gaussian number of the GMM model.

[0039] The convolution kernel is used to perform a convolution operation on the input features to obtain relevant feature information. After the convolutional neural network model is trained, the training set and test set data are input into the trained convolutional neural network model, and the activation value of the bottleneck layer is used as the output and saved in time series.

[0040] Furthermore, the classification model based on the bidirectional long short-term memory neural network includes two BiLSTM layers, a Flatten processing layer and a Softmax classifier;

[0041] Each BiLSTM layer contains 128 units;

[0042] Flatten processing layer, used to convert feature vectors into one-dimensional features;

[0043] A Softmax classifier is used to make a decision based on the one-dimensional feature.

[0044] Another object of the present invention is to provide an end-to-end recording device source identification system based on time series supervector representation learning for implementing the end-to-end recording device source identification method based on time series supervector representation learning, the end-to-end recording device source identification system based on time series supervector representation learning comprising:

[0045] MFCC feature extraction module, used to extract recording device source feature information of recording device source data during training and recognition;

[0046] The GMM-UBM model building module is used to build a universal background model for recording device source identification and obtain the GMM model of each target device source through MAP adaptation;

[0047] SGMM feature extraction module, used to extract the Gaussian mean matrix of the target device recording data in time sequence;

[0048] Convolutional neural network module, used to train convolutional neural networks and extract deep bottleneck features;

[0049] The recurrent neural network module is used to train the BiLSTM network with the extracted time series bottleneck features for automatic identification and classification.

[0050] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the end-to-end recording device source identification method based on temporal supervector representation learning.

[0051] Combining all the above technical solutions, the advantages and positive effects of the present invention are as follows:

[0052] The present invention can effectively detect and distinguish a large number of recording device models, and fully considers the particularity of digital audio itself, thereby improving the accuracy and efficiency of the algorithm; when performing a recording device source task, the present invention requires a small amount of voice data, each audio file only needs a length of several seconds, and does not need to use non-voice segments specifically for recognition; the data can be fully utilized, and the method of using GMM-UBM to extract the Gaussian mean matrix eliminates the interference of other information in the voice data on the recording device source recognition task.

[0053] The present invention extracts the time-series Gaussian mean matrix features from the time-series MFCC features, and simultaneously retains its spatial information and time-series information on a single feature.

[0054] The network model designed by the present invention is composed of a combination of two types of neural networks, which realizes the effective reuse of the source-related feature information of the recording device. The present invention first uses a convolutional neural network to perform a preliminary analysis and extraction of the feature information, and extracts and refines the spatial information in the audio features in the form of deep bottleneck features without destroying the input feature temporal structure. Then, a bidirectional long short-term memory neural network is used to further analyze the acquired deep bottleneck features. On the basis of reusing the spatial feature information, the temporal information in the audio features is also mined, realizing full and efficient feature utilization. The recurrent neural network takes sequence information as input and performs well in processing sequence data.

[0055] In the task of recording device source identification, the present invention fully and efficiently utilizes the recording device source feature information in the audio data, realizes the due forensic value of limited audio data evidence, and provides an executable and efficient solution to the problem of voice data source forensics. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of an end-to-end recording device source identification method based on temporal supervector representation learning provided by an embodiment of the present invention.

[0057] Figure 2 It is a flow chart of an end-to-end recording device source identification method based on temporal supervector representation learning provided by an embodiment of the present invention.

[0058] Figure 3 It is a schematic diagram of an end-to-end recording device source identification system based on temporal supervector representation learning provided by an embodiment of the present invention.

[0059] Figure 4 It is a schematic diagram of the time series Gaussian mean matrix extraction process provided by an embodiment of the present invention.

[0060] Figure 5 It is a structural diagram of a convolutional neural network provided in an embodiment of the present invention.

[0061] Figure 6 It is a structural diagram of a bidirectional long short-term memory neural network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] In view of the problems existing in the prior art, the present invention provides an end-to-end recording device source identification method based on temporal supervector representation learning. The present invention is described in detail below in conjunction with the accompanying drawings.

[0064] like Figure 1 As shown, the end-to-end recording device source identification method based on time series supervector representation learning provided by the embodiment of the present invention includes:

[0065] The Mel-frequency cepstral coefficients in the digital audio are extracted as the device source features according to the time series frames, and then divided into time series feature segments to extract the Gaussian mean matrix respectively, so as to obtain the time series Gaussian mean matrix features; the convolutional neural network is used to perform deep representation learning on the time series Gaussian mean matrix features to extract the deep bottleneck features; the bidirectional long short-term memory neural network is used to extract the time domain characteristics in the deep time series bottleneck features to identify and classify the recording device source.

[0066] like Figure 2 As shown, the end-to-end recording device source identification method based on time series supervector representation learning provided by the embodiment of the present invention includes the following steps:

[0067] S101, extracting Mel-frequency cepstral coefficient (MFCC) features from a source signal of a recording device, and retaining MFCC features of the same number of frames for each piece of audio data;

[0068] S102, using MFCC feature training to establish a Gaussian mixture universal background model, namely a GMM-UBM model, and extracting a sequential Gaussian mean matrix (SGMM) by the GMM-UBM model after segmenting the MFCC features;

[0069] S103, using the constructed convolutional neural network to perform deep representation learning, extracting deep bottleneck features from the time series Gaussian mean matrix, and retaining the time series structure of the time series Gaussian mean matrix features;

[0070] S104, performing device source identification network time series modeling, building a classification model based on a bidirectional long short-term memory neural network, and using the classification model to identify and classify the recording device source based on the extracted deep bottleneck features.

[0071] The method for extracting Mel-frequency cepstral coefficient MFCC features from a source signal of a recording device and retaining the same number of frames of MFCC features for each audio data provided by an embodiment of the present invention includes:

[0072] Preprocessing the source signal of the recording device by frame division and windowing; performing fast Fourier transform on the source signal of the recording device, and performing Mel filtering on the obtained amplitude spectrum signal using a set of Mel-scale triangular filter groups;

[0073] The logarithmic magnitude spectrum of each filter group output is calculated and discrete cosine transform is performed to obtain MFCC features. The MFCC features of the first n frames are retained for each recording data.

[0074] The preprocessing of the source signal of the recording device by framing and windowing provided in the embodiment of the present invention includes: taking a frame length of 256ms, using 1 / 2 of the frame length as a frame shift, and using a Hamming window function for windowing.

[0075] The embodiment of the present invention provides a Gaussian mixture universal background model, i.e., a GMM-UBM model, which is established by using MFCC feature training. After the MFCC features are segmented, the time series Gaussian mean matrix is ​​extracted by the GMM-UBM model, including:

[0076] Using the recording data of various recording devices, a diagonal covariance universal background model (UBM model) with n Gaussian number is trained to obtain the weight coefficient matrix, mean matrix and covariance matrix of n Gaussian mixtures; using the maximum a posteriori estimation algorithm for the feature vector of each device in the training data, the universal background model is adaptively obtained to obtain the GMM model of the specific device source;

[0077] The extracted MFCC features are divided into t continuous feature segments, and the mean matrix of GMM is extracted by the GMM-UBM model. The n mean matrices of each GMM are sequentially connected into a mean matrix with a total length of n × input MFCC dimension, that is, the time-series Gaussian mean matrix feature.

[0078] The embodiment of the present invention provides a method for performing deep representation learning using a constructed convolutional neural network, extracting deep bottleneck features from a time series Gaussian mean matrix, and retaining a time series structure of time series Gaussian mean matrix features, including:

[0079] Construct a convolutional neural network for deep representation learning, use Gaussian mean matrix features to train and test the network model;

[0080] The deep bottleneck features of the trained convolutional neural network model are extracted and saved in the form of a time series structure of Gaussian mean matrix features.

[0081] The method provided by the embodiment of the present invention for performing deep representation learning using a constructed convolutional neural network and extracting deep bottleneck features from a time series Gaussian mean matrix also includes:

[0082] The deep neural network is trained in a supervised manner using a labeled training set to build a representation model based on the input features. Based on the representation model, the activation value of the appropriate layer is selected as the deep bottleneck feature.

[0083] The steps of using the constructed convolutional neural network to perform deep representation learning, extracting deep bottleneck features from the time series Gaussian mean matrix, and retaining the time series structure of the time series Gaussian mean matrix features provided by the embodiment of the present invention are as follows:

[0084] The obtained one-dimensional feature vector is expanded into a two-dimensional feature map, and the frequency domain feature information is extracted using a convolutional neural network. The obtained SGMM features are input into the constructed convolutional neural network. The input form of each recording device source data sample is 80 feature matrices of 12*16 size arranged in time sequence, where 12 represents the MFCC feature dimension and 16 represents the Gaussian number of the GMM model.

[0085] The convolution kernel is used to perform a convolution operation on the input features to obtain relevant feature information. After the convolutional neural network model is trained, the training set and test set data are input into the trained convolutional neural network model, and the activation value of the bottleneck layer is used as the output and saved in time series.

[0086] The classification model based on the bidirectional long short-term memory neural network provided in the embodiment of the present invention includes two BiLSTM layers, a Flatten processing layer and a Softmax classifier;

[0087] Each BiLSTM layer contains 128 units;

[0088] Flatten processing layer, used to convert feature vectors into one-dimensional features;

[0089] A Softmax classifier is used to make a decision based on the one-dimensional feature.

[0090] like Figure 3 As shown, the end-to-end recording device source identification system based on time series supervector representation learning provided by the embodiment of the present invention includes:

[0091] MFCC feature extraction module, used to extract recording device source feature information of recording device source data during training and recognition;

[0092] The GMM-UBM model building module is used to build a universal background model for recording device source identification and obtain the GMM model of each target device source through MAP adaptation;

[0093] SGMM feature extraction module, used to extract the Gaussian mean matrix of the target device recording data in time sequence;

[0094] Convolutional neural network module, used to train convolutional neural networks and extract deep bottleneck features;

[0095] The recurrent neural network module is used to train the BiLSTM network with the extracted time series bottleneck features for automatic identification and classification.

[0096] The technical solution of the present invention is further described below in conjunction with specific embodiments.

[0097] Embodiment 1:

[0098] A flow chart of a recording device source identification system. The recording device source identification method and system of the present invention can be implemented according to the following steps:

[0099] Step 1.1, extract the MFCC feature information of the recording device source data. The MFCC feature extraction process includes preprocessing (framing, windowing), fast Fourier transform (FFT), Mel filtering, logarithmic operation, discrete cosine transform and other steps. The recording data of the data set are recorded by 45 different devices produced by 8 different brands. The brands of the devices include Apple, Huawei, Nubia, OPPO, vivo, Xiaomi, Samsung, and ZTE. The recording devices include smartphones and tablets. The data of each model of equipment includes a total of 642 speech sample segments, and each sample is about 10s long. The specific extraction process is as follows: First, in order to allow the recording device source signal to be stably represented in the frequency domain, the entire audio data signal needs to be frame processed, that is, the input audio data signal is framed and windowed, the frame length is 256ms, and 1 / 2 of the frame length is used as the frame shift, and the Hamming window function is used for windowing. The window function expression of the Hamming window is as follows:

[0100]

[0101] Then, the frequency spectrum of each frame is obtained by fast Fourier transform of each frame signal after frame division and windowing, and the amplitude spectrum is selected for filtering through the Mel filter group. Mel filtering is a group of frequency filter groups that imitate the auditory effect of the human ear. People's perception of frequency is divided by critical bands. According to the division of critical bands, the speech is divided into a series of frequency groups in the frequency domain to form a Mel filter group. The expression of Mel filter is as follows:

[0102] F mel =1125log(1+f / 700)

[0103] The amplitude spectrum signal is passed through a set of Mel-scale triangular filter banks whose center frequencies are evenly distributed on the Mel scale. Then the logarithmic amplitude spectrum of each filter bank output is calculated and the MFCC coefficients are obtained by discrete cosine transform (DCT).

[0104] Step 1.2, MFCC features are features extracted by time-series frames. Usually, in the process of extracting Gaussian supervector features based on MFCC, the feature information carried in MFCC is simplified and refined, and the information density and utilization efficiency are enhanced and improved, but at the same time, the time sequence information carried by MFCC is also lost. Here, the present invention extracts SGMM features by dividing the extracted MFCC features into a feature matrix segment for every several consecutive frames. The 640 frames of MFCC features obtained from each speech sample extract SGMM features of 80 feature matrix segments (8 frames / segment). The steps of extracting SGMM features are as follows: First, a diagonal covariance universal background model (UBM) with n Gaussian numbers is trained by all speech data, and the EM algorithm is used to estimate the parameters of the mixed Gaussian model to obtain the weight coefficient matrix, mean matrix and covariance matrix of n Gaussian mixtures. Then, by using the maximum a posteriori probability algorithm (MAP), the feature vector of each device in the training data is adaptively obtained to obtain the specific device source GMM. The probability density of a GMM is the weighted sum of the probability density distributions of K Gaussian components. The formula for the probability density of the mixed Gaussian is given as follows:

[0105]

[0106] where x t is the random variable of the tth time series, p k represents the sub-distribution of each Gaussian component, ω k Then the mixture weight of each Gaussian component is expressed. The complete mixture Gaussian model consists of weight parameters, mean vector and covariance matrix, which is expressed as:

[0107] λ={w k ,u k ,Σ k},k=1,…,K

[0108] Then extract the mean matrix of GMM, and finally concatenate the n mean matrices of each GMM into a mean matrix with a total length of n × input MFCC dimension. In this example, the extracted MFCC feature dimension is 12-dimensional, 640 frames of MFCC features are extracted for each speech sample, and SGMM feature extraction is performed after being divided into 80 time-series speech feature segments. The Gaussian number of the GMM model is set to 16, and finally each recording device source data sample obtains feature data with a total length of 80*12*16.

[0109] Step 2, for image-related tasks, the present invention inputs image pixels into the neural network, but for the recording device source recognition task, the original speech spectrum signal cannot be directly input into the neural network. The present invention performs a series of feature extraction on the original speech signal, expands the one-dimensional feature vector extracted by steps 1 and 2 into a two-dimensional feature map, and inputs it into the neural network, and the input size is 12*16. The extraction of deep bottleneck features is divided into two steps. First, the deep neural network is supervisedly trained using a labeled training set to construct a characterization model based on the input features. Subsequently, the activation value of the appropriate layer is selected as the deep bottleneck feature based on the characterization model. The specific operation is as follows: First, the frequency domain feature information is extracted using a convolutional neural network, and the obtained SGMM features are input into the convolutional neural network designed by the present invention. The input form of each recording device source data sample is 80 feature matrices of 12*16 size arranged in time sequence, where 12 represents the MFCC feature dimension and 16 represents the Gaussian number of the GMM model. The convolution kernel is used to perform a convolution operation on the input feature to obtain relevant feature information. This operation can extract deep feature information from the feature. Compared with the image recognition task, the input feature map size obtained by the device source recognition is much smaller than that of the general image. The high-dimensional convolution will make the generated feature map very small. The use of a low-dimensional convolution layer can avoid the loss of certain features, so that the neural network can extract more representative features and improve the recognition accuracy. In addition, the present invention also adds a 1×1 convolution kernel to reduce the dimension, which allows information to be transmitted through fewer connections, thereby obtaining sparser features. After the convolutional neural network model is trained, the training set and the test set data are input into the trained convolutional neural network model, and the activation value of the bottleneck layer is used as the output and saved in time series.

[0110] Step 4, after the deep bottleneck features are extracted by the convolutional neural network, the present invention uses BiLSTM to further extract feature information, making full use of the timing feature information carried by the features. The LSTM network can operate the timing state of each unit through the gate control structure, delete or add information, and transmit useful feature information to the next timing unit. BiLSTM is the abbreviation of bidirectional long-term and short-term memory, which is a combination of forward LSTM and backward LSTM. Compared with the structure of LSTM, the prediction of BiLSTM is determined by the first few inputs and the last few inputs, which will make the prediction results more accurate and make it more representative when extracting the timing features of the device source. Here, the present invention designs two layers of BiLSTM layers as a network model for training, each layer containing 128 units. The Softmax classifier is used for classification judgment. After the network, the Flatten operation is performed to convert the feature vector into one dimension and enter the Softmax classifier for judgment.

[0111] The present invention uses two neural networks, takes advantage of deep learning, and uses different network strategies to deal with different characteristics in speech data, effectively utilizing the characteristic information of the frequency domain characteristics and time domain characteristics of speech data at the same time. In the comparative test with other methods on the same data set, the recognition accuracy of the present invention has achieved a leading rate.

[0112] Table 1 Experimental process comparison experiment

[0113]

[0114] The technical solution of the present invention is further described below in conjunction with the working principle.

[0115] The working principle of the present invention is as follows:

[0116] From recording to generating files, audio data files must go through multiple stages such as induction current, operational amplification, analog-to-digital conversion, voice coding, channel coding and modulation. Due to the tolerances of electronic components produced by various brands and models of recording equipment, the transmission of electrical signals generated by recording in electronic circuits will be different, resulting in different transfer functions from voice signals to the final voice coding modulation. Transfer functions vary in different devices, and this difference becomes more obvious when recording the same sound using audio from different manufacturers. When performing audio recording tasks, different recording devices will leave inherent differences in the sounds they record.

[0117] Audio signals have both time domain and frequency domain characteristics. Existing device source identification methods often use the frequency characteristics of speech signals for identification, ignoring the timing information contained in continuous speech signals. From the time domain point of view, the present invention cannot observe the differences between recordings of different devices. Therefore, the present invention divides the continuous speech signal into frames for frequency domain analysis, and uses a recurrent neural network to further extract the timing information contained in the continuous segments. In the feature extraction stage, the present invention extracts the Gaussian mean matrix of the speech signal according to the time sequence segmentation, and the extracted SGMM features contain both the time domain and frequency domain information of the speech signal.

[0118] The present invention compares the difference in recognition effect between the classic machine learning algorithm SVM and the classic deep learning method DNN in recording device source recognition under the same feature input, using MFCC and GSV as feature input.

[0119] The experimental results show that when performing recording device source identification tasks, the GSV features extracted by the Gaussian mixture model have stronger device source feature information representation capabilities than the original MFCC features. However, since the traditional algorithm does not have the ability to perform time series analysis and cannot fully utilize all the information in the SGMM features, it is necessary to design a corresponding network model for identification in order to consider both its spatial complexity and time complexity.

[0120]

[0121] The present invention uses different optimizers to train network models and compares their performance. Different optimizers have their own advantages and disadvantages, and compares the impact of different optimizers on training results.

[0122] Optimizer Model epochs Accuracy Adam The present invention 100 98.74％ RMSprop The present invention 100 98.42％

[0123] In the description of the present invention, unless otherwise specified, "plurality" means two or more than two; the orientations or positional relationships indicated by the terms "upper", "lower", "left", "right", "inner", "outer", "front end", "rear end", "head", "tail", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0124] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0125] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. An end-to-end recording device source identification method based on temporal supervector representation learning, characterized in that: The end-to-end recording device source identification method based on temporal supervector representation learning includes: Extract the Mel-frequency cepstral coefficients in the digital audio as the device source features according to the time series frames, and then extract the Gaussian mean matrix respectively by dividing it into time series feature segments to obtain the Sequential Gaussian Mean Matrix (SGMM) features; The convolutional neural network is used to perform deep representation learning on the time series Gaussian mean matrix features and extract deep bottleneck features. The bidirectional long short-term memory neural network is used to extract the time domain characteristics in the deep time series bottleneck features and identify and classify the recording device source. The end-to-end recording device source identification method based on temporal supervector representation learning comprises the following steps: Step 1: extract the Mel-frequency cepstral coefficient (MFCC) features from the source signal of the recording device, and retain the MFCC features of the same number of frames for each audio data; Step 2: Use MFCC feature training to establish a Gaussian mixture universal background model, namely the GMM-UBM model, and extract the time series Gaussian mean matrix by the GMM-UBM model after segmenting the MFCC features; Step 3: Use the constructed convolutional neural network to perform deep representation learning, extract deep bottleneck features from the time series Gaussian mean matrix, and retain the time series structure of the time series Gaussian mean matrix features; Step 4: Perform time series modeling of the device source identification network, build a classification model based on a bidirectional long short-term memory neural network, and use the classification model to identify and classify the recording device source based on the extracted deep bottleneck features.

2. The end-to-end recording device source identification method based on temporal supervector representation learning as claimed in claim 1, characterized in that: The step of extracting the Mel-frequency cepstral coefficient (MFCC) feature from the source signal of the recording device and retaining the MFCC feature of the same number of frames for each audio data includes: Preprocessing the source signal of the recording device by frame division and windowing; performing fast Fourier transform on the source signal of the recording device, and performing Mel filtering on the obtained amplitude spectrum signal using a set of Mel-scale triangular filter groups; The logarithmic magnitude spectrum of each filter group output is calculated and discrete cosine transform is performed to obtain MFCC features. The MFCC features of the first n frames are retained for each recording data.

3. The end-to-end recording device source identification method based on temporal supervector representation learning as claimed in claim 2, characterized in that: The preprocessing of the source signal of the recording device by framing and windowing includes: taking a frame length of 256 ms, taking 1 / 2 of the frame length as a frame shift, and using a Hamming window function for windowing.

4. The end-to-end recording device source identification method based on temporal supervector representation learning as claimed in claim 1, characterized in that: In step 2, the Gaussian mixture universal background model, i.e., the GMM-UBM model, is established by using MFCC feature training, and the time series Gaussian mean matrix is ​​extracted by the GMM-UBM model after segmenting the MFCC features, including: Using the recording data of various recording devices, a diagonal covariance universal background model (UBM model) with n Gaussian number is trained to obtain the weight coefficient matrix, mean matrix and covariance matrix of n Gaussian mixtures; using the maximum a posteriori estimation algorithm for the feature vector of each device in the training data, the universal background model is adaptively obtained to obtain the GMM model of the specific device source; The extracted MFCC features are divided into t continuous feature segments, and the mean matrix of GMM is extracted by the GMM-UBM model. The n mean matrices of each GMM are sequentially connected into a mean matrix with a total length of n × input MFCC dimension, that is, the time-series Gaussian mean matrix feature.

5. The end-to-end recording device source identification method based on temporal supervector representation learning as claimed in claim 1, characterized in that: In step 3, the deep representation learning is performed using the constructed convolutional neural network, the deep bottleneck features are extracted from the time series Gaussian mean matrix, and the time series structure of the time series Gaussian mean matrix features is retained, including: Construct a convolutional neural network for deep representation learning, use Gaussian mean matrix features to train and test the network model; The deep bottleneck features of the trained convolutional neural network model are extracted and saved in the form of a time series structure of Gaussian mean matrix features.

6. The end-to-end recording device source identification method based on temporal supervector representation learning as claimed in claim 5, characterized in that: The method of using the constructed convolutional neural network to perform deep representation learning, extracting deep bottleneck features from the time series Gaussian mean matrix, and retaining the time series structure of the time series Gaussian mean matrix features also includes: Use labeled training sets to conduct supervised training on deep neural networks and build a representation model based on input features; based on the representation model, select the activation value of the appropriate layer as the deep bottleneck feature; The specific steps are as follows: The obtained one-dimensional feature vector is expanded into a two-dimensional feature map, and the frequency domain feature information is extracted using a convolutional neural network. The obtained SGMM features are input into the constructed convolutional neural network. The input form of each recording device source data sample is 80 feature matrices of 12*16 size arranged in time sequence, where 12 represents the MFCC feature dimension and 16 represents the Gaussian number of the GMM model. The convolution kernel is used to perform a convolution operation on the input features to obtain relevant feature information. After the convolutional neural network model is trained, the training set and test set data are input into the trained convolutional neural network model, and the activation value of the bottleneck layer is used as the output and saved in time series.

7. The end-to-end recording device source identification method based on temporal supervector representation learning as claimed in claim 1, characterized in that: The classification model based on the bidirectional long short-term memory neural network includes two BiLSTM layers, a Flatten processing layer and a Softmax classifier; Each BiLSTM layer contains 128 units; Flatten processing layer, used to convert feature vectors into one-dimensional features; A Softmax classifier is used to make a decision based on the one-dimensional feature.

8. An end-to-end recording device source identification system based on time series supervector representation learning that implements the end-to-end recording device source identification method based on time series supervector representation learning as described in any one of claims 1 to 7, characterized in that: The end-to-end recording device source identification system based on temporal supervector representation learning includes: MFCC feature extraction module, used to extract recording device source feature information of recording device source data during training and recognition; The GMM-UBM model building module is used to build a universal background model for recording device source identification and obtain the GMM model of each target device source through MAP adaptation; SGMM feature extraction module, used to extract the Gaussian mean matrix of the target device recording data in time sequence; Convolutional neural network module, used to train convolutional neural networks and extract deep bottleneck features; The recurrent neural network module is used to train the BiLSTM network with the extracted time series bottleneck features for automatic identification and classification.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the end-to-end recording device source identification method based on temporal supervector representation learning as described in any one of claims 1-7.

Images