A method for confirming a voice device source

Abstract

The present invention discloses a method for confirming the source of a speech device, comprising the following steps: S1: windowing and framing the speech signal in each speech recording in the training set corpus to obtain a two-dimensional array as input; S2: for all speech recordings in the training set corpus, convert them into a two-dimensional array according to step S1, and form a training set together with the corresponding recording device annotations, while constructing the training set, the batch size used when training the neural network is also set, and then sent to a deep residual neural network containing an attention mechanism for model training; S3: using the trained model to confirm the speech to be tested. The present invention can accurately extract the recording device feature representation from the speech recording, so that the matching accuracy is high in the open set device source confirmation problem.

Description

Technical Field

[0001] The present invention relates to digital voice blind forensics in the field of digital audio forensics, and in particular to a voice device source confirmation method, in particular to a digital voice device source confirmation method based on a deep residual network with an attention mechanism. Background Art

[0002] Digital voice, as an important type of audio data, is closely related to people's lives. Moreover, digital voice recordings often appear in courts as judicial evidence and in case detection. Since audio recording files can be easily edited and tampered with using audio editing software such as Audition, before a voice recording can be accepted as evidence in court or relevant judicial institutions, its authenticity must be authenticated from the perspective of multimedia content security.

[0003] Digital voice device source forensics is a type of digital voice blind forensics technology. This type of technology attempts to directly extract the recording device fingerprint from the voice recording, and then authenticates the authenticity of the digital voice based on the consistency and coherence of the acquired recording device fingerprint. It can be applied to the identification and tracing of forged multimedia information.

[0004] At present, deep learning technology has been applied in many fields and improved its previous performance. It has also been applied to the field of digital voice device source forensics. However, the existing deep learning-based device source forensics technologies are mostly based on conventional convolutional neural network (CNN) technology or conventional deep residual network (ResNet) technology, without considering new deep learning technologies such as attention mechanism. In addition, the existing technologies generally target a closed set device source identification problem, that is, given a voice recording and N known closed set recording devices, determine which of the N devices the voice recording comes from. In real life, the more common problem is the open set device source confirmation problem, that is, given two voice recordings, determine whether the two voice recordings come from the same voice device; or given a voice recording and a recording device, determine whether the voice recording comes from the recording device. At present, there is almost no deep learning-based technology specifically for solving the open set device source confirmation problem. Summary of the invention

[0005] In view of the defects of the prior art, the present invention provides a method for confirming the source of a voice device.

[0006] A method for confirming a voice device source comprises the following steps:

[0007] S1: The speech signal in each speech recording in the training set corpus is windowed and framed, and then converted to the frequency domain by fast Fourier transform (FFT). Then, the N-dimensional frequency domain feature parameters are extracted. M frames are extracted for each speech recording to obtain a A two-dimensional array takes it as input;

[0008] S2: All speech recordings in the training set corpus are converted into a two-dimensional array according to step S1, and together with their corresponding recording device annotations, they form a training set. While constructing the training set, the batch size used for neural network training is also set, and then sent to the deep residual neural network with attention mechanism for model training;

[0009] S3: Use the trained model to perform confirmation detection on the speech to be tested.

[0010] Optionally, the step S3 comprises: converting the two voice recordings to be tested according to step S1 and then sending them to the trained deep neural network to obtain the recording device feature representation (recorderembedding, re) output by the neural network respectively, and then calculating the distance scale of the two feature representations, and comparing the distance scale with the threshold value. Compare the two audio clips to determine whether they come from the same recording device.

[0011] Alternatively, the voice recording to be tested is converted according to step S1 and then sent to the trained deep neural network to obtain the recording device feature representation output by the neural network of the recording to be tested, and the device feature representation of the known recording of the recording device is also extracted, and then the arithmetic mean is taken as the feature representation of the recording device, the distance between the two feature representations is calculated, and the distance is compared with the threshold value. Compare to determine whether the voice comes from the recording device.

[0012] Furthermore, the structure of the deep residual neural network with attention mechanism described in step S2 is first a two-dimensional convolutional layer, followed by multiple residual network modules with attention mechanism, followed by an encoding layer (Encoding) to compress the frame layer feature map output by the previous network into a feature vector of a separate sentence layer, and finally a fully connected layer to convert it into a device feature representation of a specified dimension.

[0013] The present invention provides a digital voice device source confirmation method based on a deep residual neural network with an attention mechanism for the open-set recording device source confirmation problem. The method is based on a deep residual network with an attention mechanism, and can accurately extract recording device feature representations from voice recordings, thereby achieving high matching accuracy in the open-set device source confirmation problem.

[0014] Different from other related technologies for closed-set device source identification, the present invention mainly focuses on the problem of open-set digital voice device source confirmation, and provides a digital voice device source confirmation method based on a deep residual network with an attention mechanism. The first step is to extract a two-dimensional feature parameter array for each voice recording in units of sentences; the second step is to send the extracted data to a deep residual network with an attention mechanism; the third step is to input the frame-level feature map output by the deep neural network, and then encode the input data through the encoding layer to output a sentence-level feature vector; the fourth step is to obtain the device feature representation of the voice recording through a fully connected layer through the output vector of the encoding layer; the fifth step is to calculate a distance scale based on the extracted device feature representation to determine whether the two voice recordings are from the same recording device or whether a certain voice recording is recorded by a certain recording device. The method of the present invention utilizes advanced deep learning technology, can extract accurate recording device fingerprints, and has high accuracy in device source confirmation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of the device source confirmation method based on the deep residual network with attention mechanism of the present invention.

[0016] Figure 2 Schematic diagram of the attention module used in the embodiment provided by the present invention. DETAILED DESCRIPTION

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, so that the above-mentioned and other objects, features and advantages of the present invention will be clearer. The same reference numerals in all the drawings indicate the same parts. The drawings are not deliberately drawn to scale, and the focus is on illustrating the subject matter of the present invention.

[0018] Figure 1 The flowchart of the method provided by the present invention. The training process of this embodiment requires a device recording library. This embodiment selects a recording library containing 14 recording devices (specifically mobile phones) as the training sound library. Each mobile phone in the sound library contains 240 sentences of recording, and each sentence is about 3 seconds long. Half of the corpus (that is, 120 sentences of recording) of each device is taken as the training set corpus.

[0019] The implementation steps of this embodiment are as follows:

[0020] S1: For each speech recording in the training set corpus, the speech signal is windowed and framed. The window function selects the Hamming window, the frame length is 25ms, and the frame shift is 10ms. Each frame signal is converted to the frequency domain by FFT, and then the 40-dimensional Mel filterbank coefficients are extracted. 200 frames are extracted for each speech recording, so that each speech recording is converted into a 40×200 two-dimensional array as input.

[0021] S2: For all the voice recordings in the training corpus, they are converted into a two-dimensional array according to step S1, and together with the corresponding recording device source annotation, they form a training set. The size of the training batch set in the training set is the number of types of recording devices in the training set (14 in this embodiment). Then the training set is sent to a deep residual neural network with an attention mechanism for model training. The neural network training adopts the Adam algorithm, and the loss function can adopt the softmax function.

[0022] The structure and configuration of the deep residual network with attention mechanism are shown in Table 2. The size of the convolution kernel of the first convolutional layer is , the output feature map is 16, followed by 4 groups of residual modules with attention mechanism. The convolution kernel size used by all residual modules here is , the size of the output feature map of each group of residual modules is 16, 32, 64, and 128 respectively. There are 3 residual modules with an output feature map size of 16, 4 residual modules with an output feature map size of 32, 6 residual modules with an output feature map size of 64, and 3 residual modules with an output feature map size of 128. The encoding layer uses the SAP encoding method, and then outputs the final feature representation through a fully connected layer. Here, the output dimension of the fully connected layer is set to 512. In addition, in this network structure, a batch normalization operation is performed by default after each 2D convolution operation, and the activation function used in the network is the Relu function.

[0023] Table 2 The structure and configuration of the deep residual network with attention mechanism in this example

[0024]

[0025] The attention mechanism module used in the residual module adopts , the process diagram is as follows Figure 2 As shown. The specific implementation is to first perform global average pooling on the input feature map to convert the feature map into 1×1×the number of channels:

[0026] (1)

[0027] Then comes the fully connected layer and sigmoid activation function, which adjusts the feature map of 1×1×number of channels to become the weight of each feature map:

[0028] (2)

[0029] here represents the Relu function, Represents the sigmoid function. Then the obtained weight is multiplied by the original input feature map to obtain a new feature map:

[0030] (3)

[0031] The specific implementation of the SAP method used in the encoding layer is: Send it to a fully connected layer to get a corresponding hidden representation sequence , the fully connected layer uses the tanh activation function, which is shown in the following formula:

[0032] (4)

[0033] Then introduce a learnable parameter , specifically, the importance of a frame is measured through the following softmax function to obtain the importance weight:

[0034] (5)

[0035] Thus, the feature representation e of the entire sentence can be expressed as a weighted sum of all features at the frame level:

[0036] (6)

[0037] For each speech recording in the training corpus, it is converted into a two-dimensional data according to step S1, and together with its corresponding recording device source annotation, it constitutes a training set, and then sent to the deep residual neural network with attention mechanism shown in Table 1 for model training. The Adam algorithm is used for neural network training, and the loss function is softmax loss. The learning rate is set to 0.001, and the learning rate is decayed by 0.05 every 10 iterations. The total number of iterations of model training is set to 200. The softmax loss function based on the recording device feature representation output by the neural network is defined as follows:

[0038] (7)

[0039] Here, C represents the type of recording devices in the training set (14 in this example), and N represents the number of recorded sentences contained in each batch (it is assumed that each recorded sentence in the batch comes from a different recording device, so in this example, ). In this case, the device feature obtained by converting a certain speech recording sentence in a batch of the training set through the neural network is represented as , and its corresponding category is marked as .

[0040] S3: Device source confirmation scenario 1: Given two voice recordings T1 and T2, determine whether they come from the same recording device. The two voice recordings T1 and T2 to be tested are converted according to step S1 and sent to the trained deep neural network to obtain the device feature representations re1 and re2 of the corresponding deep residual neural network outputs, and then the squared Euclidean distance between re1 and re2 is calculated and compared with the threshold value. If the comparison is less than the threshold value, it is determined whether the two voice recordings are from the same recording device source, otherwise it is determined that they are not from the same recording device source.

[0041] Device source confirmation scenario 2: Given a test recording T1 and a recording device D1 (and m sentences recorded by the recording device), determine whether the test recording comes from the recording device. After the N sentences of the D1 device are converted according to step S1, they are sent to the trained deep neural network to obtain m feature representations: Then take the arithmetic mean of these m feature representations as the feature representation of the recording device:

[0042] (8)

[0043] Similarly, the recording T1 to be tested is converted according to step S1 and then sent to the trained deep neural network to obtain the feature representation of its corresponding deep residual neural network output Then calculate and The squared Euclidean distance and the threshold By comparison, if it is less than the threshold value, it is determined that the test speech does come from the recording device, otherwise it is determined that it does not come from the recording device.

[0044] Many specific details are described in the above description to facilitate a full understanding of the present invention. However, the above description is only a preferred embodiment of the present invention. The present invention can be implemented in many other ways different from those described herein, so the present invention is not limited to the specific implementation disclosed above. At the same time, any person familiar with the art can make many possible changes and modifications to the technical solution of the present invention using the methods and technical contents disclosed above without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A method for confirming a voice device source, characterized in that: The following steps are involved: S1: The speech signal in each speech recording in the training set corpus is windowed and framed, and then converted to the frequency domain by fast Fourier transform (FFT), and then N-dimensional frequency domain feature parameters are extracted. M frames are extracted for each speech recording to obtain an N×M two-dimensional array as input; S2: All speech recordings in the training set corpus are converted into a two-dimensional array according to step S1, and together with their corresponding recording device annotations, they form a training set. While constructing the training set, the batch size used for neural network training is also set, and then sent to the deep residual neural network with attention mechanism for model training; S3: Use the trained model to perform confirmation detection on the speech to be tested; The neural network training adopts the Adam algorithm, and the loss function adopts the softmax function; The step S2 includes: the attention mechanism module adopted by the residual module is SE block, firstly, the input feature map is globally averaged and pooled, and the feature map is converted into 1×1×the number of channels: Then comes the fully connected layer and sigmoid activation function, which adjusts the feature map of 1×1×number of channels to become the weight of each feature map: s=F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z)) (2) Here δ represents the Relu function, σ represents the sigmoid function; then the obtained weight is multiplied by the feature map of the original input to obtain a new feature map: The step S2 comprises: The specific implementation of the SAP method used in the coding layer is: the feature map sequence {x1, x2, ..., x L } is sent to a fully connected layer to obtain a corresponding hidden representation sequence {h1,h2,…,h L }, the fully connected layer uses the tanh activation function, which is shown in the following formula: h t =tanh(Wx t +b) (4) Then a learnable parameter μ is introduced to measure the importance of a frame through the following softmax function to obtain the importance weight: Thus, the feature representation e of the entire sentence can be expressed as a weighted sum of all features at the frame level:

2. The confirmation method according to claim 1, characterized in that: The step S3 comprises: converting the two voice recordings to be tested according to the step S1 and sending them into the trained deep neural network respectively, obtaining the recording device feature representations output by the neural network respectively, and then calculating the distance scale of the two feature representations, and comparing the distance scale with the threshold value θ to determine whether the two voice recordings come from the same recording device source.

3. The confirmation method according to claim 1, characterized in that: The step S3 comprises: converting the voice recording to be tested according to step S1 and then sending it into a trained deep neural network to obtain the recording device feature representation output by the neural network of the voice recording to be tested, and also extracting the device feature representation of the known recording of the recording device, and then taking the arithmetic mean as the feature representation of the recording device, calculating the distance between the two feature representations, and comparing the distance with the threshold value θ to determine whether the speech comes from the recording device.

4. The confirmation method according to claim 1, characterized in that: The structure of the deep residual neural network with attention mechanism described in step S2 is first a two-dimensional convolutional layer, followed by multiple residual network modules with attention mechanism, then an encoding layer to compress the frame layer feature map output by the previous network into a feature vector of a separate sentence layer, and finally a fully connected layer to convert it into a device feature representation of a specified dimension.

5. The confirmation method according to claim 4, characterized in that: The structure of the deep residual neural network with attention mechanism described in step S2 is as follows:

6. The confirmation method according to claim 1, characterized in that: The softmax loss function is defined as follows: Among them, C represents the type of recording equipment in the training set, and N represents the number of recorded sentences contained in each batch.

7. The confirmation method according to claim 6, characterized in that: In the network structure, a group normalization operation is performed after each 2D convolution operation, and the activation function used in the network is the Relu function.

Images