Partially-forged speech detection method and system based on speech spectrum characteristics and deep learning

Abstract

The invention discloses a partially-forged voice detection method and system based on speech spectrum features and deep learning, relates to the technical field of voice authenticity detection, and solves the problem that the existing forged voice detection method cannot effectively detect forged voice carrying short and small forged fragments, so that the detection accuracy is low. The method comprises the following steps of: extracting a Mel language spectrogram of voice, and dividing the Mel language spectrogram into a plurality of language spectrum sub-graphs according to a time direction; inputting the plurality of speech score sub-graphs into a pre-trained deep learning forged speech detection model composed of a linear projection layer, a Transformer encoder network and an xLSTM discriminator network, and obtaining real scores representing the corresponding speech score sub-graphs; and real scores of each speech spectrum sub-graph are fused to obtain a speech-level real score, so that a detection result is obtained, deep representation of the speech spectrum can be learned, the difference between true and false speech segments can be effectively captured, and high-accuracy detection of part of forged speech is realized.

Description

Technical Field

[0001] The present application relates to the technical field of speech authenticity detection, and in particular to a method and system for detecting partially forged speech based on spectral features and deep learning. Background Art

[0002] With the widespread adoption of automated speaker verification technology in areas such as smart devices and online banking, its security issues are becoming increasingly prominent. Automated speaker verification systems are vulnerable to various forgery attacks, including text-to-speech synthesis, voice conversion, and replay attacks. These attacks attempt to deceive automated speaker verification systems by simulating or replacing the speaker's voice characteristics.

[0003] Traditional forged speech detection methods primarily target completely forged speech, where the entire utterance is generated through forgery techniques. However, real-world attacks can be more subtle, forging only a portion of an utterance. This new scenario is known as "partial forgery." In this scenario, the attacker embeds a synthesized or converted speech segment into the real utterance, forging a portion of the utterance while leaving the rest authentic. This attack poses new challenges to existing forged speech detection methods, as they typically rely on utterance-level feature aggregation and may not be able to effectively detect short forged segments.

[0004] Therefore, it is particularly important to develop new forged speech detection methods that can effectively detect and locate short, forged speech segments. These methods for detecting partially forged speech need to be able to discriminate at a finer temporal resolution to improve the accuracy of partially forged speech detection. This patent proposes a deep learning detection algorithm and system based on spectral features and deep learning. It can adapt to the detection needs of partially forged speech scenarios and provide more accurate forged speech detection and location capabilities. Summary of the Invention

[0005] The purpose of this application is to overcome the problem that existing forged speech detection methods cannot effectively detect forged speech containing short forged segments, resulting in low detection accuracy, and to provide a partial forged speech detection method and system based on spectral features and deep learning.

[0006] In a first aspect, a method for detecting partially forged speech based on spectral features and deep learning is provided, comprising:

[0007] Get the voice to be detected;

[0008] Extracting a Mel-level speech spectrum of the speech to be detected, and dividing the extracted Mel-level speech spectrum into a plurality of first spectrum subgraphs according to a time direction;

[0009] Inputting the plurality of first spectrogram subgraphs into a trained deep learning forged speech detection model to obtain a true score corresponding to each first spectrogram subgraph, wherein the true score is the proportion of true speech within the time interval covered by each first spectrogram subgraph;

[0010] The true scores of each first spectrogram subgraph are fused to obtain the utterance-level true score.

[0011] In some possible implementations, the structure of the deep learning forged voice detection model includes:

[0012] Linear projection layer, used to map the spectrogram subgraph into a high-dimensional embedding vector;

[0013] Position encoding, which adds the subgraph embedding vector to the position embedding vector to add position information;

[0014] A Transformer encoder network, consisting of multiple stacked Transformer blocks, each layer of which includes a self-attention mechanism and a feedforward neural network. The Transformer encoder network is used to capture long-range dependencies between speech features to extract deep feature representations;

[0015] The xLSTM discriminator network is used to capture the semantic, structural or logical irregularities between various deep features and output the true score corresponding to each spectrogram subgraph.

[0016] In some possible implementations, the training process of the deep learning forged speech detection model includes:

[0017] Obtaining a training data set including sample audio and a segmented speech tag corresponding to each sample audio, wherein the segmented speech tag is used to identify a real speech segment and a forged speech segment in the sample audio;

[0018] Extracting the Mel spectrogram of the sample audio and dividing it into multiple second spectrogram subgraphs in the time direction;

[0019] Generate a mixed label sequence corresponding to each second spectrogram subgraph from the segmented speech labels by a label mixing method;

[0020] Inputting the plurality of second spectrogram subgraphs into a deep learning forged speech detection model to obtain a true score prediction sequence;

[0021] Calculating the binary cross entropy between the true score prediction sequence and the mixed label sequence to obtain a loss value;

[0022] Optimize the parameters of the deep learning forged voice detection model based on the loss value until the loss value converges or reaches the number of iterative training times to obtain a trained deep learning forged voice detection model.

[0023] In some possible implementations, in the process of optimizing the parameters of the deep learning forged voice detection model, the gradients of the various parameters in the deep learning forged voice detection model are obtained based on the loss value, and the parameters of the deep learning forged voice detection model are updated using a gradient descent algorithm to gradually reduce the loss value.

[0024] In some possible implementations, the method for calculating the utterance-level true score includes: performing adaptive threshold-weighted fusion on the true scores of each spectrogram subgraph to obtain the utterance-level true score.

[0025] In some possible implementations, the formula for calculating the utterance-level true score is:

[0026]

[0027] Among them, S utterance is the true score at the discourse level, S i is the true score of the i-th spectrogram subgraph, N is the total number of spectrogram subgraphs, θ is the dynamic threshold, and α is the weight gain coefficient.

[0028] Secondly, a partial forged speech detection system based on spectral features and deep learning is provided, including:

[0029] A voice acquisition module is used to acquire the voice to be detected;

[0030] A feature extraction module is used to extract the Mel language spectrum of the speech to be detected, and divide the extracted Mel language spectrum into multiple first spectrum subgraphs in the time direction;

[0031] A detection module, configured to input the plurality of first spectrogram subgraphs into a trained deep learning forged speech detection model to obtain a true score corresponding to each first spectrogram subgraph, wherein the true score is the proportion of true speech within the time interval covered by each first spectrogram subgraph;

[0032] The output module is used to fuse the true scores of each first spectrogram subgraph to obtain the true score at the utterance level and output it.

[0033] In some possible implementations, a model training module is further included, and the model training module includes:

[0034] A sample acquisition submodule is used to obtain a training data set containing sample audio and a segmented speech tag corresponding to each sample audio, wherein the segmented speech tag is used to identify the real speech segment and the forged speech segment in the sample audio;

[0035] A feature extraction submodule is used to extract the Mel spectrogram of the sample audio and divide it into multiple second spectrogram subgraphs in the time direction;

[0036] A mixed label sequence generation submodule, configured to generate a mixed label sequence corresponding to each second spectrogram subgraph from the segmented speech labels by a label mixing method;

[0037] An input submodule, configured to input the plurality of second spectrogram subgraphs into a deep learning forged speech detection model to obtain a true score prediction sequence;

[0038] A loss value calculation submodule, configured to calculate the binary cross entropy between the true score prediction sequence and the mixed label sequence to obtain a loss value;

[0039] The parameter optimization submodule is used to optimize the parameters of the deep learning forged voice detection model based on the loss value until the loss value converges or reaches the number of iterative training times to obtain a trained deep learning forged voice detection model.

[0040] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable medium stores program code for execution by a device, the program code including steps for executing the method in any one of the implementations of the first aspect.

[0041] In a fourth aspect, an electronic device is provided, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements a method as in any one of the implementations of the first aspect described above.

[0042] The present application has the following beneficial effects: the present application extracts the Mel spectrogram of the speech to be detected, and divides the Mel spectrogram into multiple spectrogram subgraphs in the time direction, uses a pre-trained deep learning forged speech detection model to perform a true score of the authenticity prediction corresponding to each spectrogram subgraph, and obtains the discourse-level true score by fusing the true scores of each spectrogram subgraph, thereby obtaining the detection result of the speech to be detected, can learn the deep representation of the speech spectrogram, and effectively capture the differences between true and false speech segments, thereby achieving high-accuracy detection of some forged speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings that constitute a part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application.

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 Flowchart of the method for partially forged speech detection based on spectral features and deep learning according to Example 1 of the present application;

[0046] Figure 2 This is a simplified flowchart of the method for partially forged speech detection based on spectral features and deep learning in Example 1 of the present application;

[0047] Figure 3 This is a flowchart of spectral feature extraction in the partially forged speech detection method based on spectral features and deep learning in Example 1 of the present application;

[0048] Figure 4 This is a flowchart of the deep learning forged speech detection model training in the partial forged speech detection method based on spectral features and deep learning in Example 1 of the present application;

[0049] Figure 5 This is a structural block diagram of the Transformer encoder in the partial forged speech detection method based on spectral features and deep learning in Example 1 of the present application;

[0050] Figure 6 This is a structural block diagram of the xLSTM discriminator in the partial forged speech detection method based on spectral features and deep learning in Example 1 of the present application;

[0051] Figure 7 This is a structural block diagram of a partially forged speech detection system based on spectral features and deep learning according to Example 2 of the present application;

[0052] Figure 8 This is a schematic diagram of the internal structure of the electronic device of Example 4 of the present application.

[0053] Reference numerals:

[0054] 100, speech acquisition module; 200, feature extraction module; 300, detection module; 400, output module; 500, model training module; 501, sample acquisition submodule; 502, feature extraction submodule; 503, mixed label sequence generation submodule; 504, input submodule; 505, loss value calculation submodule; 506, parameter optimization submodule. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] Example 1

[0057] like Figure 1 and Figure 2 As shown, the method for detecting partially forged speech based on spectral features and deep learning in Example 1 of the present application includes:

[0058] S100, obtaining the voice to be detected;

[0059] Specifically, the speech to be detected can be input from the user's microphone or pre-recorded and synthesized by a criminal using a recording device. The speech to be detected can be stored in computer memory using various lossless audio encoding methods and loaded by a program.

[0060] S200, extracting the Mel-level speech spectrum of the speech to be detected, and dividing the extracted Mel-level speech spectrum into a plurality of first spectrum sub-graphs according to the time direction;

[0061] Specifically, the acquired speech to be detected is input into a feature extraction module, where a mel-spectrogram is extracted through pre-emphasis, framing and windowing, fast Fourier transform of each frame, application of a mel filter bank, logarithmization, and mean normalization. The extracted mel-spectrogram is then further processed to divide the spectrogram into multiple first spectrogram subgraphs along the time axis, forming a spectrogram subgraph sequence.

[0062] like Figure 3 As shown, the extraction of the first spectrum subgraph includes:

[0063] Step 201: pre-emphasis processing;

[0064] Step 202: framing and windowing;

[0065] Step 203: Perform fast Fourier transform on each frame of speech;

[0066] Step 204: Apply a Mel filter bank.

[0067] Step 205: Take the logarithm.

[0068] Step 206: Normalize the spectrogram mean.

[0069] When step 201 is specifically implemented, the pre-emphasis processing is implemented by passing the voice signal through a first-order high-pass filter. Its mathematical expression is:

[0070]

[0071] in, Represents the signal after pre-emphasis, represents the original signal, is the pre-emphasis coefficient, which is 0.95.

[0072] Specifically, when implementing step 202, the pre-emphasized speech signal is framed and windowed. First, the speech signal is framed according to the set frame length and frame shift, and then the selected window function is applied to each frame signal. The purpose of the framing is to limit the analysis and processing of the speech signal to a shorter time period in order to better capture the local characteristics of the speech signal. The frame length is set to 25ms and the frame shift is set to 10ms. The purpose of the windowing is to reduce the discontinuity of the transition between frames. The window function is selected as a Hamming window, and its mathematical expression is:

[0073]

[0074] Wherein, N represents the frame length taken above, and n ranges from 0 to N-1. Take 0.46.

[0075] Specifically, when implementing step 203, the purpose of performing a fast Fourier transform on each frame of speech is to convert the time domain signal into the frequency domain to form a spectrum. Fast Fourier transform is an efficient algorithm for calculating discrete Fourier transform. The formula of discrete Fourier transform is:

[0076]

[0077] in, represents the frequency domain signal after transformation, k is the frequency index; is a time domain signal; N represents the Fourier transform length, which is twice the length of the frame.

[0078] Specifically, in step 204, the Mel filter bank is applied to the spectrum to simulate the human ear's perception of different frequencies. The Mel filter bank consists of multiple bandpass filters, each with a center frequency distributed according to the Mel scale. The Mel scale is a nonlinear frequency scale that more closely resembles the human ear's perception of sound. Its relationship with frequency f can be expressed as follows:

[0079]

[0080] In this embodiment, the number of filters in the Mel filter bank can be adjusted as needed, and is typically between 20 and 40. The output of each filter is the sum of the energy within the frequency range.

[0081] Specifically, when implementing step 205, the purpose of taking the logarithm is to convert the energy value output by the Mel filter bank into the logarithmic domain. This step can enhance the dynamic range of the spectrum, making the small-amplitude frequency components more obvious in the logarithmic domain, as shown in the following formula:

[0082]

[0083] in, is the energy value output by the Mel filter bank; in order to avoid the logarithm being negative infinity, a small .

[0084] Specifically, when implementing step 206, the spectrogram mean is normalized to make the features of different speech signals comparable. This step is achieved by subtracting the mean of each frequency channel and dividing by the standard deviation, as shown in the following formula:

[0085]

[0086] in, is the mean of the kth frequency channel; is the standard deviation of the kth frequency channel.

[0087] S300, inputting the plurality of first spectrogram subgraphs into a trained deep learning forged speech detection model to obtain a true score corresponding to each first spectrogram subgraph, wherein the true score is the proportion of the real speech in the time interval covered by each first spectrogram subgraph;

[0088] Specifically, the structure of the deep learning forged speech detection model includes: linear projection layer, positional encoding, Transformer encoder network and xLSTM discriminator network. The following is an introduction to the role of each structure:

[0089] The linear projection layer is used to map the spectrogram subgraph into a high-dimensional embedding vector. The linear projection layer is implemented by a linear network to map the spectrogram subgraph into a high-dimensional embedding vector. The linear projection layer usually consists of a weight matrix W and a bias vector b. Its calculation formula is:

[0090]

[0091] in, is the i-th spectrogram subgraph of the input.

[0092] Position encoding is used to add the subgraph embedding vector to the position embedding vector to add position information. Position encoding is used to provide the model with information about the position of each element in the input sequence. In this embodiment, sine and cosine functions are used to generate position encoding, and the formula is:

[0093]

[0094] in, Indicates the The position encoding vector component values; pos is the position index, i is the dimension index; dim is the vector dimension of the model.

[0095] The Transformer encoder network includes multiple stacked Transformer blocks, each layer of which contains a self-attention mechanism and a feedforward neural network. The Transformer encoder network is used to capture long-distance dependencies between speech features to extract deep feature representations.

[0096] like Figure 5 As shown in Figure 1, the Transformer encoder network consists of multiple identical stacked Transformer blocks. Each block contains a multi-head self-attention mechanism and a feedforward neural network. Residual connections and layer normalization are used to facilitate the training of deep networks. The self-attention mechanism allows the model to dynamically distribute attention weights between different input positions, thereby capturing long-range dependencies in the input sequence. The self-attention calculation formula is as follows:

[0097]

[0098] Among them, Q, K and V represent the query, key and value matrices respectively. Is the size of K, which is used to scale the dot product to prevent the gradient from disappearing. The feedforward neural network is a simple two-layer neural network that performs a nonlinear transformation on the output of the self-attention mechanism. The calculation formula is:

[0099]

[0100] in, 、 is the weight matrix, b1 and b2 are bias vectors.

[0101] The xLSTM discriminator network is used to capture the semantic, structural or logical irregularities between various deep features and output the true score corresponding to each spectrogram subgraph.

[0102] like Figure 6As shown in the figure, the xLSTM discriminator network consists of multiple identical stacked mLSTM blocks. In each block, the input is first layer-normalized, and then up-projected using two up-projection layers with a projection factor of 2. The output is used once as the external output gate and once as the input of the mLSTM unit. The output of the mLSTM unit is multiplied point-by-point with the output of the external output gate to achieve component gating. Finally, after passing through the down-projection layer with a projection factor of 2, it is added to the residual connection of the input to form the output of the mLSTM block. The calculation formula of the mLSTM unit is as follows:

[0103] ;

[0104] ;

[0105] ;

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] ;

[0112] in, represents the input vector at time t, is the matrix storage unit, and denote the normalizer state and hidden state respectively, 、 and Represents input, forget and output gates respectively. Weight 、 and Learnable projection matrices for vector queries, keys, and values, respectively.

[0113] like Figure 4 As shown in Figure 2, the training process of the deep learning forged speech detection model includes:

[0114] Step 301: Obtain a training dataset containing sample audio and the segmented speech labels corresponding to each audio;

[0115] Steps 302 and 304: extracting the mel spectrogram 303 of the speech and dividing it into multiple spectrogram subgraphs 305 in the time direction;

[0116] Step 313: Generate a mixed label sequence 314 corresponding to each spectrogram subgraph from the segmented speech labels by a label mixing method;

[0117] Steps 306-312: Input the spectrogram subgraph into the deep learning forged speech detection model to obtain a true score prediction sequence 312;

[0118] Step 315: Calculate the loss value 313 between the true score prediction sequence and the mixed label sequence;

[0119] Step 316: Optimize the parameters 314 of the deep learning forged speech detection model based on the loss value.

[0120] Specifically, in step 301, an audio training set for training a deep learning forged speech detection model is obtained. The training set includes multiple audio samples and pre-labeled segmented speech tags. The segmented speech tags identify the portion of an audio sample that is authentic speech, with a label of 1; the portion that is forged speech, with a label of 0.

[0121] Specifically, during steps 302 and 304, the sample audio from the training set is input into a feature extraction module to obtain a mel-spectrogram 303. The mel-spectrogram is then divided into multiple subgraphs 305 along the time axis at a given resolution, thereby obtaining a spectrogram subgraph sequence. In the present invention, a 25 ms resolution is preferably used for the spectrogram subgraph division.

[0122] When implementing step 313, the proportion of real speech in the time interval covered by each spectrogram is calculated based on the time boundary of the segmented speech label and the time division granularity of the spectrogram subgraph. Specifically: for each spectrogram subgraph i, determine its corresponding time interval ; Traverse the segmented speech labels and count the total duration of the speech that is marked as real speech (labeled as 1) within the time interval ; Calculate the proportion of real speech in the subgraph time interval to obtain the mixed label, the calculation formula is:

[0123]

[0124] Repeat the above steps for all segmented spectrogram subgraphs to generate the corresponding mixed label sequence , where N is the number of spectrogram subgraphs corresponding to a single speech sample. This method quantifies the degree of local forgery through soft labels, alleviating the ambiguity of traditional hard labels when subgraphs cross the real / forgery boundary, enabling the model to learn more refined local forgery features.

[0125] When implementing steps 306-312, the spectrogram subgraph 305 is input into the linear projection layer 306 and mapped into a high-dimensional embedding vector 307, and the CLS flag vector is concatenated in front; the high-dimensional embedding vector is added to the position encoding vector 308 to obtain an embedding feature with position information; the embedding feature is input into the Transformer encoder network 309 to obtain a deep feature representation 310; the deep feature representation is input into the xLSTM discriminator network 311 to obtain a true score prediction sequence 312 corresponding to each spectrogram subgraph.

[0126] Specifically, when implementing step 315, the binary cross entropy loss between the true score prediction sequence of each spectrogram subgraph and the mixed label sequence of each spectrogram subgraph is calculated to quantify the error of the model in predicting each spectrogram subgraph, thereby guiding the optimization and adjustment of the model. The binary cross entropy loss function is used to measure the difference between the probability distribution predicted by the model and the true label, and its calculation formula is:

[0127]

[0128] Where B is the number of speech samples in a mini-batch; N is the number of spectrogram subgraphs corresponding to each speech; is the mixed label of the nth spectrogram subgraph of the bth speech sample, is the true score prediction value of the nth spectrogram subgraph of the bth speech sample output by the xLSTM discriminator network.

[0129] Specifically, when implementing step 316, the gradient of each parameter in the deep learning forged speech detection model is calculated based on the binary cross entropy loss value, and the model parameters are updated using a gradient descent algorithm to gradually reduce the loss value. The present invention preferably uses the AdamW optimizer to optimize the model.

[0130] The spectrogram subgraph sequence obtained in step S200 is input into a deep learning forged speech detection model. A linear projection layer maps the spectrogram subgraphs to high-dimensional embedding vectors. Positional encoding is added to add the subgraph embedding vectors to the positional embedding vectors to increase position information. A Transformer encoder network extracts deep feature representations. The xLSTM discriminator network outputs the true score for each spectrogram subgraph.

[0131] S400: Fusing the true scores of the first spectrogram subgraphs to obtain an utterance-level true score.

[0132] Specifically, the true score of each spectrogram subgraph obtained in step S300 is input into the score fusion unit to obtain the utterance-level true score. The utterance-level score represents the confidence of the detection method in the authenticity of the detected speech, thereby obtaining the detection result.

[0133] The score fusion unit performs adaptive threshold weighted fusion on the true score prediction sequence of the spectrogram subgraph to obtain the true score at the utterance level. The calculation formula is:

[0134]

[0135] Where N is the number of spectrogram subgraphs; is the dynamic threshold, Take the scores of all subgraphs -quantile, is the weight gain coefficient; and Can be adjusted to specific applications, Determines the number of sub-graphs involved in fusion and affects the coverage of detection, while Determines the weight distribution within the low-molecular-weight graph and affects the ability to distinguish the degree of forgery.

[0136] In this embodiment, in order to solve the problem that the existing forged speech detection method cannot effectively detect forged speech with short forged fragments, the Mel-level speech spectrum is extracted from the speech to be detected and divided into pairs of spectrograms in the time direction. Then, a pre-trained deep learning forged speech detection model is used to identify the authenticity of each spectrogram subgraph and output the true score corresponding to each spectrogram subgraph. Finally, the true scores of each spectrogram subgraph are fused to obtain the discourse-level true score, thereby obtaining the detection result. The deep representation of the speech spectrogram can be learned, and the differences between true and false speech segments can be effectively captured. By dividing the Mel-level speech spectrum into multiple spectrograms and then using the model for prediction, the short forged fragments carried in the forged speech can be detected, thereby achieving high-accuracy detection of some forged speech.

[0137] Example 2

[0138] like Figure 7 As shown, the partially forged speech detection system based on spectral features and deep learning involved in Example 2 of the present application includes:

[0139] The voice acquisition module 100 is used to acquire the voice to be detected;

[0140] The feature extraction module 200 is used to extract the Mel-level speech spectrum of the speech to be detected, and divide the extracted Mel-level speech spectrum into a plurality of first spectrum subgraphs according to the time direction;

[0141] A detection module 300 is configured to input the plurality of first spectrogram subgraphs into a trained deep learning forged speech detection model to obtain a true score corresponding to each first spectrogram subgraph, wherein the true score is the proportion of true speech within the time interval covered by each first spectrogram subgraph;

[0142] The output module 400 is configured to fuse the true scores of the first spectrogram subgraphs to obtain and output the utterance-level true score.

[0143] In an optional embodiment, a model training module 500 is further included, and the model training module 500 includes:

[0144] The sample acquisition submodule 501 is used to obtain a training data set including sample audio and a segmented speech tag corresponding to each sample audio, wherein the segmented speech tag is used to identify the real speech segment and the forged speech segment in the sample audio;

[0145] The feature extraction submodule 502 is used to extract the Mel-spectrogram of the sample audio and divide it into multiple second spectrogram subgraphs in the time direction;

[0146] A mixed label sequence generating submodule 503 is configured to generate a mixed label sequence corresponding to each second spectrogram subgraph from the segmented speech labels by using a label mixing method;

[0147] An input submodule 504 is configured to input the plurality of second spectrogram subgraphs into a deep learning forged speech detection model to obtain a true score prediction sequence;

[0148] A loss value calculation submodule 505 is used to calculate the binary cross entropy between the true score prediction sequence and the mixed label sequence to obtain a loss value;

[0149] The parameter optimization submodule 506 is used to optimize the parameters of the deep learning forged speech detection model based on the loss value until the loss value converges or reaches the number of iterative training times, so as to obtain a trained deep learning forged speech detection model.

[0150] It should be noted that other specific implementation methods of the partial forged speech detection system based on spectral features and deep learning in this embodiment can be found in the specific implementation methods of the partial forged speech detection method based on spectral features and deep learning mentioned above. To avoid redundancy, they will not be repeated here.

[0151] Example 3

[0152] A computer-readable storage medium according to embodiment 3 of the present application, wherein the computer-readable storage medium stores program code for execution by a device, the program code including steps for executing the method in any one of the implementations in embodiment 1 of the present application;

[0153] Among them, the computer-readable storage medium can be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM); the computer-readable storage medium can store program code, and when the program stored in the computer-readable storage medium is executed by the processor, the processor is used to execute the steps of the method in any one of the implementation methods in Example 1 of the present application.

[0154] Example 4

[0155] like Figure 8 As shown, an electronic device involved in Example 4 of the present application includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the method in any one of the implementations in Example 1 of the present application;

[0156] Among them, the processor can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the method in any one of the implementation methods in Example 1 of the present application.

[0157] The processor may also be an integrated circuit electronic device with signal processing capabilities. In the implementation process, each step of the method in any one of the implementations in Example 1 of the present application may be completed by hardware integrated logic circuits in the processor or software instructions.

[0158] The above-mentioned processor can also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in combination with its hardware, completes the functions required to be executed by the units included in the data processing device of the embodiment of the present application, or executes the method in any one of the implementation modes in Example 1 of the present application.

[0159] The above are only preferred specific implementations of this application; however, the scope of protection of this application is not limited thereto. Any person skilled in the art who, within the technical scope disclosed in this application, makes equivalent substitutions or modifications based on the technical solutions and improved concepts of this application shall be covered by the scope of protection of this application.

Claims

1. A method for detecting partially forged speech based on spectral features and deep learning, characterized in that: include: Get the voice to be detected; Extracting a Mel-level speech spectrum of the speech to be detected, and dividing the extracted Mel-level speech spectrum into a plurality of first spectrum subgraphs according to a time direction; Inputting the plurality of first spectrogram subgraphs into a trained deep learning forged speech detection model to obtain a true score corresponding to each first spectrogram subgraph, wherein the true score is the proportion of true speech within the time interval covered by each first spectrogram subgraph; The true scores of each first spectrogram subgraph are fused to obtain the utterance-level true score.

2. The method for partially forged speech detection based on spectral features and deep learning according to claim 1, characterized in that: The structure of the deep learning forged voice detection model includes: Linear projection layer, used to map the spectrogram subgraph into a high-dimensional embedding vector; Position encoding, which adds the subgraph embedding vector to the position embedding vector to add position information; A Transformer encoder network, consisting of multiple stacked Transformer blocks, each layer of which includes a self-attention mechanism and a feedforward neural network. The Transformer encoder network is used to capture long-range dependencies between speech features to extract deep feature representations; The xLSTM discriminator network is used to capture the semantic, structural or logical irregularities between various deep features and output the true score corresponding to each spectrogram subgraph.

3. The method for partially forged speech detection based on spectral features and deep learning according to claim 1 or 2, characterized in that: The training process of the deep learning forged speech detection model includes: Obtaining a training data set including sample audio and a segmented speech tag corresponding to each sample audio, wherein the segmented speech tag is used to identify a real speech segment and a forged speech segment in the sample audio; Extracting the Mel spectrogram of the sample audio and dividing it into multiple second spectrogram subgraphs in the time direction; Generate a mixed label sequence corresponding to each second spectrogram subgraph from the segmented speech labels by a label mixing method; Inputting the plurality of second spectrogram subgraphs into a deep learning forged speech detection model to obtain a true score prediction sequence; Calculating the binary cross entropy between the true score prediction sequence and the mixed label sequence to obtain a loss value; Optimize the parameters of the deep learning forged voice detection model based on the loss value until the loss value converges or reaches the number of iterative training times to obtain a trained deep learning forged voice detection model.

4. The method for partially forged speech detection based on spectral features and deep learning according to claim 3, characterized in that: In the process of optimizing the parameters of the deep learning forged voice detection model, the gradients of the various parameters in the deep learning forged voice detection model are obtained based on the loss value, and the parameters of the deep learning forged voice detection model are updated using a gradient descent algorithm to gradually reduce the loss value.

5. The method for partially forged speech detection based on spectral features and deep learning according to claim 1, characterized in that: The method for calculating the true score at the utterance level includes: performing adaptive threshold weighted fusion on the true scores of each spectrogram subgraph to obtain the true score at the utterance level.

6. The method for partially forged speech detection based on spectral features and deep learning according to claim 5, characterized in that: The formula for calculating the true score at the discourse level is: ； Among them, S utterance is the true score at the discourse level, S i is the true score of the i-th spectrogram subgraph, N is the total number of spectrogram subgraphs, θ is the dynamic threshold, and α is the weight gain coefficient.

7. A partially forged speech detection system based on spectral features and deep learning, characterized in that: include: A voice acquisition module is used to acquire the voice to be detected; A feature extraction module is used to extract the Mel language spectrum of the speech to be detected, and divide the extracted Mel language spectrum into multiple first spectrum subgraphs in the time direction; A detection module, configured to input the plurality of first spectrogram subgraphs into a trained deep learning forged speech detection model to obtain a true score corresponding to each first spectrogram subgraph, wherein the true score is the proportion of true speech within the time interval covered by each first spectrogram subgraph; The output module is used to fuse the true scores of each first spectrogram subgraph to obtain the true score at the utterance level and output it.

8. The partially forged speech detection system based on spectral features and deep learning according to claim 7 is characterized in that: It also includes a model training module, which includes: A sample acquisition submodule is used to obtain a training data set containing sample audio and a segmented speech tag corresponding to each sample audio, wherein the segmented speech tag is used to identify the real speech segment and the forged speech segment in the sample audio; A feature extraction submodule is used to extract the Mel spectrogram of the sample audio and divide it into multiple second spectrogram subgraphs in the time direction; A mixed label sequence generation submodule, configured to generate a mixed label sequence corresponding to each second spectrogram subgraph from the segmented speech labels by a label mixing method; An input submodule, configured to input the plurality of second spectrogram subgraphs into a deep learning forged speech detection model to obtain a true score prediction sequence; A loss value calculation submodule, configured to calculate the binary cross entropy between the true score prediction sequence and the mixed label sequence to obtain a loss value; The parameter optimization submodule is used to optimize the parameters of the deep learning forged voice detection model based on the loss value until the loss value converges or reaches the number of iterative training times to obtain a trained deep learning forged voice detection model.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program codes for execution by a device, wherein the program codes include steps for executing the method according to any one of claims 1 to 6.

10. An electronic device, characterized in that: The electronic device includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the method according to any one of claims 1 to 6.

Images