Method for positioning and detecting forged voice sample

Abstract

The invention provides a positioning and detecting method for a forged voice sample. The method comprises the following steps: obtaining; inputting the advanced representation matrix and the time-frequency feature matrix corresponding to the voice to be detected into a first fusion unit in a feature fusion module to obtain a first fusion feature matrix; inputting the advanced representation matrix and the time-frequency feature matrix corresponding to the voice to be detected into a second fusion unit in the feature fusion module to obtain a second fusion feature matrix; and inputting the first fusion feature matrix and the second fusion feature matrix into a third fusion unit in a feature fusion module to obtain a target fusion feature matrix, and performing positioning and detection of a forged voice sample on the to-be-detected voice based on the target fusion feature matrix. According to the invention, the authenticity of the whole voice can be judged, and the tampered area can be accurately positioned.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of speech detection, in particular to a method for positioning and detecting a fake speech sample. BACKGROUND

[0002] With the rapid development of artificial intelligence and digital speech processing technology, speech spoofing technology has reached an unprecedented level of realism, and can generate speech content that is almost indistinguishable from real people. In recent years, the research of speech spoofing detection technology mainly focuses on the direction of end-to-end deep learning model.

[0003] However, the traditional speech spoofing detection model can only judge the authenticity of the whole speech, and lacks the ability to accurately locate the tampered area. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a method for positioning and detecting a fake speech sample, which can not only judge the authenticity of the whole speech, but also accurately locate the tampered area.

[0005] In a first aspect, the present application provides a method for positioning and detecting a fake speech sample, which comprises: obtaining a to-be-detected speech, and a high-level representation matrix and a time-frequency feature matrix corresponding to the to-be-detected speech; the high-level representation matrix contains phoneme content, speaker identity and prosody information of the to-be-detected speech in each time period; the time-frequency feature matrix contains the logarithmic amplitude of energy of each frequency band in each time period in the to-be-detected speech; inputting the high-level representation matrix and the time-frequency feature matrix into a first fusion unit in a feature fusion module to map the first splicing features of the high-level representation matrix and the time-frequency feature matrix from a high-dimensional space to a pre-set low-dimensional space, and obtain a first fusion feature matrix; inputting the high-level representation matrix and the time-frequency feature matrix into a second fusion unit in the feature fusion module to perform feature fusion based on the correlation strength matrix between the high-level representation matrix and the time-frequency feature matrix, and obtain a second fusion feature matrix; the correlation strength matrix reflects the interaction and dependency relationship between the high-level representation matrix and the time-frequency feature matrix; inputting the first fusion feature matrix and the second fusion feature matrix into a third fusion unit in the feature fusion module to obtain a target fusion feature matrix, and positioning and detecting a fake speech sample of the to-be-detected speech based on the target fusion feature matrix.

[0006] In a possible implementation, the high-level representation matrix is obtained according to the following steps: inputting the to-be-detected speech into a target feature extraction module to obtain a high-level representation matrix; The target feature extraction module is trained based on the initial feature extraction module; the initial feature extraction module is trained based on unlabeled raw speech samples.

[0007] In one possible implementation, the initial feature extraction module is trained through the following steps: Obtain heterogeneous speech samples, fake speech samples, and enhanced speech samples corresponding to the original speech samples; the heterogeneous speech samples are obtained by replacing local speech segments in the original speech samples with other speech segments. The initial feature extraction module is trained for the first time according to the local heterogeneous reconstruction task; the local heterogeneous reconstruction task refers to the model training task of reconstructing the replaced local speech segments in the heterogeneous speech samples and restoring them to the original speech samples. The initial feature extraction module, which was completed in the first training, is trained a second time based on the real-fake comparison learning task to obtain the final initial feature extraction module. The real-fake contrast learning task refers to using the original speech sample and the fake speech sample as negative sample pairs, and the original speech sample and the enhanced speech sample as positive sample pairs, so that the initial feature extraction module performs contrast learning, thereby generating a priori high-level representation matrix; the priori high-level representation matrix includes the phoneme content, speaker identity and prosodic information of the speech at various time periods.

[0008] In one possible implementation, the step of inputting the high-level representation matrix and the time-frequency feature matrix into the second fusion unit of the feature fusion module to perform feature fusion based on the correlation strength matrix between the high-level representation matrix and the time-frequency feature matrix to obtain a second fused feature matrix includes: The high-level representation matrix is ​​converted into a query vector, and the time-frequency feature matrix is ​​converted into a key vector, so as to calculate the correlation strength matrix between the high-level representation matrix and the time-frequency feature matrix; The time-frequency feature matrix is ​​converted into a value vector, and the feature points in the value vector are weighted and summed based on the correlation strength matrix to obtain the second fusion feature matrix.

[0009] In one possible implementation, the step of inputting the first fused feature matrix and the second fused feature matrix into the third fusion unit of the feature fusion module to obtain the target fused feature matrix includes: The second concatenated features of the first fusion feature matrix and the second fusion feature matrix are normalized to obtain the target fusion feature matrix.

[0010] In one possible implementation, the step of locating and detecting forged speech samples based on the target fusion feature matrix includes: The target fusion feature matrix is ​​input into the global classification unit in the speech classification module to obtain the predicted global true / false classification result corresponding to the speech to be detected. The target fusion feature matrix is ​​input into the forgery localization unit in the speech classification module to obtain the predicted local true / false classification results of speech segments in each time period of the speech to be detected; the speech classification module is trained based on speech detection samples and corresponding localization and detection labels; the localization and detection labels include the actual global true / false classification results corresponding to the speech detection samples and the actual local true / false classification results of each speech frame. Based on the predicted local real / fake classification results of each speech frame, the location of the fake speech segment in the speech to be detected is determined.

[0011] In one possible implementation, the target loss function of the speech classification module includes a first loss function, a second loss function, and a third loss function; The first loss function is used to measure the consistency between the actual global true / false classification result and the predicted global true / false classification result corresponding to the speech detection sample; the second loss function is used to measure the consistency between the actual local true / false classification result and the predicted local true / false classification result of each speech frame in the speech detection sample; and the third loss function is used to measure the logical consistency between the predicted global true / false classification result corresponding to the speech detection sample and the predicted local true / false classification result of each speech frame in the speech detection sample.

[0012] Secondly, embodiments of this application also provide a device for locating and detecting forged voice samples, the device comprising: The acquisition module is used to acquire the speech to be detected, and the high-level representation matrix and time-frequency feature matrix corresponding to the speech to be detected; the high-level representation matrix includes the phoneme content, speaker identity and prosodic information of the speech to be detected in each time period; the time-frequency feature matrix includes the logarithmic amplitude of the energy in each frequency band of the speech to be detected in each time period. The input module is used to input the high-level representation matrix and the time-frequency feature matrix into the first fusion unit in the feature fusion module, so as to map the first concatenated features of the high-level representation matrix and the time-frequency feature matrix from the high-dimensional space to the preset low-dimensional space to obtain the first fused feature matrix; The input module is further configured to input the high-level representation matrix and the time-frequency feature matrix into the second fusion unit in the feature fusion module, so as to perform feature fusion based on the correlation strength matrix between the high-level representation matrix and the time-frequency feature matrix to obtain a second fused feature matrix; the correlation strength matrix reflects the interaction and dependency between the high-level representation matrix and the time-frequency feature matrix; The input module is further configured to input the first fusion feature matrix and the second fusion feature matrix into the third fusion unit in the feature fusion module to obtain a target fusion feature matrix, so as to locate and detect forged speech samples of the speech to be detected based on the target fusion feature matrix.

[0013] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method for locating and detecting fake voice samples as described in any of the first aspects.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the method for locating and detecting forged voice samples as described in any of the first aspects.

[0015] This application provides a method for locating and detecting forged speech samples. The method includes: acquiring; inputting the high-level representation matrix and time-frequency feature matrix corresponding to the speech to be detected into a first fusion unit in a feature fusion module to obtain a first fusion feature matrix; inputting the high-level representation matrix and time-frequency feature matrix corresponding to the speech to be detected into a second fusion unit in the feature fusion module to obtain a second fusion feature matrix; inputting the first fusion feature matrix and the second fusion feature matrix into a third fusion unit in the feature fusion module to obtain a target fusion feature matrix, so as to locate and detect forged speech samples based on the target fusion feature matrix. This application can determine both the overall authenticity of the speech and the precise location of the tampered area. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a method for locating and detecting forged voice samples provided in an embodiment of this application is shown; Figure 2 A flowchart illustrating the training process of the speech detection model provided in an embodiment of this application is shown. Figure 3 This illustration shows a schematic diagram of a device for locating and detecting fake voice samples according to an embodiment of this application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0019] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0020] To enable those skilled in the art to utilize the content of this application, and in conjunction with the specific application scenario of "speech detection," the following implementation methods are provided. For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is primarily described within the "speech detection field," it should be understood that this is merely an exemplary embodiment.

[0021] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0022] The following is a detailed description of a method for locating and detecting fake voice samples provided in the embodiments of this application.

[0023] Reference Figure 1 The diagram shown is a flowchart illustrating a method for locating and detecting forged voice samples according to an embodiment of this application. The exemplary steps of this embodiment are described below: S101. Obtain the speech to be detected, as well as the high-level representation matrix and time-frequency feature matrix corresponding to the speech to be detected.

[0024] In the embodiments of this application, the speech to be detected refers to the temporal speech signal that needs to be detected for spoofing in various scenarios that require speech spoofing detection (such as enterprise remote meetings and collaborations, social media and content platform review, etc.). The "high-level representation matrix" is a frame-level feature extracted by the target feature extraction module, which can condense and represent the key information of the speech to be detected. Specifically, it includes the phoneme content, speaker identity and prosodic information of the speech to be detected in each time period. Next, the information contained in the high-level representation matrix will be explained one by one: (1) Phoneme content: "What exactly is this speech saying" - refers to the pronunciation information corresponding to each speech frame, that is, the Chinese or English phoneme sequence (such as "sh-ih4-men"). (2) Speaker identity: "The fingerprint of who is speaking" - features related to the speaker's timbre. No matter what the content is, as long as it comes from the same person, the projection of the high-level representation matrix in a certain subspace will be close to each other; if it is a different person, the projection will be far away. (3) Prosodic information: In addition to "what is said (phoneme content)" and "who is saying (speaker identity)", it also retains "how it is said" - that is, suprasegmental features such as rhythm, stress, intonation, and pauses (prosodic information). For example, the rising intonation at the end of an interrogative sentence and the high-pitched intensity of anger will be written into this set of values ​​in a measurable way.

[0025] Specifically, the high-level representation matrix is ​​obtained according to the following steps: the speech to be detected is input into the target feature extraction module to obtain the high-level representation matrix. The target feature extraction module is obtained through supervised training based on the initial feature extraction module; the initial feature extraction module is obtained through unsupervised training based on unlabeled raw speech samples.

[0026] In this embodiment of the application, the voice to be detected is... Input the target feature extraction module to extract frame-level features (i.e., high-level representation matrix). );in, The duration corresponding to the speech to be detected. This represents the number of time periods corresponding to the high-level representation matrix (i.e., the dimension of the rows in the high-level representation matrix). This represents the dimension of the high-level representation vector corresponding to each time period in the high-level representation matrix (i.e., the dimension of the columns of the high-level representation matrix). It is a set of real numbers. Therefore, the high-level representation matrix contains high-level representation vectors corresponding to each time period; each dimension of the high-level representation vector corresponding to each time period does not directly correspond to a single physical quantity, but is a dense matrix learned by the model, which contains rich contextual information, specifically including phoneme content, speaker identity and prosodic information.

[0027] Here, the speech generated by advanced speech forgery techniques may have subtle differences from real speech at high-level semantic levels (such as the smoothness of phoneme transitions and the naturalness of prosody). The high-level representation matrix extracted by the target feature extraction module has powerful representational capabilities, enabling it to capture these semantic inconsistencies and improve the accuracy of speech detection.

[0028] Furthermore, the time-frequency feature matrix includes the logarithmic amplitude of energy in each frequency band of the speech to be detected over various time periods. Specifically, the time-frequency feature matrix is ​​an interpretable physical quantity that directly reflects the energy distribution of the speech to be detected in the time and frequency domains. Each feature point in the time-frequency feature matrix represents the logarithmic amplitude of energy in the speech to be detected over a specific time period and frequency band, simulating the nonlinear perception of frequency by the human ear. Many speech forgery methods leave physical traces in the time-frequency domain, such as: splicing traces: energy discontinuities and fundamental frequency abrupt changes may occur at splicing points. Generation traces: neural vocoders or generation models may produce specific frequency resonances or unnatural harmonic structures. Compression traces: after splicing speech from different sources, the encoded compression noise may be inconsistent. These local, subtle physical anomalies can be directly captured by time-frequency features. Therefore, using the time-frequency feature matrix for speech forgery detection will improve detection accuracy.

[0029] Specifically, the time-frequency feature matrix corresponding to the speech to be detected is obtained through the following steps: Step 1: Pre-emphasize the speech to be detected to balance high-frequency energy, and then divide the signal into frames to ensure local stability.

[0030] In the embodiments of this application, pre-emphasis refers to a simple filtering operation that "raises the high-frequency part first". The purpose is to counteract the natural high-frequency attenuation of the speech to be detected, so that the subsequent detection process is more sensitive to high frequencies.

[0031] Step 2: Apply Fast Fourier Transform (FFT) to each speech frame in the pre-emphasized speech to convert the time-domain signal into a linear spectrum. Then, square the magnitude of the spectral modulus of the FFT-transformed speech features to obtain the power spectrum of each frequency component. ; in, Let be the power spectrum of the k-th frequency component. This represents the number of feature points in the speech features after the Fast Fourier Transform. Let be the feature value of the nth feature point in the speech features after Fast Fourier Transform. The imaginary unit, It is the base of the natural logarithm.

[0032] Step 3: Use the Mel filter bank to scale the linear frequencies corresponding to the speech to be detected. Converted to Mel scale for simulating nonlinear perception in the human ear: .

[0033] in, It is the Mel scale of the linear frequency scale f.

[0034] Step 3: For each filter, perform a weighted summation of the power spectrum to output the Mel band energy. ; in, Let be the response of the m-th Mel filter to the k-th linear frequency. Let be the bandwidth energy of the m-th filter.

[0035] Step 4: Take the logarithm of the Mel band energy to dynamically compress the range and enhance the low-energy components: ; in, Use small constants to avoid numerical overflow. Time-frequency characteristic matrix. , This represents the number of frequency bands corresponding to each time period in the time-frequency feature matrix (i.e., the dimension of the columns of the time-frequency feature matrix). This represents the number of time periods corresponding to the time-frequency feature matrix (i.e., the dimension of the rows of the time-frequency feature matrix).

[0036] In addition, if Then, the time dimension of the high-level representation matrix and the time-frequency feature matrix also needs to be aligned through interpolation or pooling.

[0037] S102. Input the high-level representation matrix and the time-frequency feature matrix into the first fusion unit in the feature fusion module to map the first concatenated features of the high-level representation matrix and the time-frequency feature matrix from the high-dimensional space to the preset low-dimensional space to obtain the first fused feature matrix.

[0038] In the embodiments of this application, directly concatenating these two features results in an inconsistent and ill-conditioned mixed feature matrix. Therefore, this application proposes feature fusion based on projection mapping, aiming to map the concatenated two types of features to a unified space. The formula is as follows: ; ; in, This is the first concatenated feature of the high-level representation matrix and the time-frequency feature matrix. For high-level representation matrices, The time-frequency characteristic matrix, This is the first fusion feature matrix. Let be the learnable projection weight matrix of the first fusion unit. The learnable bias term for the first fusion unit (dimension: ), Let be the dimension of each row vector in the first fused feature matrix. The first splicing feature The dimensions of each row vector in the vector.

[0039] Here, the first fusion unit combines a first splicing feature that may be up to several thousand dimensions. Dimensionality reduction to a relatively low and fixed dimension (i.e., a pre-defined low-dimensional space). This ensures that the number of parameters in all subsequent modules (the third fusion unit, the speech classification module) is only related to... Related to, but related to the original feature dimension and Decoupling makes model design more flexible and lightweight. It significantly reduces computational load and improves the efficiency of model training and inference. It also plays a role in information compression and refinement, forcing the network to retain the most critical information and improving the robustness of features.

[0040] S103. Input the high-level representation matrix and the time-frequency feature matrix into the second fusion unit in the feature fusion module to perform feature fusion based on the correlation strength matrix between the high-level representation matrix and the time-frequency feature matrix, and obtain the second fused feature matrix.

[0041] In this embodiment, simple feature concatenation or addition essentially delegates the judgment to subsequent fully connected layers, which is a static and blind fusion. Speech detection models require a large amount of parameters and data to learn this complex "context-anomaly" association, resulting in extremely low efficiency in low-sample scenarios. Therefore, the second fusion unit utilizes a cross-modal attention mechanism to achieve feature fusion based on the correlation strength matrix between the high-level representation matrix and the time-frequency feature matrix. The correlation strength matrix reflects the interaction and dependency between the high-level representation matrix and the time-frequency feature matrix.

[0042] Specifically, the high-level representation matrix and the time-frequency feature matrix are input into the second fusion unit in the feature fusion module to perform feature fusion based on the correlation strength matrix between the high-level representation matrix and the time-frequency feature matrix, resulting in a second fused feature matrix, including: Step 1: Convert the high-level representation matrix into a query vector and the time-frequency feature matrix into a key vector to calculate the correlation strength matrix between the high-level representation matrix and the time-frequency feature matrix.

[0043] In this embodiment, the association strength matrix includes the association strength values ​​between each feature point in the high-level representation matrix and each feature point in the time-frequency feature matrix. Specifically, the high-level representation matrix is ​​converted into a query vector, the time-frequency feature matrix is ​​converted into a key vector, and the association strength matrix between the high-level representation matrix and the time-frequency feature matrix is ​​calculated using the following formula: ; ; in, The correlation strength matrix, For query vector, For key vectors, This is a transpose. The similarity matrix between the query matrix and the key matrix (i.e., the correlation strength matrix between the high-level representation matrix and the time-frequency feature matrix) reflects the correlation strength between high-level semantics and local acoustics. To query the projection matrix of the matrix, The key matrix is ​​the projection matrix.

[0044] Step 2: Convert the time-frequency feature matrix into a value vector, and perform a weighted summation of each eigenvalue in the time-frequency feature matrix based on the correlation strength matrix to obtain the second fused feature matrix.

[0045] i. Convert the time-frequency feature matrix into a value vector, and calculate the attention weight matrix based on the association strength matrix; the attention weight matrix includes the degree of attention each feature point in the query vector pays to each feature point in the value vector.

[0046] In the embodiments of this application, the calculated attention weight matrix makes it easier for subsequent global classification units / spoofing localization units to grasp the feature weight relationships at various time points and frequency ranges, thereby improving the speech detection progress.

[0047] The time-frequency feature matrix is ​​converted into a value vector using the following formula, and the attention weight matrix is ​​calculated based on the correlation strength matrix and the time-frequency feature matrix. .

[0048] ; ; Where V is the value vector; scaling factor This is used to prevent the Softmax gradient from vanishing due to excessively large dot product values; Softmax is used to normalize along the dimensions of the bond matrix to obtain the weight distribution. The projection matrix is ​​the value matrix.

[0049] ii. Based on the attention weight matrix, the feature points in the value vector are weighted and summed to obtain the second fusion feature matrix.

[0050] In the embodiments of this application, ; in, To determine the degree of attention paid by the i-th feature point (high-level feature) in the query vector to the j-th feature point (time-frequency feature) in the value vector; Let be the eigenvalue of the j-th feature point in the value vector; The second fusion feature matrix The fusion feature of the i-th feature point in the data; The features are an adaptive aggregation of global acoustic information, focusing on the local anomalies most relevant to the current semantics. These are attention-weighted features. The focus is on acoustic clues related to authentication.

[0051] S104. Input the first fusion feature matrix and the second fusion feature matrix into the third fusion unit in the feature fusion module to obtain the target fusion feature matrix, so as to locate and detect the forged speech sample of the speech to be detected based on the target fusion feature matrix.

[0052] In this embodiment, to enhance training stability, the second concatenated features of the first and second fused feature matrices are normalized to obtain the target fused feature matrix, mapping the feature interval to the range [0,1]. The formula is as follows:

[0053] in, , To fuse the feature matrix for the target, The number of time periods corresponding to the target fused feature matrix. Let LayerNorm[c1] be the fusion feature dimension corresponding to each time period in the target fusion feature matrix, and LayerNorm[c1] be the normalization operation (also known as the normalization layer).

[0054] Further, based on the target fusion feature matrix, the localization and detection of forged speech samples of the speech to be detected includes: Step 1: Input the target fusion feature matrix into the global classification unit in the speech classification module to obtain the predicted global true / false classification result corresponding to the speech to be detected.

[0055] In this embodiment, the global classification unit can perform authenticity identification on the target fusion feature matrix and return the predicted global authenticity classification result of the input speech to be detected. [c2] The following is a detailed description of inputting the target fusion feature matrix into the global classification unit in the speech classification module to obtain the predicted global authenticity classification result corresponding to the speech to be detected: First, the target feature matrix is ​​fused using an average pooling layer (AvgPooling). Pooling along the time dimension is expressed by the following formula: ; in, The feature matrix after pooling. Let be the target fusion feature vector corresponding to the t-th time period in the target fusion feature matrix.

[0056] Then, the probability that the detected speech x is fake is output through a fully connected layer. Its formula is expressed as follows: ; in, This is the classification weight matrix. For classification bias vector, This is the Sigmoid activation function.

[0057] Finally, if the probability of forgery is greater than the preset forgery probability threshold, the predicted global true / false classification result of the speech to be detected is a forged speech sample.

[0058] Step 2: Input the target fusion feature matrix into the forgery localization unit in the speech classification module to obtain the predicted local true and false classification results of speech segments in each time period of the speech to be detected.

[0059] Currently, most existing technologies only perform time-domain positioning. However, forgery traces often exist in the spatiotemporal domain, frequency domain, and even higher-level feature structures. Furthermore, many subtle forgery traces manifest as specific correlation patterns between time-domain and frequency-domain features, which cannot be effectively captured by standalone time-domain or frequency-domain analysis. This leads to problems such as incomplete positioning information and poor interpretability of forged positioning.

[0060] The forgery localization unit designed in this application extracts key forgery information from the input target fusion feature matrix through a time-frequency attention network. By continuously optimizing the time-frequency weight matrix, the time-frequency position of the forged speech sample is obtained, thereby achieving the localization output of the forged segment. The following is a detailed description of the forgery localization unit: i. Fuse the feature matrix of target [c3] Input the temporal branch of the time-frequency attention network in the fake localization unit and extract the temporal feature vector.

[0061] In this embodiment, the target fusion feature matrix is ​​first compressed through the average pooling layer in the temporal branch to extract an initial temporal feature matrix with temporal representativeness. The initial temporal feature matrix is ​​sequentially passed through the first one-dimensional convolutional layer Conv 1D combining the ReLU activation function and the second one-dimensional convolutional layer Conv 1D combining the Sigmoid function in the temporal branch to obtain the final temporal feature vector. .

[0062] ii. Fuse the preprocessed target feature matrix Input the frequency domain branch of the time-frequency attention network in the fake localization unit and extract the frequency domain feature vector.

[0063] In this embodiment, the target fusion feature matrix is ​​first passed through an average pooling layer in the frequency domain branch to focus on the distribution characteristics of spectral energy, thereby extracting the initial frequency domain feature matrix. The initial frequency domain feature matrix is ​​sequentially passed through the first one-dimensional convolutional layer Conv 1D combining the ReLU activation function and the second one-dimensional convolutional layer Conv 1D combining the Sigmoid function in the frequency domain branch to obtain the final frequency domain feature vector. .

[0064] The ReLU activation function introduces nonlinearity and filters redundant information, while the Sigmoid function normalizes the feature weights to the [0,1] interval, forming a feature importance weight vector in both the time and frequency domains. and .

[0065] iii. Using outer product operations to transform the time-domain feature vectors and frequency domain eigenvectors Feature fusion is performed to generate a two-dimensional time-frequency joint attention matrix, the formula of which is expressed as follows:

[0066] in, A two-dimensional time-frequency joint attention matrix The Middle The weight value of the k-th frequency component in each time dimension (the higher the weight, the more important the region is for speech forgery localization; this matrix can explicitly model the cross-correlation of time-frequency features). iv. Pass the time-frequency joint attention matrix sequentially through a two-dimensional convolutional Conv2D layer, a KAN (Kernelized Attention Network) layer, a Softmax normalization function, and a temporal decoder to obtain the predicted local true / false classification results of speech segments in each time period of the speech to be detected.

[0067] In this embodiment, to further uncover forgery traces, the network uses two-dimensional convolution to perform local pattern analysis on the attention matrix and combines this with the global perception capability of the KAN (Kernelized Attention Network) layer to capture subtle forgery features across the time and frequency domains. The network uses a temporal decoder to convert frame-by-frame forgery confidence scores into continuous timestamp predictions, outputting not only an overall authenticity judgment but also precise location of forged segments. For example, if the speech to be detected is divided into three time periods, this embodiment can yield the predicted local authenticity classification results (true or forged) for the first time period, the predicted local authenticity classification results for the second time period, and the predicted local authenticity classification results for the third time period.

[0068] Furthermore, this application requires pre-training of the speech detection model before detecting the speech to be detected. The speech detection model includes a target feature extraction module, a feature fusion module, and a speech classification module. The target feature extraction module is obtained through supervised training based on the initial feature extraction module; the initial feature extraction module is obtained through unsupervised training based on unlabeled raw speech samples.

[0069] Reference Figure 2 The diagram shown is a flowchart of the training process for the speech detection model provided in this application embodiment. The specific training process is as follows: Step 1: Perform unsupervised training on the initial feature extraction module based on the original speech samples.

[0070] In this embodiment of the application, the initial feature extraction module is trained through the following steps: i. Obtain heterogeneous speech samples, fake speech samples, and enhanced speech samples corresponding to the original speech samples. Heterogeneous speech samples are obtained by replacing local speech segments in the original speech samples with other speech segments.

[0071] In this application's embodiments, obtaining heterogeneous speech samples corresponding to the original speech samples includes: randomly replacing a small segment of the original speech sample with a segment from other speech or an artificially generated abnormal speech fragment, or randomly injecting specific acoustic noise into a small segment of the original speech sample to obtain heterogeneous speech samples. Obtaining forged speech samples corresponding to the original speech samples includes: using speech processing tools (such as vocoders and speech effects processors) to perform various controllable degradation processes on the unlabeled original speech samples (such as low bit rate compression, resampling distortion, and adding linear filtering effects) to artificially create a batch of "fake forged" samples (i.e., forged speech samples). Enhanced speech samples refer to clearer speech generated by applying a series of data enhancement techniques to the original speech samples.

[0072] ii. The initial feature extraction module is trained for the first time based on the local heterogeneous reconstruction task; the pre-local heterogeneous reconstruction task refers to the model training task of reconstructing the replaced local speech segments in heterogeneous speech samples and restoring them to the original speech samples.

[0073] In this embodiment, heterogeneous speech samples are input into the initial feature extraction module to obtain reconstructed speech; the initial feature extraction module is updated with the goal of minimizing the similarity between the reconstructed speech and the original speech sample until it exceeds a preset stopping condition, otherwise the initial feature extraction module is trained.

[0074] Here, the local heterogeneous reconstruction task forces the initial feature extraction module to learn the normal statistical regularities of the speech signal. In order to accurately reconstruct the damaged region, the initial feature extraction module must learn to identify anomalous inconsistencies in its surrounding context, thereby keenly capturing unnatural acoustic patterns that are highly consistent with the target of forgery detection.

[0075] iii. Based on the true / false comparison learning task, the initial feature extraction module completed in the first training is trained a second time to obtain the final initial feature extraction module.

[0076] In this embodiment of the application, the real vs. fake speech contrast learning task refers to using the original speech sample and the fake speech sample as negative sample pairs, and the original speech sample and the enhanced speech sample as positive sample pairs, so that the initial feature extraction module performs contrast learning, thereby generating a priori high-level representation matrix; the priori speech feature matrix includes the phoneme content, speaker identity and prosodic information of the speech at each time period.

[0077] Specifically, the original speech samples, fake speech samples, and enhanced speech samples are respectively input into the initial feature extraction module after the first training, to obtain the first prior speech feature matrix, the second prior speech feature matrix, and the third prior speech feature moment. The initial feature extraction module after the first training is updated with the goal of maximizing the similarity between the first and second prior speech feature matrices and minimizing the similarity between the second and third prior speech feature moments. The update continues until the similarity exceeds a preset stopping condition. Otherwise, the initial feature extraction module after the first training continues to be trained.

[0078] This task enables the model to establish a prior representation of speech authenticity in the early stages of learning, which provides an excellent parameter initialization point for subsequent few-sample fine-tuning, geared towards the true / false discrimination task.

[0079] Here, in the unsupervised pre-training phase, the model learns general speech representations from a large-scale unlabeled corpus through a self-supervised learning objective. The input is an unlabeled speech waveform. ,in Representing time-domain signals, the original waveform is transformed into latent feature representations using a convolutional neural network. , d represents the time step after downsampling, and d represents the feature dimension.

[0080] Step 2: Supervised training is performed on the final initial feature extraction module based on cloned speech and adversarial speech generated from the original speech samples to fine-tune the final initial feature extraction module and obtain the target feature extraction module. In this embodiment, the adversarial speech is obtained by adding perturbations to the human auditory perception features corresponding to the original speech sample. Specifically, the initial speech detection module is composed of the final initial feature extraction module, the constructed feature fusion module, and the constructed speech classification module; cloned speech, adversarial speech, and the original speech sample are used as speech samples, and the actual global true / false classification results corresponding to each speech sample and the actual local true / false classification results of each speech frame are used as labels to update the model parameters in the initial feature extraction module (at this stage, the feature fusion module and the speech classification module are not retrained). In the fine-tuning stage, the speech detection model is tested on the labeled dataset. Further fine-tuning, among which, This represents the actual true / false classification label for the i-th speech sample (including the actual global true / false classification result and the actual local true / false classification results for each speech frame). During the fine-tuning phase, the speech detection module employs cross-entropy loss. Conduct training: ; in, For the speech detection model, the i-th speech sample The probability that the actual global true / false classification result is true. Representing the The actual global true / false classification result of a speech sample This represents the total number of speech samples.

[0081] Here, the optimization goal of the speech detection model is to adjust the parameters to be updated in the initial feature extraction module. To minimize classification loss :

[0082] in, This represents the parameters to be updated in the initial feature extraction module. The loss function during the fine-tuning phase. Finding the loss function Minimum parameter .

[0083] Furthermore, the process of generating cloned speech and adversarial speech in this embodiment is as follows: i. Conventional data augmentation (such as noise addition, speed adjustment, and pitch shifting) mainly alters the vocal tract features or playback effect of speech, but cannot create new fake semantic content or generate speech content with features of other speakers. The most effective way to increase the coverage of training data is to collect more fake samples, but this is impossible in scenarios with few samples. The process by which this application's embodiment uses a timbre cloning module to generate cloned speech corresponding to the original speech sample is as follows: To address the challenge of scarce data for specific forgery algorithms, the voice cloning module can utilize a small number of original speech samples as voice references. Through transfer learning, it generates a large number of similar forged speech samples, thus resolving the problems of overfitting and insufficient generalization caused by small-sample training. This voice cloning module needs to construct clean speech samples from multiple speakers, covering gender, age, and accent, as the target library for voice cloning. It selects high-performance voice cloning algorithms such as ChatTTS, F5-TTS, and CosyVoice, then clones the few forged speech samples to multiple target voices. Furthermore, it combines generation strategies such as noise addition, resampling, and fundamental frequency adjustment to further expand the generated data and improve the scenario coverage of the expanded data.

[0084] For example, assuming the original few-sample data consists of 100 entries, the timbre cloning algorithm includes 5 types, the target timbre library includes 50 people, and the generation strategies such as noise addition and resampling include 3 types, then after the timbre cloning module, 100*5*50*3=75000 expanded voices can be obtained.

[0085] ii. Common data augmentation (such as adding noise, speed adjustment, and pitch shifting) primarily improves the model's robustness to natural variations and channel distortion. Adversarial examples are specifically designed for anti-spoofing / localization models; they are typically located near the model's decision boundary. Adding these examples to training is equivalent to proactively and purposefully "hardening" the model in its most uncertain and vulnerable areas. This directly leads to the model learning a clearer, sharper, and more robust decision boundary. It enhances the ability to distinguish between real and fake samples, significantly reducing the possibility of being "fooled" by minor perturbations, thereby directly improving the model's security and reliability. Furthermore, in situations where data is scarce, adversarial augmentation is an effective means of creating valuable training samples.

[0086] Specifically, the adversarial speech generation process for generating adversarial speech corresponding to the original speech samples using the adversarial generation module is as follows: by applying a small perturbation to the original samples, deceptive adversarial samples are generated, enabling the model to learn the attack patterns of such methods during training, thereby improving the robustness of the anti-counterfeiting model in detecting maliciously tampered or deceptive samples.

[0087] Common adversarial example generation techniques often originate from the image domain, and their adversarial perturbations are usually constrained to... Under the norm, but speech has unique temporal sequence and human auditory masking effect, simple Norm constraints can generate physically unrealizable or perceptible perturbations. To address the differences in norm constraints between the image and speech domains, this invention proposes three adversarial example generation methods based on image adversarial example generation methods and incorporating the characteristics of speech Fourier transform: The first generation method: the modified Fast Gradient Sign Method (FGSM), the process of which is as follows: ; ; in, The original input sample, To convert speech into a function of the space perceived by the human ear, yes The human ear's perceived features corresponding to the original speech samples output by the function. This is the prediction of the global true / false classification result for the speech detection model. Let L be the actual global true / false classification result corresponding to the original speech sample, and let L be the loss function of the speech detection model. For the loss function L on the input gradient operator, For symbolic functions, For the amplitude of the disturbance, Adversarial speech in the frequency domain generated in the human ear's perception space. The inverse Fourier transform converts speech from the human ear's perceptual feature space back into a speech sequence. Adversarial speech in the time domain.

[0088] Here, the attack and constraints of this method occur in the human ear's perceived feature space, rather than the original waveform space, thus avoiding the difference in norm constraints between the original FGSM algorithm in the image and speech domains. Similarly, the added perturbation is also guided to... The perceived components are relatively concentrated in certain frequency bands rather than being evenly distributed across all sample points, which is more consistent with the auditory masking effect.

[0089] The second generation method: Modified Projected Gradient Descent (PGD), which updates and projects the gradient into the perturbation constraint space through multiple small steps.

[0090]

[0091] in, For iteration round index; Preset step size; The projection function ensures that the disturbance does not exceed the threshold. ; For the first Adversarial speech in the frequency domain generated after one iteration of the FGSM algorithm For the first Adversarial speech in the time domain generated after one iteration of the FGSM algorithm.

[0092] Here, the entire iterative optimization process takes place within the human ear's perception space. In the process of projection function It also operates on perceptual features, which can generate stronger and more covert adversarial examples. The perturbation is precisely controlled within the perceptual dimensions that the human ear is sensitive to, achieving the same attack effect with less perceptual distortion, or achieving greater attack strength with the same perceptual budget.

[0093] The third generation method: the optimization-based C&W method, which directly optimizes the objective function, minimizing the perturbation while simultaneously deceiving the model. ; in, express Confrontational Voice Distance in space as perceived by the human ear Let c be the norm, and c be the attack strength; defined by... Physical constraints, thereby ensuring It is a physically achievable speech waveform.

[0094] Here, by systematically generating and utilizing adversarial examples, the forgery detection model can learn a more generalized distribution of forgery features, thereby more reliably detecting maliciously tampered content in practical applications.

[0095] Step 3: The target feature extraction module, the constructed feature fusion module, and the constructed speech classification module constitute the intermediate speech detection module; cloned speech, adversarial speech, and original speech samples are used as speech samples, and the actual global true / false classification results corresponding to each speech sample and the actual local true / false classification results of each speech frame are used as labels to update the speech classification module in the intermediate speech detection module, thus obtaining the final speech detection module.

[0096] In this embodiment, the speech classification module is trained based on speech detection samples and corresponding localization and detection labels. The localization and detection labels include the actual global true / false classification result corresponding to the speech detection sample and the actual local true / false classification result of each speech frame. The target loss function of the speech classification module includes a first loss function, a second loss function, and a third loss function. The first loss function is used to measure the consistency between the actual global true / false classification result and the predicted global true / false classification result corresponding to the speech detection sample. The second loss function is used to measure the consistency between the actual local true / false classification result and the predicted local true / false classification result of each speech frame in the speech detection sample. The third loss function is used to measure the logical consistency between the predicted global true / false classification result corresponding to the speech detection sample and the predicted local true / false classification result of each speech frame in the speech detection sample.

[0097] Here, existing forgery localization techniques rely on frame-by-frame forgery labels for training. The loss function is typically a weighted sum of global classification loss and local localization loss. The core problem with this approach is the assumption that global and local judgments will automatically remain consistent. In cases of sparse or noisy labels, this forgery localization method is highly prone to absurd situations where "globally classified as true, but locally displaying forged segments" or vice versa. This application addresses this problem by introducing... The third loss, which requires no additional annotation cost, acts as a logical regularization term, forcing the model's behavior to conform to the common sense that "if the whole is false, then there must be at least one false segment." The following is a description of the training loss module's process: The dual-branch structure of the anti-spoofing task requires the design of a joint loss function to balance global and local anti-spoofing objectives. The global classification unit provides the overall anti-spoofing result for the speech, and its output scalar... This represents the probability that the speech is fake (predicting the global true / false classification result). The fake localization unit provides the predicted local true / false classification results for speech segments at various time periods, and its output sequence... ,in This represents the probability that the t-th time period is fake.

[0098] The target loss function for training the speech classification module It consists of three parts: ; in, The weighting hyperparameters balance the importance of each loss term. The global classification loss (first loss function); This is the local localization loss (second loss function); This is the consistency loss (the third loss function).

[0099] Wherein, the first loss function The overall counterfeit detection results are supervised using binary cross-entropy (BCE), defined as follows: ; in, This represents the actual global true / false classification result of the speech detection sample. This represents the predicted global true / false classification result (the probability that a speech detection sample is a fake speech sample).

[0100] Second loss function Frame-by-frame labels for the training data are required during the training process. And using frame-by-frame BCE loss, defined as follows: ; in, Let T be the probability that the speech segment in the t-th time period of the speech detection sample is a fake speech sample, and let T be the total number of speech segments in the time period of the speech detection sample. This represents the actual global true / false classification result of the speech segment in the t-th time period of the speech detection samples. If the actual global true / false classification result of each speech frame in the t-th time period of the speech detection sample is true, then the actual global true / false classification result of the speech segment in the t-th time period of the speech detection sample is true; otherwise, it is fake speech.

[0101] Third loss function To ensure that global and local results are logically consistent, the following definition applies: ; in, To predict the probability that a speech detection sample in the global true / false classification results is fake, The third loss function is used to constrain the global probability by maximizing the probability of forgery in the predicted local true / false classification results for speech segments at each time period. Not lower than the local maximum probability This means that at least one time period in the fake voice sample is detected.

[0102] It should be noted that the authenticity classification result should include the forgery probability and the authenticity identifier (0 or 1).

[0103] Based on the same inventive concept, this application also provides a device for locating and detecting fake voice samples, which corresponds to the method for locating and detecting fake voice samples. Since the principle of the device in this application is similar to the method for locating and detecting fake voice samples described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0104] Reference Figure 3 The diagram shown is a schematic of a device for locating and detecting forged voice samples according to an embodiment of this application. The device includes: The acquisition module 301 is used to acquire the speech to be detected, and the high-level representation matrix and time-frequency feature matrix corresponding to the speech to be detected; the high-level representation matrix includes the phoneme content, speaker identity and prosodic information of the speech to be detected in each time period; the time-frequency feature matrix includes the logarithmic amplitude of energy in each frequency band of the speech to be detected in each time period. Input module 302 is used to input the high-level representation matrix and the time-frequency feature matrix into the first fusion unit in the feature fusion module, so as to map the first concatenated features of the high-level representation matrix and the time-frequency feature matrix from the high-dimensional space to the preset low-dimensional space to obtain the first fused feature matrix; The input module 302 is further configured to input the high-level representation matrix and the time-frequency feature matrix into the second fusion unit in the feature fusion module, so as to perform feature fusion based on the correlation strength matrix between the high-level representation matrix and the time-frequency feature matrix to obtain a second fused feature matrix; the correlation strength matrix reflects the interaction and dependency between the high-level representation matrix and the time-frequency feature matrix; The input module 302 is further configured to input the first fusion feature matrix and the second fusion feature matrix into the third fusion unit in the feature fusion module to obtain a target fusion feature matrix, so as to locate and detect forged speech samples of the speech to be detected based on the target fusion feature matrix.

[0105] like Figure 4As shown in the embodiment of this application, an electronic device 400 includes a processor 401, a memory 402, and a bus. The memory 402 stores machine-readable instructions that can be executed by the processor 401. When the electronic device is running, the processor 401 communicates with the memory 402 via the bus. The processor 401 executes the machine-readable instructions to perform the steps of the above-described method for locating and detecting fake voice samples.

[0106] Specifically, the memory 402 and processor 401 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 401 runs the computer program stored in the memory 402, it can execute the above-mentioned method for locating and detecting fake voice samples.

[0107] Corresponding to the above-described method for locating and detecting forged voice samples, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method for locating and detecting forged voice samples.

[0108] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0109] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0111] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method for locating and detecting forged voice samples described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, external hard drives, ROM, RAM, magnetic disks, or optical disks.

[0112] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for locating and detecting forged voice samples, characterized in that, The method includes: The process involves acquiring the speech to be detected, as well as the high-level representation matrix and time-frequency feature matrix corresponding to the speech to be detected. The high-level representation matrix contains the phoneme content, speaker identity, and prosodic information of the speech to be detected in each time period. The time-frequency feature matrix contains the logarithmic amplitude of the energy in each frequency band of the speech to be detected in each time period. The high-level representation matrix and the time-frequency feature matrix are input into the first fusion unit in the feature fusion module to map the first concatenated features of the high-level representation matrix and the time-frequency feature matrix from the high-dimensional space to the preset low-dimensional space to obtain the first fused feature matrix. The high-level representation matrix and the time-frequency feature matrix are input into the second fusion unit in the feature fusion module to perform feature fusion based on the correlation strength matrix between the high-level representation matrix and the time-frequency feature matrix, thereby obtaining a second fused feature matrix; the correlation strength matrix reflects the interaction and dependency between the high-level representation matrix and the time-frequency feature matrix; The first fusion feature matrix and the second fusion feature matrix are input into the third fusion unit in the feature fusion module to obtain the target fusion feature matrix, so as to locate and detect forged speech samples of the speech to be detected based on the target fusion feature matrix.

2. The method for locating and detecting forged voice samples according to claim 1, characterized in that, The high-level representation matrix is ​​obtained by following these steps: The speech to be detected is input into the target feature extraction module to obtain the high-level representation matrix; The target feature extraction module is trained based on the initial feature extraction module; the initial feature extraction module is trained based on unlabeled raw speech samples.

3. The method for locating and detecting forged voice samples according to claim 2, characterized in that, The initial feature extraction module is trained using the following steps: Obtain heterogeneous speech samples, fake speech samples, and enhanced speech samples corresponding to the original speech samples; the heterogeneous speech samples are obtained by replacing local speech segments in the original speech samples with other speech segments. The initial feature extraction module is trained for the first time according to the local heterogeneous reconstruction task; the local heterogeneous reconstruction task refers to the model training task of reconstructing the replaced local speech segments in the heterogeneous speech samples and restoring them to the original speech samples. The initial feature extraction module, which was completed in the first training, is trained a second time based on the real-fake comparison learning task to obtain the final initial feature extraction module. The real-fake contrast learning task refers to using the original speech sample and the fake speech sample as negative sample pairs, and the original speech sample and the enhanced speech sample as positive sample pairs, so that the initial feature extraction module performs contrast learning, thereby generating a priori high-level representation matrix; the priori high-level representation matrix includes the phoneme content, speaker identity and prosodic information of the speech at various time periods.

4. The method for locating and detecting forged voice samples according to claim 1, characterized in that, The step of inputting the high-level representation matrix and the time-frequency feature matrix into the second fusion unit of the feature fusion module to perform feature fusion based on the correlation strength matrix between the high-level representation matrix and the time-frequency feature matrix to obtain a second fused feature matrix includes: The high-level representation matrix is ​​converted into a query vector, and the time-frequency feature matrix is ​​converted into a key vector, so as to calculate the correlation strength matrix between the high-level representation matrix and the time-frequency feature matrix; The time-frequency feature matrix is ​​converted into a value vector, and the feature points in the value vector are weighted and summed based on the correlation strength matrix to obtain the second fusion feature matrix.

5. The method for locating and detecting forged voice samples according to claim 4, characterized in that, The step of inputting the first fused feature matrix and the second fused feature matrix into the third fusion unit in the feature fusion module to obtain the target fused feature matrix includes: The second concatenated features of the first fusion feature matrix and the second fusion feature matrix are normalized to obtain the target fusion feature matrix.

6. The method for locating and detecting forged voice samples according to claim 1, characterized in that, The step of locating and detecting forged speech samples based on the target fusion feature matrix includes: The target fusion feature matrix is ​​input into the global classification unit in the speech classification module to obtain the predicted global true / false classification result corresponding to the speech to be detected. The target fusion feature matrix is ​​input into the spoofing localization unit in the speech classification module to obtain the predicted local true / false classification results of speech segments in each time period of the speech to be detected; the speech classification module is trained based on speech detection samples and corresponding localization and detection labels; the localization and detection labels include the actual global true / false classification results corresponding to the speech detection samples and the actual local true / false classification results of each speech frame; Based on the predicted local real / fake classification results of each speech frame, the location of the fake speech segment in the speech to be detected is determined.

7. The method for locating and detecting forged voice samples according to claim 6, characterized in that, The target loss function of the speech classification module includes a first loss function, a second loss function, and a third loss function; The first loss function is used to measure the consistency between the actual global true / false classification result and the predicted global true / false classification result corresponding to the speech detection sample; the second loss function is used to measure the consistency between the actual local true / false classification result and the predicted local true / false classification result of each speech frame in the speech detection sample; and the third loss function is used to measure the logical consistency between the predicted global true / false classification result corresponding to the speech detection sample and the predicted local true / false classification result of each speech frame in the speech detection sample.

8. A device for locating and detecting forged voice samples, characterized in that, The device includes: The acquisition module is used to acquire the speech to be detected, and the high-level representation matrix and time-frequency feature matrix corresponding to the speech to be detected; the high-level representation matrix includes the phoneme content, speaker identity and prosodic information of the speech to be detected in each time period; the time-frequency feature matrix includes the logarithmic amplitude of the energy in each frequency band of the speech to be detected in each time period. The input module is used to input the high-level representation matrix and the time-frequency feature matrix into the first fusion unit in the feature fusion module, so as to map the first concatenated features of the high-level representation matrix and the time-frequency feature matrix from the high-dimensional space to the preset low-dimensional space to obtain the first fused feature matrix; The input module is further configured to input the high-level representation matrix and the time-frequency feature matrix into the second fusion unit in the feature fusion module, so as to perform feature fusion based on the correlation strength matrix between the high-level representation matrix and the time-frequency feature matrix to obtain a second fused feature matrix; the correlation strength matrix reflects the interaction and dependency between the high-level representation matrix and the time-frequency feature matrix; The input module is further configured to input the first fusion feature matrix and the second fusion feature matrix into the third fusion unit in the feature fusion module to obtain a target fusion feature matrix, so as to locate and detect forged speech samples of the speech to be detected based on the target fusion feature matrix.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method for locating and detecting fake voice samples as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method for locating and detecting fake voice samples as described in any one of claims 1 to 7.

Images