Graph attention based method and system for detecting fake speech

Abstract

The application provides a kind of based on graph attention fake voice detection method and system, through pre-emphasis and self-supervised pre-training model and single classification learning, make up the fake voice layer and the defect of insufficient training data;When extracting high-level feature representation, attention is paid to the channel, capturing more rich channel information;When using graph calculation for graph modeling, pay attention to the frequency domain sub-band and time domain section with more abundant information;In the multi-level collaborative heterogeneous graph attention fusion mechanism, the influence and interaction of time domain and frequency domain information are fully considered, and more efficient, more general performance and more accurate recognition rate of fake voice detection are realized.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a method and system for detecting fake speech based on graph attention. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] The rapid development of digital voice communication has led to increasingly frequent applications of digital voice, resulting in digital audio and related materials becoming among the most popular multimedia applications, widely used in news, business, politics, entertainment, military, and science. Simultaneously, the rapid development of social networks with information-sharing capabilities and powerful yet user-friendly media editing tools has exacerbated security concerns regarding the authenticity of media data. Voice-based automatic speaker authentication systems are also vulnerable to attacks involving forged speech, such as playback (pre-recorded audio), text-to-speech (converting text into spoken words), and speech conversion (converting speech from a source speaker to a target speaker). Among these, the rapid advancement of speech synthesis technology has made synthetic speech attacks (including text-to-speech (TTS) and speech conversion (VC)) an increasingly significant threat to speaker verification systems. Therefore, forged speech detection has become crucial.

[0004] Existing research on spoofed speech detection primarily focuses on front-end feature extraction to effectively represent speech and better detect forgery traces. Then, the extracted features are effectively modeled for back-end classification and decision-making methods to complete the spoofed speech detection. Feature extraction for speech spoofing detection mainly relies on speech signal processing methods. Researchers primarily explore effective acoustic features capable of detecting spoofed speech, extracting a series of acoustic features from the input waveform using methods such as Fast Fourier Transform. Examples include spectrograms based on spectral energy information, residual logarithmic amplitude spectra, and features utilizing constant Q-value transforms. While various acoustic features have achieved varying degrees of effectiveness in spoofed speech detection, Sahidulah et al.'s analysis of dozens of acoustic features on the ASVspoof2019 dataset revealed that no single feature can effectively handle attacks using unknown spoofing methods, exhibiting insufficient generalization ability.

[0005] The inspiration for backend classification after frontend feature extraction comes from the automatic speaker verification task based on Gaussian Mixture Models (GMMs). This approach assumes an input waveform is true or false and trains two GMMs to perform maximum likelihood analysis on real and fake data. With the development of deep learning, related methods have gradually been applied to speech forgery detection and have achieved remarkable results. For example, methods using convolutional networks like LCNN and ResNet, as well as methods combining temporal convolutional networks to model temporal information, have all achieved good results in the field of speech forgery detection. However, these methods linearly model time-domain frames or frequency-domain sub-band representations, ignoring the information richness of different frequency-domain sub-bands or time-domain segments relative to their neighbors, making it difficult to distinguish which are the most different.

[0006] The current mainstream decision-making method treats the fake speech detection task as a binary classification problem. However, it has some sensitive issues, such as classification performance depending on the number and distribution of fake samples in the dataset, insufficient fake samples, different distributions of test and training data, and unknown forgery methods. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a method and system for detecting forged speech based on graph attention. By using a self-supervised pre-trained model to obtain potential speech information from pre-emphasized speech, considering channel information, and employing a multi-level collaborative heterogeneous graph attention fusion mechanism, this method fully considers the interactive information of multi-level graph structures, achieving higher efficiency, stronger generalization performance, and more accurate recognition rate for forged speech detection.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] The first aspect of this invention provides a method for detecting fake speech based on graph attention.

[0010] A graph attention-based method for detecting fake speech includes the following steps:

[0011] The raw audio data is preprocessed, and then the primary feature vector is obtained through a speech self-supervised model.

[0012] Based on the primary feature vectors, a high-level feature representation of the original waveform is extracted through a residual coding module with channel attention;

[0013] Based on the high-level representation of the original waveform, time-domain and frequency-domain feature map representations are constructed through graph computation.

[0014] The time-domain feature map representation and the frequency-domain feature map representation are fused through time-frequency heterogeneous graph attention to obtain the time-frequency heterogeneous graph feature representation;

[0015] The time-domain graph feature representation and the frequency-domain graph feature representation are linearly transformed, and then the time-domain graph attention weight and the frequency-domain graph attention weight are extracted respectively. The time-domain graph feature representation and the frequency-domain graph feature representation are then added and fused point by point to extract the time-frequency graph attention weight.

[0016] Based on the attention weights of the time-domain graph, the attention weights of the frequency-domain graph, and the attention weights of the time-frequency graph, the feature representations of the time-frequency heterogeneous graph are weighted and fused to obtain three weighted graph feature representations.

[0017] The three weighted graph feature representations are heterogeneously fused and decomposed to obtain three interactive weighted graph feature representations.

[0018] The maximum value node and the average value node of the three interactive weighted graph features are extracted and concatenated. Based on the single-classification loss function, the fake speech detection result is obtained.

[0019] As a further limitation of the first aspect of the present invention, the original audio data is preprocessed, and then a processed primary feature vector is obtained through a speech self-supervised model, including:

[0020] The original audio is pre-emphasized, and the pre-emphasized speech data is fed into a self-supervised pre-trained model. The training loss is minimized by the weighted cross-entropy objective function. After passing through a fully connected layer added to the output, the primary feature vector is obtained.

[0021] As a further limitation of the first aspect of the present invention, based on the primary feature vector, a high-level feature representation of the original waveform is extracted through a residual coding module with channel attention, including:

[0022] The residual coding module with channel attention is a squeeze-excited residual coding module, which includes six basic units. In each basic unit, batch normalization, SELU activation and two-dimensional convolution are performed twice on the input primary feature vector in sequence. Then, attention is assigned to the channels, the obtained feature vector is added to the recorded primary feature vector and max pooling is used to obtain the feature coding vector. After iteratively combining the input and output of the six basic units, a high-level feature representation of the original speech is obtained.

[0023] As a further limitation of the first aspect of the invention, based on the high-level representation of the original waveform, a time-domain feature map representation and a frequency-domain feature map representation are constructed through graph computation, including:

[0024] Max pooling is performed on the high-level feature representation to learn separate frequency domain and time domain feature representations. Graph attention mechanisms are then applied to the frequency domain and time domain feature representations respectively to extract time domain feature graph representations and frequency domain feature graph representations.

[0025] As a further limitation of the first aspect of the present invention, the time-domain feature map representation and the frequency-domain feature map representation are fused through time-frequency heterogeneous graph attention to obtain a time-frequency heterogeneous graph feature representation, including:

[0026] The time-domain feature map representation and the frequency-domain feature map representation are projected onto another latent space with a common dimension using a linear transformation, and then fed into the time-frequency heterogeneous graph attention layer to construct a fused time-frequency heterogeneous graph feature representation.

[0027] As a further limitation of the first aspect of the present invention, the time-domain graph feature representation and the frequency-domain graph feature representation are linearly transformed, and then the time-domain graph attention weight and the frequency-domain graph attention weight are extracted respectively. The time-domain graph feature representation and the frequency-domain graph feature representation are then added and fused point by point to extract the time-frequency graph attention weight, including:

[0028] The obtained time-domain graph feature representation and frequency-domain graph feature representation are linearly transformed to make their number of nodes the same as the dimension of the time-frequency heterogeneous graph. Then, graph attention is applied to the time-domain feature graph representation and frequency-domain feature graph representation respectively to extract the time-domain graph attention weight and frequency-domain graph attention weight. Finally, the time-domain feature graph representation and frequency-domain feature graph representation are added point by point to apply graph attention to extract the time-frequency graph attention weight.

[0029] As a further limitation of the first aspect of the present invention, the time-frequency heterogeneous graph feature representations are weighted and fused according to the time-domain graph attention weights, the frequency-domain graph attention weights, and the time-frequency graph attention weights to obtain three weighted graph feature representations, including:

[0030] The time-domain graph attention weight, frequency-domain graph attention weight, and time-frequency graph attention weight are applied to the time-frequency heterogeneous graph, respectively, and graph attention and graph pooling are applied to extract the time-domain weighted graph, frequency-domain weighted graph, and time-frequency weighted graph.

[0031] As a further limitation of the first aspect of the present invention, the three weighted graph feature representations obtained are heterogeneously fused and decomposed to obtain three interactive weighted graph feature representations, including:

[0032] An edge, including self-connection, is established between every two nodes in the three weighted graph feature representations to construct a multi-level collaborative heterogeneous weighted fusion graph, where the weight of each edge is learned using a self-attention mechanism;

[0033] Aggregate the weights of each node and the edges it connects to obtain information about each node's neighboring nodes;

[0034] Each node information and the aggregated information are projected onto the target dimension through two projection vectors and fused to obtain the feature representation of each new node in the weighted fusion graph, thereby obtaining a multi-level collaborative heterogeneous weighted fusion graph.

[0035] Batch normalization is applied to the obtained multi-level collaborative heterogeneous weighted fusion graph, and SELU activation is performed. The weighted fusion graph is sliced ​​according to the number of nodes in the three weighted graphs before fusion to extract the interactive weighted graph feature representation.

[0036] As a further limitation of the first aspect of the present invention, the maximum value node and the average value node of the three interactive weighted graph features are extracted and concatenated. Based on the single-classification loss function, the forged speech detection result is obtained, including:

[0037] The maximum value node and the average value node of the three interactive weighted graph features are extracted and concatenated to obtain the final feature representation;

[0038] The final feature representation is fed into a single-classification loss function, and the model is optimized end-to-end to obtain the forged speech classification and detection results.

[0039] A second aspect of the present invention provides a forged speech detection system based on graph attention.

[0040] A graph attention-based fake speech detection system includes:

[0041] The primary feature representation acquisition module is configured to: preprocess the raw audio data and then obtain the processed primary feature vector through a speech self-supervised model;

[0042] The advanced feature representation acquisition module is configured to extract advanced feature representations of the original waveform from the primary feature vectors using a residual coding module with channel attention.

[0043] The feature map representation acquisition module is configured to construct time-domain and frequency-domain feature map representations based on the high-level representation of the original waveform through graph computation.

[0044] The time-frequency heterogeneous graph feature representation acquisition module is configured to fuse the time-domain feature map representation and the frequency-domain feature map representation through time-frequency heterogeneous graph attention to obtain the time-frequency heterogeneous graph feature representation.

[0045] The attention weight acquisition module is configured to: perform a linear transformation on the time-domain graph feature representation and the frequency-domain graph feature representation, then extract the time-domain graph attention weight and the frequency-domain graph attention weight respectively, and then perform point-by-point addition and fusion of the time-domain graph feature representation and the frequency-domain graph feature representation to extract the time-frequency graph attention weight;

[0046] The weighted graph feature representation acquisition module is configured to: perform weighted fusion of the time-frequency heterogeneous graph feature representations based on the time-domain graph attention weights, frequency-domain graph attention weights, and time-frequency graph attention weights to obtain three weighted graph feature representations;

[0047] The interactive weighted graph feature representation acquisition module is configured to perform heterogeneous fusion and decomposition on the three obtained weighted graph feature representations to obtain three interactive weighted graph feature representations.

[0048] The fake speech decision module is configured to extract the maximum value node and the average value node from the three interactive weighted graph features, concatenate them, and obtain the fake speech detection result based on the single classification loss function.

[0049] Compared with the prior art, the beneficial effects of the present invention are:

[0050] 1. This invention innovatively proposes a spoofing speech detection method and system based on graph attention. By using a self-supervised pre-trained model to obtain potential speech information from pre-emphasized speech, considering channel information, and fully taking into account the interaction information of multi-level graph structures, it achieves spoofing speech detection with higher efficiency, stronger generalization performance and more accurate recognition rate.

[0051] 2. This invention innovatively proposes a method and system for detecting fake speech based on graph attention. By using pre-emphasis and self-supervised pre-training modules, it learns the potential information of real and fake speech, making up for the shortcomings of the endless emergence of fake speech and insufficient training data.

[0052] 3. This invention innovatively proposes a graph attention-based method and system for detecting fake speech. It uses the SE-ResNet feature encoding module to allocate attention to channels when extracting high-level feature representations, thereby capturing richer channel information.

[0053] 4. This invention innovatively proposes a forged speech detection method and system based on graph attention. It uses a graph computing module to perform non-Euclidean data manifold modeling on high-level features across different frequency sub-bands and time segments, in order to focus on frequency sub-bands and time segments with richer information.

[0054] 5. This invention innovatively proposes a method and system for detecting fake speech based on graph attention. It uses time-frequency heterogeneous graph attention to fuse separate time-domain and frequency-domain graphs into a single time-frequency heterogeneous graph. It employs a multi-level collaborative heterogeneous graph attention fusion mechanism to simultaneously emphasize the influence of time-domain and frequency-domain information on the fused graph when performing graph feature fusion, and fully considers the interaction between the time-domain and frequency-domain, thereby capturing richer time-frequency domain features.

[0055] 6. This invention innovatively proposes a method and system for detecting fake speech based on graph attention. It uses a single-classification learning module to identify real speech during end-to-end model optimization, while treating others as abnormal, thus making up for the deficiency of the endless stream of fake speech data.

[0056] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0057] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0058] Figure 1 This is a flowchart illustrating the graph attention-based forged speech detection method provided in Embodiment 1 of the present invention.

[0059] Figure 2 This is a structural diagram of the SE-ResNet basic module (SE-ResNet block) provided in Embodiment 1 of the present invention;

[0060] Figure 3 This is a structural diagram of the SE-ResNet feature encoding module provided in Embodiment 1 of the present invention;

[0061] Figure 4 This is a structural diagram of the multi-level collaborative heterogeneous graph attention fusion mechanism provided in Embodiment 1 of the present invention. Detailed Implementation

[0062] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0063] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0064] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0065] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0066] Example 1:

[0067] like Figure 1As shown, Embodiment 1 of the present invention provides a method for detecting fake speech based on graph attention. Through a self-supervised pre-training module, it learns the potential information of real and fake speech and extracts primary features of the original speech. Through an SE-ResNet feature encoding module, it allocates attention to channel information to extract high-level feature representations. Using a graph computation module, it performs non-Euclidean data manifold modeling on the high-level features across different frequency sub-bands and time segments, extracting separate time-domain and frequency-domain graph feature representations. Using time-frequency heterogeneous graph attention, it fuses the separate time-domain and frequency-domain graphs into a single time-frequency heterogeneous graph. Using a multi-level collaborative heterogeneous graph attention fusion mechanism, it simultaneously emphasizes the influence of time-domain and frequency-domain information on the fused graph, extracting graph feature representations of time-domain and frequency-domain interaction, and then cascading the interactive graph features for final real / fake speech classification. Finally, through a single-classification learning module, it performs end-to-end model optimization.

[0068] Specifically, it includes the following processes:

[0069] S1: Perform preprocessing on the raw audio data and feed it into the speech self-supervised model to obtain the processed primary feature vector;

[0070] S2: Input the primary feature vector obtained through S1 into the squeeze excitation residual (SE-ResNet) feature encoding module to extract the high-level feature representation of the original waveform;

[0071] S3: Using the graph computation module, the high-level representation of the original waveform is used to construct a time-domain feature map and a frequency-domain feature map, resulting in two graph feature representations;

[0072] S4: By using time-frequency heterogeneous graph attention, the time-domain feature map representation and the frequency-domain feature map representation obtained in S3 are fused to obtain a time-frequency heterogeneous graph feature representation;

[0073] S5: In the multi-level collaborative heterogeneous graph attention fusion mechanism, the time-domain graph feature representation and frequency-domain graph feature representation obtained in S3 are linearly transformed, and then the time-domain graph attention weight and frequency-domain graph attention weight are extracted respectively. Then, the time-domain graph feature representation and frequency-domain graph feature representation are added and fused point by point to extract the time-frequency graph attention weight and obtain three multi-level graph attention weights.

[0074] S6: The three multi-level graph attention weights obtained in S5 are weighted and fused with the time-frequency heterogeneous graph feature representations obtained in S4 to obtain three weighted graph feature representations;

[0075] S7: Perform heterogeneous fusion and decomposition on the three weighted graph feature representations obtained in S6 to obtain three interactive weighted graph feature representations;

[0076] S8: Extract the maximum value node and the average value node from the three interactive weighted graph features obtained in S7, concatenate them, and then feed them into a single-classification loss function for real and fake speech classification.

[0077] In S1, the original speech is pre-emphasized and fed into a pre-trained self-supervised model, including:

[0078] S1.1: Each original audio segment is trimmed or concatenated to a duration of k seconds, and then pre-emphasis is applied to obtain pre-processed speech. The calculation process is as follows:

[0079] p(t)=s(t)-α·s(t-1))(1)

[0080] Where s(t) represents the input speech signal value, α represents the pre-emphasis coefficient, set to 0.97, and p(t) represents the speech signal value after pre-emphasis.

[0081] S1.2: Feed the preprocessed speech data into the self-supervised pre-trained model, use the weighted cross-entropy objective function to minimize the training loss, fine-tune the self-supervised pre-trained model wav2vec 2.0, add a fully connected layer at the output end to reduce the output size, and obtain the primary feature vector X.

[0082] In S2, high-level feature representations of the original waveform are extracted, such as... Figure 2 The diagram shown is a structural diagram of the basic modules of SE-ResNet. Figure 3 The diagram shows the structure of the SE-ResNet feature encoding module. Specifically, it consists of six stacked basic squeeze-activated residual encoding modules. Each basic module first records the input primary feature vector X, then performs two batch normalization (BN), SELU activation, and a 2×5 two-dimensional convolution (Conv2d). Next, the SENet feature encoding module is used to improve the channel information. Finally, the obtained feature vector is added to the recorded primary feature vector and max pooling (MaxPool2d) is used to obtain the feature encoding vector X′. The calculation process of the basic module is as follows:

[0083] X′=MaxPool2d(X+SE(Conv2d(Selu(BN(Conv2d(Selu(BN(X)))))))) (2)

[0084] By superimposing multiple basic modules and iteratively updating the feature encoding vector, a high-level feature representation S of the original speech is obtained.

[0085] In S3, two graph feature representations are obtained, including:

[0086] Max pooling is performed on the absolute values ​​of the extracted high-level features S to learn separate frequency domain and time domain feature representations. Then, graph attention mechanisms are applied to the frequency domain and time domain feature representations respectively to extract time domain feature graph representations and frequency domain feature graph representations. The calculation process is as follows:

[0087] G t =graph_module(max) F (abs(S))) (3)

[0088] G s =graph_module(max) T (abs(S))) (4)

[0089] in, S∈R C×F×T G t and G s Representing the time-domain plot and frequency-domain plot respectively, N t and N s D represents the number of nodes in the time-domain graph and the frequency-domain graph, respectively. t and D s Represent the dimensions of the time-domain plot and the frequency-domain plot, respectively; C, F, and T represent the number of channels, frequency spectral lines, and time frames of the high-level feature S, respectively, and max F This indicates that max pooling is performed across the frequency domain on high-level features, max T This indicates that max pooling is applied across the time domain to high-level features, and graph_module represents a combination of graph attention and graph pooling layers.

[0090] In S4, a time-frequency heterogeneous graph feature representation is obtained, including:

[0091] Concurrent modeling is performed on the extracted time-domain and frequency-domain feature maps with different node dimensions using a time-frequency heterogeneous graph attention mechanism, such as... Figure 4 As shown, firstly, a linear transformation is used to project the time-domain and frequency-domain maps onto another latent space with a common dimension. Then, this is input into the time-frequency heterogeneous map attention layer to construct a fused time-frequency heterogeneous map feature representation. The calculation process is as follows:

[0092] G st =heterog_module(W t (G t )||W s (G s (5)

[0093] Among them, W t and W s Let represent the projection matrices that project the time-domain plot and the frequency-domain plot onto the target dimension, respectively. Represents a time-frequency heterogeneity graph, Nst and D st These represent the number of nodes and the number of dimensions of the time-frequency heterogeneous graph, respectively. || represents the concatenation and addition operation of the graph nodes, and heteroog_module represents the attention layer of the heterogeneous graph.

[0094] In S5, attention weights for three multi-level graphs are obtained, including:

[0095] Linear transformations are performed on the obtained time-domain and frequency-domain graph features to make their number of nodes the same as the dimension of the time-frequency heterogeneous graph. Then, graph attention is applied to the time-domain and frequency-domain feature maps respectively to extract the time-domain graph attention weights and frequency-domain graph attention weights. Finally, graph attention is applied to the time-domain and frequency-domain feature maps point by point to extract the time-frequency graph attention weights. The calculation process is as follows:

[0096] α t =GAT(W map_t (G t (6)

[0097] α s =GAT(W map_s (G s (7)

[0098]

[0099] Where, α t α s and α st W represents the attention weights in the time-domain plot, frequency-domain plot, and time-frequency plot, respectively. map_* Represents a linear transformation. This represents point-by-point addition, and GAT represents graph attention operation.

[0100] In S6, three weighted graph feature representations are obtained, including:

[0101] The obtained time-domain graph attention weights, frequency-domain graph attention weights, and time-frequency graph attention weights are used to weight the resulting time-frequency heterogeneous graph, and graph attention and graph pooling are applied to extract the time-domain weighted graph, frequency-domain weighted graph, and time-frequency weighted graph. The calculation process is as follows:

[0102] G′ t =graph_module(G t ·α t (9)

[0103] G′ s =graph_module(G s ·α s (10)

[0104] G′st =graph_module(G st ·α st (11)

[0105] in, and These are time-domain, frequency-domain, and time-frequency weighted graphs, respectively.

[0106] In S7, three interaction-weighted graph feature representations are obtained, including:

[0107] S7.1: For each pair of nodes in the resulting three weighted graphs, establish an edge including self-connections, and use a self-attention mechanism to learn the weight of each edge. Specifically:

[0108] The resulting three weighted graphs are used to calculate the weights of each edge, including self-connections, between each pair of graphs using a different projection vector, for a total of nine different projection vectors. The calculation process is as follows:

[0109]

[0110] Among them, e * α represents the feature vector of a node. v，n W represents the weight between node v and node n. pro For learnable projection vectors, Let n represent the set of adjacent nodes of node n, and ⊙ represent element-wise multiplication.

[0111] S7.2: Aggregate the adjacent nodes of each node by the weight of the edge between every two nodes. Specifically:

[0112] The weights of each node and its connected edges are aggregated to obtain information about each node's neighboring nodes. The calculation process is as follows:

[0113]

[0114] Where, m n For the aggregated information of node n, e v This represents the feature vector of node v.

[0115] S7.3: By aggregating information, the feature representation of each new node in the weighted fusion graph is extracted to obtain a multi-level collaborative heterogeneous weighted fusion graph. Specifically:

[0116] Each node's information and the obtained aggregated information are projected onto the target dimension through two projection vectors and then fused to obtain a new feature representation for each node in the weighted fused graph. The calculation process is as follows:

[0117] o n =SeLu(BN(Watt (m n )+W res (e n (14)

[0118] Among them, o n W represents the eigenvector of node n in the weighted fusion graph. att To project the aggregated information onto the projection vector of the target dimension, W res To project the unchanged node feature vectors onto the target dimension.

[0119] S7.4: The multi-level collaborative heterogeneous weighted fusion graph G is decomposed by applying batch normalization and SELU activation. Then, the weighted fusion graph is sliced ​​according to the number of nodes in the three weighted graphs before fusion to extract the interactive weighted graph feature representation. The calculation process is as follows:

[0120] G″ t =narrow t (SeLu(BN(G))) (16)

[0121] G″ s =narrow s (SeLu(BN(G))) (17)

[0122] G″ st =narrow st (SeLu(BN(G))) (18)

[0123] Where G is a multi-level collaborative heterogeneous weighted fusion graph, BN is a normalization layer, SeLu is a SeLu activation layer, and narrow is a slicing function, G″ t G″ s and G″ st These are the interaction time domain, interaction frequency domain, and interaction time-frequency weighted graphs, respectively.

[0124] In S8, specifically, this includes:

[0125] S8.1: Extract the maximum value node and the average value node from the three interactive weighted graph features obtained, and concatenate them to obtain the final feature representation.

[0126] S8.2: The final feature representation is fed into the speaker attractor multicenter single-classification loss function to perform end-to-end optimization of the model. The calculation process of the single-classification loss function is as follows:

[0127]

[0128] in,

[0129]

[0130] Let y be the final feature representation of the i-th utterance in the batch. i For true and false labels, s i For the corresponding speaker, To represent the average final feature of the normalized speaker's discourse, It is the average final feature representation of speaker s learned from the training set, α is the scaling factor, m0 and m1 are the margins of the true and false classes, respectively, N is the number of utterances in the batch, and d i To calculate the cosine similarity between the final feature representation and the speaker's average final feature representation.

[0131] This invention overcomes the shortcomings of rampant fake speech and insufficient training data by using a self-supervised pre-trained model and single-classification learning. When extracting high-level feature representations, it assigns attention to channels, capturing richer channel information. When using graph computing for graph modeling, it focuses on the more information-rich frequency domain sub-bands and time domain segments. In the multi-level collaborative heterogeneous graph attention fusion mechanism, it fully considers the influence and interaction of time domain and frequency domain information, thereby capturing richer time-frequency domain features. Ultimately, it achieves more accurate and better generalization in fake speech detection.

[0132] Example 2:

[0133] Embodiment 2 of the present invention provides a forged speech detection system based on graph attention, comprising:

[0134] The primary feature representation acquisition module is configured to: preprocess the raw audio data and then obtain the processed primary feature vector through a speech self-supervised model;

[0135] The advanced feature representation acquisition module is configured to extract advanced feature representations of the original waveform from the primary feature vectors using a residual coding module with channel attention.

[0136] The feature map representation acquisition module is configured to construct time-domain and frequency-domain feature map representations based on the high-level representation of the original waveform through graph computation.

[0137] The time-frequency heterogeneous graph feature representation acquisition module is configured to fuse the time-domain feature map representation and the frequency-domain feature map representation through time-frequency heterogeneous graph attention to obtain the time-frequency heterogeneous graph feature representation.

[0138] The attention weight acquisition module is configured to: perform a linear transformation on the time-domain graph feature representation and the frequency-domain graph feature representation, then extract the time-domain graph attention weight and the frequency-domain graph attention weight respectively, and then perform point-by-point addition and fusion of the time-domain graph feature representation and the frequency-domain graph feature representation to extract the time-frequency graph attention weight;

[0139] The weighted graph feature representation acquisition module is configured to: perform weighted fusion of the time-frequency heterogeneous graph feature representations based on the time-domain graph attention weights, frequency-domain graph attention weights, and time-frequency graph attention weights to obtain three weighted graph feature representations;

[0140] The interactive weighted graph feature representation acquisition module is configured to perform heterogeneous fusion and decomposition on the three obtained weighted graph feature representations to obtain three interactive weighted graph feature representations.

[0141] The fake speech decision module is configured to extract the maximum value node and the average value node from the three interactive weighted graph features, concatenate them, and obtain the fake speech detection result based on the single classification loss function.

[0142] The working methods of each module of the system are the same as those provided in Embodiment 1, and will not be repeated here.

[0143] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting forged speech based on graph attention, characterized in that, Includes the following processes: The raw audio data is preprocessed, and then the primary feature vector is obtained through a speech self-supervised model. Based on the primary feature vectors, a high-level feature representation of the original waveform is extracted through a residual coding module with channel attention; Based on the high-level representation of the original waveform, time-domain and frequency-domain feature map representations are constructed through graph computation. The time-domain feature map representation and the frequency-domain feature map representation are fused through time-frequency heterogeneous graph attention to obtain the time-frequency heterogeneous graph feature representation; The time-domain graph feature representation and the frequency-domain graph feature representation are linearly transformed, and then the time-domain graph attention weight and the frequency-domain graph attention weight are extracted respectively. The time-domain graph feature representation and the frequency-domain graph feature representation are then added and fused point by point to extract the time-frequency graph attention weight. Based on the attention weights of the time-domain graph, the attention weights of the frequency-domain graph, and the attention weights of the time-frequency graph, the feature representations of the time-frequency heterogeneous graph are weighted and fused to obtain three weighted graph feature representations. The three weighted graph feature representations are heterogeneously fused and decomposed to obtain three interactive weighted graph feature representations. The maximum value node and the average value node of the three interactive weighted graph features are extracted and concatenated. Based on the single-classification loss function, the fake speech detection result is obtained.

2. The graph attention-based forged speech detection method as described in claim 1, characterized in that, The raw audio data undergoes preprocessing, and then the processed primary feature vector is obtained through a speech self-supervised model, including: The original audio is pre-emphasized, and the pre-emphasized speech data is fed into a self-supervised pre-trained model. The training loss is minimized by the weighted cross-entropy objective function. After passing through a fully connected layer added to the output, the initial feature vector is obtained.

3. The graph attention-based forged speech detection method as described in claim 1, characterized in that, Based on the primary feature vectors, a high-level feature representation of the original waveform is extracted using a residual coding module with channel attention, including: The residual coding module with channel attention is a squeeze-excited residual coding module, which includes six basic units. In each basic unit, batch normalization, SELU activation and two-dimensional convolution are performed twice on the input primary feature vector in sequence. Then, attention is assigned to the channels, the obtained feature vector is added to the recorded primary feature vector and max pooling is used to obtain the feature coding vector. After iteratively combining the input and output of the six basic units, a high-level feature representation of the original speech is obtained.

4. The graph attention-based forged speech detection method as described in claim 1, characterized in that, Based on the high-level representation of the original waveform, time-domain and frequency-domain feature map representations are constructed through graph computation, including: Max pooling is performed on the high-level feature representation to learn separate frequency domain and time domain feature representations. Graph attention mechanisms are then applied to the frequency domain and time domain feature representations respectively to extract time domain feature graph representations and frequency domain feature graph representations.

5. The graph attention-based forged speech detection method as described in claim 1, characterized in that, The time-domain feature map representation and the frequency-domain feature map representation are fused through time-frequency heterogeneous graph attention to obtain the time-frequency heterogeneous graph feature representation, including: The time-domain feature map representation and the frequency-domain feature map representation are projected onto another latent space with a common dimension using a linear transformation, and then fed into the time-frequency heterogeneous graph attention layer to construct a fused time-frequency heterogeneous graph feature representation.

6. The graph attention-based forged speech detection method as described in claim 1, characterized in that, The time-domain and frequency-domain feature representations are linearly transformed, and then the time-domain and frequency-domain attention weights are extracted separately. Finally, the time-domain and frequency-domain feature representations are fused point-by-point to extract the time-frequency map attention weights, including: The obtained time-domain graph feature representation and frequency-domain graph feature representation are linearly transformed to make their number of nodes the same as the dimension of the time-frequency heterogeneous graph. Then, graph attention is applied to the time-domain feature graph representation and frequency-domain feature graph representation respectively to extract the time-domain graph attention weight and frequency-domain graph attention weight. Finally, the time-domain feature graph representation and frequency-domain feature graph representation are added point by point to apply graph attention to extract the time-frequency graph attention weight.

7. The graph attention-based forged speech detection method as described in claim 1, characterized in that, Based on the attention weights of the time-domain graph, the frequency-domain graph, and the time-frequency graph, the feature representations of the time-frequency heterogeneous graph are weighted and fused to obtain three weighted graph feature representations, including: The time-domain graph attention weight, frequency-domain graph attention weight, and time-frequency graph attention weight are applied to the time-frequency heterogeneous graph, respectively, and graph attention and graph pooling are applied to extract the time-domain weighted graph, frequency-domain weighted graph, and time-frequency weighted graph.

8. The graph attention-based forged speech detection method as described in claim 1, characterized in that, The three weighted graph feature representations are heterogeneously fused and decomposed to obtain three interactive weighted graph feature representations, including: An edge, including self-connection, is established between every two nodes in the three weighted graph feature representations to construct a multi-level collaborative heterogeneous weighted fusion graph, where the weight of each edge is learned using a self-attention mechanism; Aggregate the weights of each node and the edges it connects to obtain information about each node's neighboring nodes; Each node information and the aggregated information are projected onto the target dimension through two projection vectors and fused to obtain the feature representation of each new node in the weighted fusion graph, thereby obtaining a multi-level collaborative heterogeneous weighted fusion graph. Batch normalization is applied to the obtained multi-level collaborative heterogeneous weighted fusion graph, and SELU activation is performed. The weighted fusion graph is sliced ​​according to the number of nodes in the three weighted graphs before fusion to extract the interactive weighted graph feature representation.

9. The graph attention-based forged speech detection method as described in claim 1, characterized in that, The maximum and average value nodes of the three interactive weighted graph features are extracted and concatenated. Based on the single-classification loss function, the forged speech detection results are obtained, including: The maximum value node and the average value node of the three interactive weighted graph features are extracted and concatenated to obtain the final feature representation; The final feature representation is fed into a single-classification loss function, and the model is optimized end-to-end to obtain the forged speech classification and detection results.

10. A forged speech detection system based on graph attention, characterized in that, include: The primary feature representation acquisition module is configured to: preprocess the raw audio data and then obtain the processed primary feature vector through a speech self-supervised model; The advanced feature representation acquisition module is configured to extract advanced feature representations of the original waveform from the primary feature vectors using a residual coding module with channel attention. The feature map representation acquisition module is configured to construct time-domain and frequency-domain feature map representations based on the high-level representation of the original waveform through graph computation. The time-frequency heterogeneous graph feature representation acquisition module is configured to: fuse the time-domain feature map representation and the frequency-domain feature map representation through time-frequency heterogeneous graph attention to obtain the time-frequency heterogeneous graph feature representation; The attention weight acquisition module is configured to: perform a linear transformation on the time-domain graph feature representation and the frequency-domain graph feature representation, then extract the time-domain graph attention weight and the frequency-domain graph attention weight respectively, and then perform point-by-point addition and fusion of the time-domain graph feature representation and the frequency-domain graph feature representation to extract the time-frequency graph attention weight; The weighted graph feature representation acquisition module is configured to: perform weighted fusion of the time-frequency heterogeneous graph feature representations based on the time-domain graph attention weights, frequency-domain graph attention weights, and time-frequency graph attention weights to obtain three weighted graph feature representations; The interactive weighted graph feature representation acquisition module is configured to perform heterogeneous fusion and decomposition on the three obtained weighted graph feature representations to obtain three interactive weighted graph feature representations. The fake speech decision module is configured to extract the maximum value node and the average value node from the three interactive weighted graph features, concatenate them, and obtain the fake speech detection result based on the single classification loss function.

Images