Audio forgery detection method and device, computer equipment and storage medium
By constructing a fusion feature map of time-domain and frequency-domain features using graph neural networks and attention mechanisms, the problem of insufficient modeling of cross-domain feature correlation in existing technologies is solved, and high-precision audio forgery detection is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-09
- Publication Date
- 2026-03-31
AI Technical Summary
Existing audio forgery detection methods struggle to fully model the complex relationships between cross-domain features, making it difficult to achieve in-depth mining of forgery traces.
By using graph neural networks and attention mechanisms, we can establish the self-modal and cross-modal correlations of temporal and frequency domain features, construct a fused feature map, and use a classifier to detect audio forgery.
It improves the accuracy and robustness of audio forgery detection, can more fully capture the differences between real and forged speech, explicitly learns the correlation between time-domain nodes and frequency-domain nodes, and focuses on the region with the most significant forgery traces to achieve high-precision and robust detection.
Smart Images
Figure CN121768422A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence technology and voice processing technology, and in particular to an audio forgery detection method, apparatus, computer equipment, and storage medium. Background Technology
[0002] With the widespread dissemination of multimedia content and the rapid development of artificial intelligence technology, the capabilities of audio spoofing technology have been significantly enhanced. Especially in deep learning-driven deepfake audio generation technology, relevant models can generate deepfake audio that mimics the voice of a target speaker with high realism. However, under the manipulation of malicious actors, the generated deepfake audio can be used for various illegal or fraudulent activities, posing a significant threat to cybersecurity and social trust. For example, deepfake audio can be used to impersonate an applicant's voice to forge relevant records for medical insurance authentication, or to impersonate a user's voice to illegally access the bank accounts of voice-protected users, or to impersonate a user's voice for telephone fraud. These behaviors not only seriously infringe upon personal privacy and property security but also pose a serious threat to social order.
[0003] Early audio forgery detection methods primarily relied on frequency domain or time domain features for audio authentication. This involved utilizing the frequency domain characteristics of forged audio, such as short-time Fourier transforms or Mel-frequency cepstral coefficients, to capture statistical differences between forged and genuine speech. Alternatively, they analyzed the time-series characteristics of speech signals, extracting features like short-time energy, zero-crossing rate, and fundamental frequency, and combining these with deep learning models or traditional algorithms for forgery detection. However, forged audio often introduces subtle anomalies in both the time and frequency domains simultaneously, leading to insufficient robustness when using features from only one domain. In recent years, some studies have attempted to concatenate time and frequency domain features into a classifier, but this fusion approach struggles to fully model the complex relationships between cross-domain features. Traditional convolutional networks have limited capabilities in long-term time-series modeling and cross-domain interactions, making it difficult to achieve deep-level detection of forgery traces. Summary of the Invention
[0004] This invention provides an audio forgery detection method, apparatus, computer equipment, and medium to solve the technical problem that existing audio detection technologies are unable to fully model the complex relationships between cross-domain features, making it difficult to achieve in-depth mining of forgery traces.
[0005] Firstly, an audio forgery detection method is provided, including: The audio signal to be detected is preprocessed, and the frequency domain features and time domain features of the preprocessed audio signal are extracted to obtain the time domain feature sequence and the frequency domain feature sequence. Based on the time-domain feature sequence and the frequency-domain feature sequence, the correlation between the self-mode and cross-mode of the time-domain features and the frequency-domain features is established using an attention mechanism in the form of a graph neural network, thereby obtaining a fused feature map of the time-domain features and frequency-domain features. Calculate the global features of the fused feature map based on the features of each node in the fused feature map; A classifier is used to perform audio forgery detection on the global features of the fused feature map.
[0006] Secondly, an audio spoofing detection device is provided, comprising: The preprocessing and feature extraction module is used to preprocess the audio signal to be detected and extract the frequency domain features and time domain features of the preprocessed audio signal to obtain the time domain feature sequence and the frequency domain feature sequence. The graph construction module is used to establish the correlation between the self-mode and cross-mode of the time-domain features and the frequency-domain features using an attention mechanism in a graph neural network manner, based on the time-domain feature sequence and the frequency-domain feature sequence, to obtain a fused feature graph of the time-domain features and frequency-domain features. The feature calculation module is used to calculate the global features of the fused feature map based on the features of each node in the fused feature map; The forgery detection module is used to perform audio forgery detection on the global features of the fused feature map using a classifier.
[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described audio forgery detection method.
[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described audio forgery detection method.
[0009] The aforementioned audio forgery detection method, apparatus, computer equipment, and storage medium can preprocess the audio signal to be detected and extract the frequency domain and time domain features of the preprocessed audio signal to obtain a time domain feature sequence and a frequency domain feature sequence. Based on the time domain and frequency domain feature sequences, a graph neural network is used with an attention mechanism to establish the correlation between the self-modal and cross-modal characteristics of the time domain and frequency domain features, resulting in a fused feature map of the time domain and frequency domain features. The global features of the fused feature map are calculated based on the features of each node in the fused feature map. Finally, a classifier is used to perform audio forgery detection on the global features of the fused feature map. In this invention, this is applied to voice authentication in medical services in the smart healthcare field or to the speech recognition in financial services in the fintech field. When verifying voice access to protected bank accounts, an attention mechanism can be used to model and construct a spectrum based on the self-modal and cross-modal correlations of the temporal and frequency domain features of the audio signal to be detected. Simultaneously, using temporal and frequency domain features to construct the spectrum can more fully capture the differences between real and fake speech. During the construction, the correlation between temporal and frequency domain nodes is explicitly learned through the attention mechanism, which can highlight the potential differences in fake audio and extract the complementary information between the two. This makes the node features in the spectrum a fusion feature of temporal and frequency domain features based on the attention mechanism, so that nodes with more prominent fake features are given higher contribution. This allows the model to focus on the region with the most significant fake traces in the global feature representation, thereby improving the discrimination power and further improving the fake detection effect, achieving high-precision and robust audio fake detection. Attached Figure Description
[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 This is a schematic diagram of an application environment for an audio forgery detection method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an audio forgery detection method according to an embodiment of the present invention; Figure 3 yes Figure 2 A schematic diagram of a specific implementation method for step S20; Figure 4 yes Figure 3 A flowchart illustrating a specific implementation of step S22; Figure 5 yes Figure 3A flowchart illustrating a specific implementation of step S23; Figure 6 This is a schematic diagram of an audio forgery detection device according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0013] The audio forgery detection method provided in this invention can be applied to, for example... Figure 1In application environments such as healthcare and finance, intelligent assistants or intelligent customer service typically operate through a server-side architecture, where the client communicates with the server over a network. The server can acquire the audio to be detected from the client, preprocess the audio signal, and extract its frequency and time domain features to obtain time-domain and frequency-domain feature sequences. Based on these sequences, a graph neural network is used with an attention mechanism to establish the correlation between the self-modal and cross-modal characteristics of the time and frequency features, resulting in a fused feature map. The global features of the fused feature map are calculated based on the features of each node. Finally, a classifier is used to perform audio forgery detection on the global features of the fused feature map. In this invention, the detection is applied to voice authentication of medical cases in the smart healthcare field or voice-protected bank accounts in financial transactions. During voice access verification, an attention mechanism can be used to model the self-modal and cross-modal correlations of the temporal and frequency domain features of the audio signal to be detected. This involves simultaneously constructing a spectrogram using both temporal and frequency domain features, which can more fully capture the differences between real and fake speech. During construction, the correlation between temporal and frequency domain nodes is explicitly learned through the attention mechanism, highlighting potential differences in fake audio and extracting complementary information. This results in node features in the spectrogram being fused features based on the attention mechanism, assigning higher contribution to nodes with more prominent fake features. This allows the model to focus on the regions with the most significant forgery traces in the global feature representation, thereby improving discriminative power and further enhancing the fake-detection effect, achieving high-precision and robust audio forgery detection. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.
[0014] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the audio forgery detection method provided in this embodiment of the invention includes the following steps: S10: Preprocess the audio signal to be detected, and extract the frequency domain features and time domain features of the preprocessed audio signal to obtain the time domain feature sequence and the frequency domain feature sequence.
[0015] The audio forgery detection method provided by this invention can be applied to intelligent customer service or intelligent assistants in various application scenarios such as healthcare, finance, and insurance, and is typically implemented through the server side. For example, in the field of smart healthcare applications, intelligent assistants in medical insurance authentication procedures allow applicants to verify their identity, confirm their application intentions, or understand their health status through voice communication methods such as telephone authentication or remote consultation. For example, the audio could be "medical insurance application," "description of illness," or "health status," etc. Alternatively, in the field of financial applications, intelligent customer service for voice-protected bank accounts allows users to perform information setting and transaction operations on their bank accounts via voice. For example, the audio could be "password reset" or "transfer service," etc. After receiving the audio, the intelligent assistant or intelligent customer service can preprocess the audio signal to be detected and extract the frequency domain and time domain features of the preprocessed audio signal. This invention utilizes both time domain and frequency domain features and extracts complementary information between them through deep modeling, thereby achieving high-precision and robust audio forgery detection.
[0016] In step S10, the audio signal to be detected is preprocessed, specifically by normalizing the input audio signal, removing silence segments, and denoising. For example, firstly, all audio signals to be detected are uniformly sampled and their amplitude is normalized; secondly, in the time domain, silence segments before and after the audio are removed through endpoint detection to reduce invalid frames; and finally, the audio is denoised to improve the ability of subsequent models to focus on valid speech content.
[0017] In some embodiments, after preprocessing the audio signal to be detected, time-domain features and frequency-domain features are mined in parallel from the same preprocessed audio signal to capture the information differences and complementarities of speech in different representation spaces, thereby enhancing the ability to detect forgery.
[0018] Specifically, when extracting temporal features, a one-dimensional convolutional network can be used to capture local waveform morphology, such as short-term energy fluctuations, breakpoints in fake audio, or smoothing distortions. To further capture long-term dependencies, a temporal convolutional network can be added after the convolutional layers to learn cross-frame contextual information. The dilated convolutional structure of the temporal convolutional network can capture long-distance dependencies with relatively low computational cost, ultimately yielding temporal features. When extracting frequency domain features, a short-time Fourier transform (STFT) is performed on the preprocessed audio signal to convert the signal from the time domain to the frequency domain, obtaining a spectrogram. Then, a two-dimensional convolutional network is used to process the spectrogram. The two-dimensional convolutional network can simultaneously capture harmonic structures on the frequency axis and energy changes on the time axis, emphasizing spectral patterns and energy distribution, and can detect harmonic defects and unnatural frequency band energy phenomena.
[0019] S20: Based on the time-domain feature sequence and the frequency-domain feature sequence, establish the correlation between the self-mode and cross-mode of the time-domain features and the frequency-domain features using an attention mechanism in the form of a graph neural network, and obtain a fused feature map of the time-domain features and frequency-domain features.
[0020] In this step, a graph attention network is used to establish a visual correlation between the time and frequency domains to capture local waveform anomalies and discover spectral structure defects.
[0021] Specifically, such as Figure 3 As shown, step S20 includes the following steps S21-S23: S21: Project the time-domain features in the time-domain feature sequence and the frequency-domain features in the frequency-domain feature sequence to the same dimensional space to obtain a time-domain feature sequence and a frequency-domain feature sequence of the same dimension.
[0022] In this embodiment, the time-domain features and frequency-domain features can be uniformly projected onto the same dimension d to obtain the projected time-domain features. and frequency domain features Thus, a temporal feature sequence composed of all the projected temporal features of the audio signal is obtained. and the frequency domain feature sequence composed of all projected frequency domain features ;in, The number of time-domain features in the time-domain feature sequence. The number of frequency domain features in the frequency domain feature sequence. ,and The projection weight matrix, when projected onto the same dimensional space, can improve the feature representation ability and reduce computational complexity.
[0023] S22: Construct a multimodal heterogeneous graph of node association with time-domain features / frequency-domain features based on the correlation between the self-mode and cross-mode of time-domain features in the time-domain feature sequence and the frequency-domain features in the frequency-domain feature sequence of the same dimension.
[0024] In this step, a multimodal heterogeneous spectrum is constructed based on the correlations between time-domain feature self-modes, between frequency-domain feature self-modes, and between time-domain and frequency-domain cross-modes.
[0025] Specifically, such as Figure 4 As shown, step S22 includes the following steps S221-S222: S221: Create time-domain nodes and frequency-domain nodes respectively based on the time-domain feature sequence and frequency-domain feature sequence of the same dimension, and associate each time-domain node and frequency-domain node with time-domain features and frequency-domain features respectively.
[0026] S222: Based on the similarity between time-domain features associated with the time-domain nodes, the similarity between frequency-domain features associated with the frequency-domain nodes, and the feature similarity between the time-domain nodes and the frequency-domain nodes, respectively, establish edges between time-domain nodes, between frequency-domain nodes, and between time-domain nodes and frequency-domain nodes to obtain a multimodal heterogeneous graph.
[0027] In this step, feature similarities (such as cosine similarity) between time-domain nodes, between frequency-domain nodes, and between time-domain nodes and frequency-domain nodes can be calculated to obtain the correlation between time-domain features and frequency-domain features in self-mode and cross-mode. This allows the establishment of edges between different nodes, i.e., connecting different time-domain nodes, connecting different frequency-domain nodes, and connecting time-domain nodes and frequency-domain nodes based on feature similarity (e.g., connecting one time-domain node to its K most similar frequency-domain nodes) to capture cross-modal relationships that are not time-aligned but feature-similar.
[0028] For steps S221-S222 above, the multimodal heterogeneous map includes Nodes, that is, including Each time-domain node and Each frequency domain node is used to construct intramodal edges (including edges between time domain nodes and edges between frequency domain nodes) to maintain the relationship between different nodes within each mode, and intermodal edges (including edges between time domain nodes and edges between frequency domain nodes) to allow direct interaction between time domain features and frequency domain features, thereby obtaining a multimodal heterogeneous graph.
[0029] S23: Calculate the attention weights among all nodes in the multimodal heterogeneous graph using the graph attention mechanism, and perform feature aggregation on all nodes according to the attention weights to update the features associated with each node, thereby obtaining a fused feature graph in which nodes are associated with fused features of the time-domain features and frequency-domain features.
[0030] Specifically, such as Figure 5 As shown, step S23 includes the following steps S231-S233: S231: Construct an adjacency matrix based on the relationships between nodes in the multimodal heterogeneous graph.
[0031] In this step, an adjacency matrix is constructed based on the multimodal heterogeneous graph to describe the connection relationships between nodes.
[0032] S232: Calculate the attention weight between each node and its neighboring nodes using the attention mechanism based on the adjacency matrix.
[0033] Understandably, the adjacency matrix represents the connection relationship between nodes. It records in matrix form whether there is a connection between any two nodes in the multimodal heterogeneous graph (i.e. whether there is an edge between the two nodes). In the adjacency matrix, the neighboring node of node i refers to the node that is directly connected to node i by an edge. For example, if there is an edge between node i and node j, then node j is the neighboring node of node i.
[0034] In this step, based on the connection relationships between the nodes established in the adjacency matrix, the attention weight between each node and each neighboring node is calculated through an attention mechanism.
[0035] Specifically, in this embodiment, based on the connection relationships between nodes in the adjacency matrix, the formula is used... and Calculate the attention weights of each node and its neighboring nodes, where, and These represent the projected features of node i and its neighbor node j, including time-domain features and frequency-domain features, respectively. This represents the attention weight between node i and its neighbor node j, indicating the degree of influence node j has on node i. Indicates splicing characteristics, Indicates the number of nodes in the time domain. Indicates the number of nodes in the frequency domain. This represents a learnable attention vector.
[0036] S233: Based on the attention weights, aggregate the features of neighboring nodes and update the features associated with the current node.
[0037] In this step, after the attention weights are calculated, each node i gathers information from other neighboring nodes j according to the attention weights. The latest representation of each node can be determined by the weighted sum of its neighboring nodes, i.e., using the formula... We aggregate the features of neighboring nodes using a weighted method and update the feature h associated with the current node.
[0038] Understandably, the attention mechanism calculates the correlation between node i and its neighbor node j. The greater the attention, the greater the corresponding weight. By reconstructing the features of nodes through a graph attention network, it is beneficial to highlight possible forgery traces. In the subsequent global feature representation, the focus can be placed on the region where the forgery traces are most significant.
[0039] Regarding steps S231-S233 above, through feature aggregation, nodes not only retain their own features but also integrate information from neighboring nodes to obtain context-enhanced feature representations. The nodes constructed in this invention simultaneously include time-domain and frequency-domain nodes. After aggregation, time-domain features and frequency-domain features can be fused together to form a more robust and discriminative representation, which can capture forgery traces of speech in both the time and frequency domains.
[0040] S30: Calculate the global features of the fused feature map based on the features of each node in the fused feature map.
[0041] In this step, the weight of each node in the fused feature map is calculated based on the features associated with the nodes, and the features associated with all nodes are weighted and summed according to the weights to obtain the global features of the fused feature map.
[0042] Specifically, according to the formula Calculate the weight of each node in the fused feature map. Then according to the formula The global features of the graph are obtained by weighted summation of the node association features. ;in, and These represent the features associated with node i and node k, respectively. Indicates the number of nodes in the time domain. Indicates the number of nodes in the frequency domain. Let W represent the learnable attention vector, and W represent the weight matrix. and Indicates the bias term and weight. .
[0043] In this embodiment, a multilayer perceptron can be used to calculate the weight of each node. Then, by integrating global features based on weights, the importance weights of different nodes can be automatically learned, giving higher contribution to nodes with more prominent forgery features. This allows the focus to be placed on the region with the most significant forgery traces in the global feature representation, thereby improving the discriminative power.
[0044] S40: Use a classifier to perform audio forgery detection on the global features of the fused feature map.
[0045] In this step, the classifier can also be implemented using a multilayer perceptron. The multilayer perceptron performs a nonlinear mapping on the integrated global feature representation, and finally combines the sigmoid activation function to output the probability of whether it is fake audio.
[0046] Understandably, the audio forgery detection method provided by this invention can be applied to intelligent customer service or intelligent assistants in various application scenarios such as medical care, finance, and insurance. For example, in the field of smart medical applications, applicants verify their identity by means of voice communication such as telephone authentication or remote consultation. After receiving the audio related to the identity verification of the applicant, the intelligent assistant can use an attention mechanism to fuse feature maps based on the self-modal and cross-modal correlation of the time-domain and frequency-domain features of the audio signal to be detected. Then, it calculates the global features of the fused feature map based on the features of each node in the fused feature map. Then, it uses a classifier to perform audio forgery detection on the global features of the fused feature map. The intelligent assistant can also perform corresponding operations based on the audio forgery detection results. For example, when the audio forgery detection result is the genuine audio of the applicant, it returns the corresponding voice information according to the stored corresponding operation steps to guide the applicant or user to perform the next voice operation; while when the audio forgery detection result is a deep fake audio, it can issue an alarm and record the corresponding applicant information or user information.
[0047] As can be seen, in the above scheme, when performing voice authentication for medical services in the smart healthcare field or voice access verification for voice-protected bank accounts in the financial field, the attention mechanism can be used to model the self-modal and cross-modal correlations of the temporal and frequency domain features of the audio signal to be detected. That is, the graph is constructed by simultaneously using temporal and frequency domain features, which can more fully capture the differences between real and fake speech. During the construction, the correlation between temporal and frequency domain nodes is explicitly learned through the attention mechanism, which can highlight the potential differences in fake audio and extract the complementary information between the two. This makes the node features in the graph a fusion feature of temporal and frequency domain features based on the attention mechanism, so that nodes with more prominent fake features are given higher contribution. This allows the model to focus on the region with the most significant fake traces in the global feature representation, thereby improving the discrimination power and further improving the fake detection effect, achieving high-precision and robust audio fake detection.
[0048] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0049] In one embodiment, an audio spoofing detection device is provided, which corresponds one-to-one with the audio spoofing detection method described in the above embodiments. For example... Figure 6 As shown, the audio forgery detection device includes a preprocessing and feature extraction module 110, a map construction module 120, a feature calculation module 130, and a forgery detection module 140. Detailed descriptions of each functional module are as follows: The preprocessing and feature extraction module 110 is used to preprocess the audio signal to be detected and extract the frequency domain features and time domain features of the preprocessed audio signal to obtain the time domain feature sequence and the frequency domain feature sequence. The graph construction module 120 is used to establish the correlation between the self-mode and cross-mode of the time-domain features and the frequency-domain features using an attention mechanism in a graph neural network manner based on the time-domain feature sequence and the frequency-domain feature sequence, so as to obtain a fused feature graph of the time-domain features and the frequency-domain features. The feature calculation module 130 is used to calculate the global features of the fused feature map based on the features of each node in the fused feature map; The forgery detection module 140 is used to perform audio forgery detection on the global features of the fused feature map using a classifier.
[0050] In one embodiment, the map construction module 120 is specifically used for: Projecting the time-domain features in the time-domain feature sequence and the frequency-domain features in the frequency-domain feature sequence onto the same dimensional space yields a time-domain feature sequence and a frequency-domain feature sequence of the same dimension. Based on the correlation between the self-mode and cross-mode of the time-domain features in the time-domain feature sequence and the frequency-domain features in the frequency-domain feature sequence of the same dimension, a multimodal heterogeneous map of node association with time-domain features / frequency-domain features is constructed. The attention weights among all nodes in the multimodal heterogeneous graph are calculated using a graph attention mechanism, and features of all nodes are aggregated according to the attention weights to update the features associated with each node, resulting in a fused feature graph in which nodes are associated with fused features of the time-domain and frequency-domain features.
[0051] In one embodiment, the map construction module 120 is further specifically used for: Time-domain nodes and frequency-domain nodes are created based on the time-domain feature sequence and frequency-domain feature sequence of the same dimension, respectively, and each time-domain node and frequency-domain node is associated with the time-domain feature and the frequency-domain feature, respectively. By establishing edges between time-domain nodes, between frequency-domain nodes, and between time-domain nodes and frequency-domain nodes based on the similarity between time-domain features associated with the time-domain nodes, the similarity between frequency-domain features associated with the frequency-domain nodes, and the feature similarity between the time-domain nodes and frequency-domain nodes, respectively, a multimodal heterogeneous graph is obtained.
[0052] In one embodiment, the map construction module 120 is further specifically used for: Construct an adjacency matrix based on the relationships between nodes in the multimodal heterogeneous graph; The attention weight between each node and its neighboring nodes is calculated using the attention mechanism based on the adjacency matrix. The features associated with the current node are updated by weighting and aggregating the features of neighboring nodes based on the attention weights.
[0053] In one embodiment, the map construction module 120 is further specifically used for: Based on the connection relationships between nodes in the adjacency matrix, the formula is used. and Calculate the attention weights of each node and its neighboring nodes, where, and Let i and j represent the characteristics of their neighbor, respectively. This represents the attention weight between node i and its neighbor node j. Indicates splicing characteristics, Indicates the number of nodes in the time domain. Indicates the number of nodes in the frequency domain. This represents a learnable attention vector.
[0054] In one embodiment, the feature calculation module 130 is specifically used for: The weight of each node in the fused feature map is calculated based on the features associated with the nodes, and the features associated with all nodes are weighted and summed according to the weights to obtain the global features of the fused feature map.
[0055] In one embodiment, the feature calculation module 130 is further specifically used for: According to the formula Calculate the weight of each node in the fused feature map. ;in, and These represent the features associated with node i and node k, respectively. Indicates the number of nodes in the time domain. Indicates the number of nodes in the frequency domain. Let W represent the learnable attention vector, and W represent the weight matrix. and This indicates the bias term.
[0056] This invention provides an audio spoofing detection device. It preprocesses the audio signal to be detected and extracts its frequency domain and time domain features to obtain a time domain feature sequence and a frequency domain feature sequence. Based on these time domain and frequency domain feature sequences, it uses a graph neural network with an attention mechanism to establish the correlation between the self-modal and cross-modal characteristics of the time domain and frequency domain features, resulting in a fused feature map. The device then calculates the global features of the fused feature map based on the features of each node. Finally, a classifier is used to perform audio spoofing detection on the global features of the fused feature map. In other words, during operation, it can detect audio spoofing based on the time domain and frequency domain features of the audio signal to be detected. The correlation between self-modal and cross-modal features is modeled using an attention mechanism to construct a spectrum. Simultaneously, the spectrum construction using both temporal and frequency domain features can more fully capture the differences between real and fake speech. During the construction, the correlation between temporal and frequency domain nodes is explicitly learned through the attention mechanism, which can highlight the potential differences in fake audio and extract the complementary information between the two. This makes the node features in the spectrum a fusion feature of temporal and frequency domain features based on the attention mechanism, so that nodes with more prominent fake features are given higher contribution. This allows the model to focus on the region with the most significant fake traces in the global feature representation, thereby improving the discriminative power and further enhancing the fake detection effect, achieving high-precision and robust audio fake detection.
[0057] Specific limitations regarding the audio spoofing detection device can be found in the limitations of the audio spoofing detection method described above, and will not be repeated here. Each module in the aforementioned audio spoofing detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.
[0058] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side audio spoofing detection method.
[0059] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a client-side audio spoofing detection method.
[0060] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: The audio signal to be detected is preprocessed, and the frequency domain features and time domain features of the preprocessed audio signal are extracted to obtain the time domain feature sequence and the frequency domain feature sequence. Based on the time-domain feature sequence and the frequency-domain feature sequence, the correlation between the self-mode and cross-mode of the time-domain features and the frequency-domain features is established using an attention mechanism in the form of a graph neural network, thereby obtaining a fused feature map of the time-domain features and frequency-domain features. Calculate the global features of the fused feature map based on the features of each node in the fused feature map; A classifier is used to perform audio forgery detection on the global features of the fused feature map.
[0061] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: The audio signal to be detected is preprocessed, and the frequency domain features and time domain features of the preprocessed audio signal are extracted to obtain the time domain feature sequence and the frequency domain feature sequence. Based on the time-domain feature sequence and the frequency-domain feature sequence, the correlation between the self-mode and cross-mode of the time-domain features and the frequency-domain features is established using an attention mechanism in the form of a graph neural network, thereby obtaining a fused feature map of the time-domain features and frequency-domain features. Calculate the global features of the fused feature map based on the features of each node in the fused feature map; A classifier is used to perform audio forgery detection on the global features of the fused feature map.
[0062] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0063] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0064] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0065] The above-described embodiments are merely illustrative of the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention. Furthermore, any software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.
Claims
1. An audio deepfake detection method, characterized in that, The method comprises the following steps: preprocessing an audio signal to be detected, and extracting frequency domain features and time domain features of the preprocessed audio signal to obtain a time domain feature sequence and a frequency domain feature sequence; establishing the correlation between the time domain features and the frequency domain features in a self-modal and cross-modal manner by using an attention mechanism in a graph neural network according to the time domain feature sequence and the frequency domain feature sequence, to obtain a fusion feature graph of the time domain features and the frequency domain features; calculating a global feature of the fusion feature graph according to each node feature in the fusion feature graph; using a classifier to perform audio forgery detection on the global feature of the fusion feature graph.
2. The audio deepfake detection method of claim 1, wherein, The method of establishing the correlation between the time domain features and the frequency domain features in a self-modal and cross-modal manner by using an attention mechanism in a graph neural network according to the time domain feature sequence and the frequency domain feature sequence, to obtain a fusion feature graph of the time domain features and the frequency domain features, comprises the following steps: projecting the time domain features in the time domain feature sequence and the frequency domain features in the frequency domain feature sequence into the same dimensional space to obtain a time domain feature sequence and a frequency domain feature sequence in the same dimension; constructing a multi-modal heterogeneous graph with the time domain features / frequency domain features associated with the nodes according to the correlation between the time domain features in the time domain feature sequence and the frequency domain features in the frequency domain feature sequence in a self-modal and cross-modal manner in the same dimension; calculating the attention weights between all nodes in the multi-modal heterogeneous graph by using a graph attention mechanism, and performing feature aggregation on all nodes according to the attention weights to update the features associated with each node, to obtain a fusion feature graph with the time domain features and the frequency domain features associated with the nodes.
3. The audio deepfake detection method of claim 2, wherein, The method of constructing a multi-modal heterogeneous graph with the time domain features / frequency domain features associated with the nodes according to the correlation between the time domain features in the time domain feature sequence and the frequency domain features in the frequency domain feature sequence in a self-modal and cross-modal manner in the same dimension, comprises the following steps: creating time domain nodes and frequency domain nodes according to the time domain feature sequence and the frequency domain feature sequence in the same dimension respectively, and each time domain node and frequency domain node is associated with a time domain feature and a frequency domain feature respectively; establishing edges between the time domain nodes, between the frequency domain nodes, and between the time domain nodes and the frequency domain nodes according to the similarity between the time domain features associated with the time domain nodes, the similarity between the frequency domain features associated with the frequency domain nodes, and the feature similarity between the time domain nodes and the frequency domain nodes, to obtain a multi-modal heterogeneous graph.
4. The audio deepfake detection method of claim 2, wherein, The method of calculating the attention weights between all nodes in the multi-modal heterogeneous graph by using a graph attention mechanism, and performing feature aggregation on all nodes according to the attention weights to update the features associated with each node, comprises the following steps: constructing an adjacency matrix according to the relationship between the nodes in the multi-modal heterogeneous graph; calculating the attention weights of each node and neighbor nodes by using an attention mechanism according to the adjacency matrix; updating the features associated with the current node by weighting and aggregating the features of the neighbor nodes according to the attention weights.
5. The audio deepfake detection method of claim 4, wherein, The method of calculating the attention weights of each node and neighbor nodes by using an attention mechanism according to the adjacency matrix, comprises the following steps: According to the connection relationship between each node in the adjacency matrix, the attention weight of each node and its neighbor node is calculated by formula and , wherein and respectively represent the features of node i and its neighbor node j, represents the attention weight of node i and neighbor node j, represents the spliced features, represents the number of time domain nodes, represents the number of frequency domain nodes, represents the learnable attention vector.
6. The audio deepfake detection method of claim 1, wherein, The global feature of the fusion feature graph is calculated according to each node feature in the fusion feature graph, and the method comprises the following steps: The weight of each node in the fusion feature graph is calculated according to the feature associated with the node, and the global feature of the fusion feature graph is obtained by weighted sum of the features associated with all nodes according to the weight.
7. The audio deepfake detection method of claim 6, wherein, The weight of each node in the fusion feature graph is calculated according to the feature associated with the node, and the global feature of the fusion feature graph is obtained by weighted sum of the features associated with all nodes according to the weight. According to the formula The weight of each node in the fusion feature map is calculated ; wherein and denote the features associated with node i and node k, respectively, denote the number of time domain nodes, denote the number of frequency domain nodes, denote the learnable attention vector, W denotes the weight matrix, and denote the bias term.
8. An audio forgery detection apparatus characterized by comprising: The method comprises the following steps: A pre-processing and feature extraction module is configured to pre-process the audio signal to be detected, extract the frequency domain feature and the time domain feature of the pre-processed audio signal, and obtain the time domain feature sequence and the frequency domain feature sequence; A graph construction module is configured to establish the correlation between the self-modalities and the cross-modalities of the time domain feature and the frequency domain feature by using the attention mechanism in the graph neural network manner according to the time domain feature sequence and the frequency domain feature sequence, and obtain the fusion feature graph of the fusion of the time domain feature and the frequency domain feature; A feature calculation module is configured to calculate the global feature of the fusion feature graph according to each node feature in the fusion feature graph; A forgery detection module is configured to perform audio forgery detection on the global feature of the fusion feature graph by using a classifier.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the audio forgery detection method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to realize the steps of the audio forgery detection method according to any one of claims 1 to 7.