Audio detection method, device and equipment, computer storage medium and program product

Abstract

The invention discloses an audio detection method, device and equipment, a computer storage medium and a program product. The method comprises the following steps: acquiring various feature vectors corresponding to an audio to be detected; for each feature vector, features in the feature vector are aggregated through a graph attention mechanism to obtain a graph network corresponding to each feature vector, and the graph network corresponding to each feature vector comprises at least two of a text feature graph network, an emotion feature graph network and a sound feature graph network; combining the graph networks corresponding to each feature vector to obtain a plurality of graph network groups; for each graph network group, fusing features in the graph network group through a heterogeneous graph fusion mechanism to obtain a heterogeneous graph corresponding to the graph network group; and determining an audio detection result according to the graph network corresponding to each feature vector and the heterogeneous graph corresponding to each graph network group. According to the embodiment of the invention, the recognition precision of the forged audio can be improved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence security technology, and in particular relates to an audio detection method, device, equipment, computer storage medium and program product. Background Technology

[0002] With the rapid development of the field of Artificial Intelligence Generated Content (AIGC), the barrier to audio production has been significantly lowered. This trend has not only facilitated the acquisition and dissemination of high-quality audio content, but has also provided an opportunity for the widespread dissemination of fake audio information.

[0003] To address the potential risks posed by fake audio, audio detection technology has emerged. This technology can extract features from audio and use these features to identify fake audio; however, the accuracy of this method for identifying fake audio is relatively low.

[0004] Therefore, improving the accuracy of identifying fake audio is a technical problem that relevant technicians urgently need to solve. Summary of the Invention

[0005] This application provides an audio detection method, apparatus, device, computer storage medium, and program product that can improve the accuracy of identifying fake audio.

[0006] In a first aspect, embodiments of this application provide an audio detection method, comprising: acquiring multiple feature vectors corresponding to the audio to be detected, wherein the multiple feature vectors include at least two of text feature vectors, emotion feature vectors, and sound feature vectors; for each feature vector, aggregating the features in the feature vector through a graph attention mechanism to obtain a graph network corresponding to each feature vector, wherein the graph network corresponding to each feature vector includes at least two of text feature graph networks, emotion feature graph networks, and sound feature graph networks; combining the graph networks corresponding to each feature vector to obtain multiple graph network groups; for each graph network group, fusing the features in the graph network group through a heterogeneous graph fusion mechanism to obtain a heterogeneous graph corresponding to the graph network group; and determining the audio detection result based on the graph network corresponding to each feature vector and the heterogeneous graph corresponding to each graph network group.

[0007] In one implementation, for each feature vector, features in the feature vector are aggregated using a graph attention mechanism to obtain a graph network corresponding to each feature vector. This includes: for each feature vector, using multiple features from that feature vector as features corresponding to each node in an initial graph network; calculating the strength relationships between nodes in the initial graph network based on the features corresponding to each node; calculating the aggregation information corresponding to each node in the initial graph network based on the strength relationships between nodes and the features corresponding to each node; performing a dual projection mapping on each node based on the aggregation information and the features corresponding to that node to obtain target features; and updating the initial graph network based on the target features corresponding to each node to obtain the graph network corresponding to the feature vector.

[0008] In one embodiment, the graph network group includes at least a first graph network and a second graph network, wherein the first graph network is any one of a text feature graph network, an emotion feature graph network, and an audio feature graph network, and the second graph network is one of the text feature graph network, emotion feature network, and audio feature graph network that is different from the first graph network; for each graph network group, the features in the graph network group are fused through a heterogeneous graph fusion mechanism to obtain a heterogeneous graph corresponding to the graph network group, including: transforming the features corresponding to each node in the first graph network according to a preset first transformation matrix to obtain a transformed first graph network, wherein the preset first transformation matrix is ​​determined according to the features corresponding to the nodes of the first graph network; transforming the features corresponding to each node in the second graph network according to a preset second transformation matrix to obtain a transformed second graph network, wherein the preset second transformation matrix is ​​determined according to the features corresponding to the nodes of the second graph network; and connecting each node in the transformed first graph network with each node in the transformed second graph network to obtain a heterogeneous graph.

[0009] In one implementation, the audio detection result is determined based on the graph network corresponding to each feature vector and the heterogeneous graph corresponding to each graph network group. This includes: for each graph network group, calculating the global features corresponding to the heterogeneous graph using the features in the heterogeneous graph; grouping the graph network corresponding to each feature vector, the heterogeneous graph corresponding to each graph network group, and the global features corresponding to each heterogeneous graph to obtain multiple cue graph groups; for each cue graph group, fusing the maximum and average values ​​of the features in the cue graph group to obtain fused cue features; and inputting the fused cue features corresponding to each cue graph group into the audio detection neural network to determine the audio detection result.

[0010] In one implementation, the features in the clue graph group include features corresponding to the graph network or features corresponding to the heterogeneous graph. For each clue graph group, the maximum value and average value of the features in the clue graph group are fused to obtain fused clue features, including: for each graph network or each heterogeneous graph in the clue graph group, calculating the score corresponding to each node based on the features corresponding to each node in the graph network or heterogeneous graph and the weight parameters corresponding to that node; filtering the nodes in the graph network or heterogeneous graph based on the scores corresponding to each node to obtain filtered nodes; calculating the pooling information of the filtered nodes based on the features of the filtered nodes and the weights corresponding to the filtered nodes; and fusing the maximum value and average value of the pooling information to obtain the features corresponding to the graph network or the fused clue features corresponding to the heterogeneous graph.

[0011] In one embodiment, obtaining multiple feature vectors corresponding to the audio to be detected includes: inputting the audio to be detected into a text feature recognition model, extracting text features corresponding to the audio to be detected through the text feature recognition model to obtain a text feature vector; inputting the audio to be detected into a timbre feature recognition model, capturing timbre features corresponding to the audio to be detected through the timbre feature recognition model, and determining emotional feature vectors and sound feature vectors through timbre features.

[0012] In one embodiment, the audio to be detected is input into a timbre feature recognition model. The timbre feature recognition model captures the timbre features corresponding to the audio to be detected, and determines the emotional feature vector and the voice feature vector based on the timbre features. This includes: performing feature separation on the timbre features to obtain the second timbre feature by minimizing the mutual information between the first timbre feature and the second timbre feature, wherein the first timbre feature contains preset sensitive information, and the second timbre feature does not contain preset sensitive information; and determining the emotional feature vector and the voice feature vector based on the second timbre feature.

[0013] In one implementation, the audio detection results also include the name of the algorithm used to create the fake audio, as well as the scores for the graph network and the heterogeneous graph.

[0014] Secondly, embodiments of this application provide an audio detection device, the device comprising: The acquisition module is used to acquire multiple feature vectors corresponding to the audio to be detected. The multiple feature vectors include at least two of the following: text feature vectors, emotion feature vectors, and sound feature vectors. The first determining module is used to aggregate the features in each feature vector through a graph attention mechanism to obtain a graph network corresponding to each feature vector. The graph network corresponding to each feature vector includes at least two of the following: text feature graph network, sentiment feature graph network, and voice feature graph network. The second determining module is used to combine the graph networks corresponding to each feature vector to obtain multiple graph network groups; The fusion module is used to fuse the features in each graph network group through a heterogeneous graph fusion mechanism to obtain the corresponding heterogeneous graph for that graph network group. The detection module is used to determine the audio detection result based on the graph network corresponding to each feature vector and the heterogeneous graph corresponding to each graph network group.

[0015] Thirdly, embodiments of this application provide an audio detection device, the device including: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the audio detection method in the first aspect or any embodiment of the first aspect.

[0016] Fourthly, a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the audio detection method of the first aspect or any embodiment of the first aspect.

[0017] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform an audio detection method as described in the first aspect or any embodiment of the first aspect.

[0018] The audio detection method, apparatus, device, computer storage medium, and program product of this application acquire at least two of the following: text feature vector, emotion feature vector, and sound feature vector corresponding to the audio to be detected. For each feature vector, a graph attention mechanism is used to aggregate the features in the feature vector to obtain a graph network corresponding to each feature vector. Further, the graph networks corresponding to each feature vector are combined to obtain multiple graph network groups. For each graph network group, a heterogeneous graph fusion mechanism is used to fuse the features in the graph network group to obtain a heterogeneous graph corresponding to the graph network group. The audio detection result is determined based on the graph network corresponding to each feature vector and the heterogeneous graph corresponding to each graph network group. It is understood that this application achieves the ability to capture audio forgery clues from multiple dimensions by integrating multiple modal information such as sound, emotion, and text. This multimodal fusion method significantly improves the accuracy and generalization ability of forgery detection. Furthermore, by utilizing graph attention and heterogeneous graph fusion mechanisms to perform detection within each graph and between different graphs, multi-angle cross-validation of the audio to be detected is achieved, improving the audio detection accuracy. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating an embodiment of the audio detection method provided in this application is shown. Figure 2 A schematic diagram of the process for generating a graph network according to an embodiment of this application is shown; Figure 3 A schematic diagram of the architecture of an initial text feature map network provided in one embodiment of this application is shown; Figure 4 This illustration shows a schematic diagram of the aggregation information corresponding to the i-th node provided in one embodiment of this application; Figure 5 A schematic diagram of the process for generating heterogeneous graphs according to an embodiment of this application is shown; Figure 6 A schematic diagram of the process for determining audio detection results according to an embodiment of this application is shown; Figure 7 A schematic diagram of the architecture of a master node provided in one embodiment of this application is shown; Figure 8 A flowchart illustrating an embodiment of the audio detection method provided in this application is shown. Figure 9 A flowchart illustrating the process of determining emotion feature vectors and voice feature vectors according to an embodiment of this application is shown; Figure 10 A schematic diagram of the architecture of an audio detection method provided in one embodiment of this application is shown; Figure 11 A flowchart illustrating an embodiment of the audio detection method provided in this application is shown. Figure 12 This is a schematic diagram of the structure of an audio detection device provided in another embodiment of this application; Figure 13 This is a schematic diagram of the structure of an audio detection device provided in another embodiment of this application. Detailed Implementation

[0021] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0023] With the rapid development of the AIGC (AI-Generated Content) field, the production speed of audio content has significantly accelerated, and the barrier to audio production has been greatly lowered. This trend has not only facilitated the acquisition of high-quality audio content but has also provided opportunities for the spread of fake audio information. Audio deepfake technology utilizes advanced machine learning and deep learning algorithms to generate highly realistic audio content, even mimicking the voice, tone, and emotional expression of specific individuals. While audio deepfake technology has certain application value in entertainment, education, and other fields, it also carries corresponding potential risks. For example, fake audio could be used for malicious activities such as spreading misinformation, causing serious harm to people.

[0024] Therefore, to address the challenges posed by deepfake audio, deepfake detection technology has emerged. The goal of deepfake detection is to identify and distinguish between genuine and fake audio through technical means, thereby protecting the public from being misled by false information. Among related technologies, feature extraction from audio can be used to identify fake audio; however, this method has a relatively low accuracy rate in identifying fake audio.

[0025] Therefore, improving the accuracy of identifying fake audio is a technical problem that relevant technicians urgently need to solve.

[0026] To address the problems of the prior art, embodiments of this application provide an audio detection method, apparatus, device, computer storage medium, and program product. The audio detection method provided in this application embodiment will be described first below.

[0027] Figure 1 A flowchart illustrating an embodiment of the audio detection method provided in this application is shown. Figure 1 As shown, audio detection includes the following steps: S110. Obtain multiple feature vectors corresponding to the audio to be detected. The multiple feature vectors include at least two of the following: text feature vector, emotion feature vector, and sound feature vector.

[0028] S120. For each feature vector, the features in the feature vector are aggregated through a graph attention mechanism to obtain the graph network corresponding to each feature vector.

[0029] S130. Combine the graph networks corresponding to each feature vector to obtain multiple graph network groups.

[0030] S140. For each graph network group, the features in the graph network group are fused through the heterogeneous graph fusion mechanism to obtain the corresponding heterogeneous graph of the graph network group.

[0031] S150. Determine the audio detection result based on the graph network corresponding to each feature vector and the heterogeneous graph corresponding to each graph network group.

[0032] In some embodiments, in S110, the audio to be detected can be acquired, and features of the audio to be detected can be extracted in multiple dimensions to obtain multiple feature vectors corresponding to the audio to be detected in multiple dimensions.

[0033] For example, the feature dimensions for feature extraction of the audio to be detected may include at least two of the three dimensions of text, emotion, and sound, thereby obtaining at least two of the following: text feature vector, emotion feature vector, and sound feature vector.

[0034] In one example, features can be extracted from the audio to be detected in three dimensions: text, emotion, and sound, resulting in text feature vectors, emotion feature vectors, and sound feature vectors.

[0035] For example, text feature vectors can be used to represent the text content in the audio to be detected. Emotion feature vectors can be used to characterize the emotions present in the audio to be detected, such as happiness, sadness, etc. Sound features can be used to characterize the acoustic features of the audio to be detected, such as the pitch, loudness, and environment in which the audio to be detected is located.

[0036] In some embodiments, in S120, for each feature vector, the features in the feature vector are aggregated using a graph attention mechanism to obtain a graph network corresponding to each feature vector. The graph network corresponding to each feature vector includes at least two of the following: a text feature graph network, a sentiment feature graph network, and a voice feature graph network.

[0037] For example, text feature vectors, sentiment feature vectors, and audio feature vectors can be aggregated using an Intra-view Graph Attention Mechanism (IGAM) to obtain the corresponding graph networks for each. That is, the graph network is used to optimize independent, flat feature vectors into structured features that can represent the internal structure and key information of the features.

[0038] The graph network obtained through the graph attention mechanism corresponds to the feature vector input to the graph attention mechanism. For example, inputting a text feature vector into the graph attention mechanism yields a text feature graph network corresponding to the text feature vector. Similarly, for sentiment and voice feature vectors, inputting them into the graph attention mechanism respectively yields a sentiment feature graph network corresponding to the sentiment feature vector and a voice feature graph network corresponding to the voice feature vector.

[0039] For example, each feature vector can include multiple features; for instance, a text feature vector can include multiple text features. Graph attention mechanisms can be used to aggregate multiple features in a feature vector to obtain a graph network that reflects the strength relationships between features.

[0040] In some alternative embodiments, Figure 2 A schematic flowchart illustrating a generative graph network provided in one embodiment of this application is shown. Figure 2 As shown, generating a graph network includes the following steps S121-S125: S121. For each feature vector, use multiple features in the feature vector as the features corresponding to each node in the initial graph network.

[0041] For example, for each feature vector, an initial graph network corresponding to that feature vector can be constructed. The number of nodes in the initial graph network can be determined based on the number of features in the feature vector, and the feature corresponding to each node in the initial graph network can be a feature from the feature vector.

[0042] In one example, for a text feature vector, an initial text feature graph network can be constructed. The number of nodes in the initial text feature graph network is determined by the number of features in the text feature vector, and the features of each node in the initial text feature graph network are the features in the text feature vector. Similarly, the process of constructing corresponding graph networks for sentiment feature vectors and voice feature vectors will not be repeated in this application.

[0043] S122. Based on the characteristics of each node, calculate the strength relationship between each node in the initial graph network.

[0044] For example, for each node in the initial graph network, the strength of the relationship between the node and other nodes is calculated based on the features corresponding to the node in the initial graph network.

[0045] In one example, the text feature vector can be represented as ,in, It includes multiple text features A text feature initial graph network is constructed based on the text feature vectors. The text feature initial graph network includes... Each node. Furthermore, the strength relationship between nodes in the initial graph network of text features can be calculated using the following formula (1):

[0046] in, The attention weights are used to represent the i-th node. With the j-th node The strength of the relationship between them. Indicates element-wise multiplication. It is a learnable mapping that adjusts the weights of relationships within the initial graph network through dot products.

[0047] Furthermore, the attention weights can be normalized using the softmax function shown in formula (2) to obtain the normalized attention weights:

[0048] in, is the normalized attention weight between node i and node j. τ is a temperature parameter used to adjust the softmax distribution, while This represents the set of neighboring nodes of node i, including node i itself.

[0049] The normalized attention weights can be used as the strength relationship between nodes in the initial graph network of text features, and the initial graph network of text features can be constructed by combining the text features in the text feature vector.

[0050] S123. Calculate the aggregation information corresponding to each node in the initial graph by using the strength relationship between each node and the features corresponding to each node.

[0051] For example, for each node in the initial graph network, feature aggregation can be performed on the node based on the strength relationship between the nodes and the features corresponding to each node, thereby obtaining the aggregation information corresponding to the node.

[0052] In one example, feature aggregation can be performed on nodes using the normalized attention weights and the features corresponding to each node. The aggregation information for the i-th node is shown below. It can be represented by the following formula (3):

[0053] S124. For each node, perform dual projection mapping based on the aggregation information and features corresponding to that node to obtain the target features.

[0054] For example, for each node, a dual projection mapping can be performed based on the aggregation information and features corresponding to that node to obtain the target features corresponding to that node.

[0055] Different weight matrices can be used to project the aggregate information and features corresponding to the nodes.

[0056] In one example, for the i-th node in the initial graph network of text features, the target features corresponding to the i-th node can be calculated using the following formula (4). :

[0057] Where SELU(·) is the activation function, and the weight matrix is... Used for projecting aggregated information. Used to project the original features.

[0058] S125. Update the initial graph network according to the target features corresponding to each node to obtain the graph network corresponding to the feature vector.

[0059] For example, the features of each node in the initial graph network are updated using the target features corresponding to each node, thereby obtaining the graph network corresponding to the feature vector.

[0060] In one example, in the initial graph network for text features, the target features corresponding to each node can be represented as follows: And by updating the features of each node in the initial graph network of text features through the target features, that is, updating... Thus, the updated text feature map network can be represented as .

[0061] Below, in conjunction with Figure 3 , Figure 4 The following examples illustrate text feature map networks. Figure 3 A schematic diagram of the architecture of an initial text feature map network provided in one embodiment of this application is shown; Figure 4 This diagram illustrates the aggregation information corresponding to the i-th node according to an embodiment of this application. For example... Figure 3 As shown, in the initial text feature map network, each feature in the text feature vector can be used as a node in the initial text feature map network. That is, the initial text feature map network can include 4 nodes, and the corresponding features can be respectively... Furthermore, the strength relationships between nodes can be calculated using formulas (1) and (2). For each node, aggregation information can be calculated using formula (3), such as... Figure 4 As shown, the aggregation information corresponding to the i-th node can be represented as Furthermore, the target features corresponding to each node can be calculated using the above formula (4). This is used to update the initial text feature map network. This yields a text feature map network.

[0062] Figure 2 In the illustrated embodiment, for each feature vector, multiple features from that feature vector are used as the features corresponding to each node in the initial graph network, and the strength relationships between each node in the initial graph network are calculated. This allows for direct calculation of the strength relationships between nodes from the feature vector, automatically constructing the optimal graph topology and more accurately capturing the complex relationships inherent in various data. Furthermore, based on the strength relationships between nodes and the features corresponding to each node, the aggregation information corresponding to each node in the initial graph network is calculated, and a dual-projection mapping is performed based on the aggregation information and the features corresponding to that node to obtain the target feature. By aggregating the features of surrounding nodes and updating node features using dual-projection mapping, the representation of nodes in the graph network is effectively enriched. That is, the aggregation operation incorporates local contextual information into each node, while dual-projection balances the importance of the node's own features and the information of neighboring nodes, enhancing semantic representation while avoiding the problem of node features becoming too similar.

[0063] In some embodiments, in S130, the text feature map network, the emotion feature map network, and the voice feature map network can be combined to obtain multiple graph network groups.

[0064] In one example, when the feature vectors include text feature vectors, emotion feature vectors, and audio feature vectors, the graph network group can include graph network groups with various combinations. For example, the graph network group can include a TE network group built from text feature graph networks and emotion feature graph networks, a TA network group built from text feature graph networks and audio feature graph networks, an AE network group built from emotion feature graph networks and audio feature graph networks, and an ATE network group built from text feature graph networks, emotion feature graph networks, and audio feature graph networks.

[0065] Furthermore, the TA network group can be used to distinguish between artificially matched content in fake audio and natural alignment in genuine audio. The AE network group can be used to evaluate the relationship between the emotional tone of a sound and its expected emotional context, helping to detect deceptive audio lacking natural emotional consistency. The TE network group can be used to evaluate the consistency between text content and related emotions; it helps identify artifacts in fake audio where emotional cues may not match the given text content. The ATE network group can be used to evaluate the consistency between sound, text, and emotion; it helps identify artifacts arising from mismatches among these three elements in fake audio. The evaluation of the graph network group can be achieved by calculating the correlation of the graph networks within the network group.

[0066] In some embodiments, in S140, for each graph network group, the features in the graph network group are fused through a heterogeneous graph fusion mechanism to obtain the heterogeneous graph corresponding to the graph network group.

[0067] For example, the Heterogeneous Graph Fusion Module (HGFM) can capture the complex interactions between graph networks in a graph network group to identify consistency between graph networks and achieve cross-modal semantic alignment and information enhancement.

[0068] The heterogeneous graph fusion mechanism can fuse the features between graph networks in a graph network group and use the fused features to construct the heterogeneous graph corresponding to the graph network group.

[0069] In some alternative embodiments, Figure 5 A schematic diagram illustrating the process of generating heterogeneous graphs according to an embodiment of this application is shown. Figure 5 As shown, generating the heterogeneous graph includes the following steps S131-S133: S131. Based on the preset first transformation matrix, transform the features corresponding to each node in the first graph network to obtain the transformed first graph network.

[0070] The preset first transformation matrix is ​​determined based on the features corresponding to the nodes in the first graph network. In one example, each element in the preset first transformation matrix can be obtained through model training.

[0071] For example, each graph network group includes at least a first graph network and a second graph network. The first graph network is any one of a text feature graph network, an emotion feature graph network, and an audio feature graph network, and the second graph network is one of the text feature graph network, the emotion feature network, and the audio feature graph network that is different from the first graph network.

[0072] For example, the features corresponding to each node in the first graph network can be transformed by the first transformation matrix, and the features corresponding to each node in the first graph network can be updated using the transformed features, thereby obtaining the transformed first graph network.

[0073] S132. Based on the preset second transformation matrix, the features corresponding to each node in the second graph network are transformed to obtain the transformed second graph network. The preset second transformation matrix is ​​determined based on the features corresponding to the nodes of the second graph network.

[0074] For example, the features corresponding to each node in the Derogatory graph network can be transformed using the second transformation matrix, and the features corresponding to each node in the second graph network can be updated using the transformed features, thereby obtaining the transformed second graph network.

[0075] For example, the preset second transformation matrix is ​​determined based on the features corresponding to the nodes in the second graph network. In one example, each element in the preset second transformation matrix can be obtained through model training.

[0076] S133. Connect each node in the transformed first graph network with each node in the transformed second graph network to obtain a heterogeneous graph.

[0077] For example, each graph network group can correspond to a heterogeneous graph. This heterogeneous graph can be obtained by connecting the nodes of the transformed first graph network and the nodes of the transformed second graph network in the corresponding network group.

[0078] Heterogeneous graphs can be used to represent complex cross-modal associations and collaborative semantics. In other words, heterogeneous graphs can capture deep information that cannot be revealed by a single graph network, thereby supporting more robust and accurate multimodal semantic understanding.

[0079] In one example, a heterogeneous TA graph is constructed using TA network groups. Let me explain. First, we can use a sound feature map network. (i.e., the first graph network) and the text feature network graph The nodes of the second graph network are interconnected to construct the initial TA heterogeneous graph. Furthermore, the TA heterogeneity diagram is analyzed using the following formula (5). The i-th node in The transformation is performed to obtain the transformed TA heterogeneous graph. .

[0080]

[0081] in, and They represent and The set of nodes, according to the different types of nodes, are to be transformed using the first transformation matrix. Second transformation matrix .

[0082] Figure 5 In the illustrated embodiment, a first transformation matrix is ​​used to transform the features corresponding to each node in the first graph network to obtain a transformed first graph network. Then, a second transformation matrix is ​​used to transform the features corresponding to each node in the second graph network to obtain a transformed second graph network. Further, the nodes in the transformed first graph network are connected to the nodes in the transformed second graph network to obtain a heterogeneous graph. This constructs a heterogeneous graph corresponding to the first and second graph networks, preserving their structures while achieving fine-grained cross-modal interaction.

[0083] In some embodiments, in S150, the audio detection result is determined based on the graph network corresponding to each feature vector and the heterogeneous graph corresponding to each graph network group.

[0084] For example, the graph network corresponding to each feature vector and the heterogeneous input multilayer perceptron (MLP) function corresponding to each graph network group can be used to fuse the features of the graph network and the heterogeneous graph to obtain the audio detection result.

[0085] Audio detection results can be used to characterize whether the audio being detected is fake. For example, the audio detection result can include whether the audio being detected is fake, or it can include whether the audio being detected is real.

[0086] In some alternative embodiments, Figure 6 A schematic diagram illustrating the flowchart for determining audio detection results according to an embodiment of this application is shown. Figure 6 As shown, determining the audio detection result includes the following steps S151-S154: S151. For each heterogeneous graph corresponding to a graph network group, calculate the global features corresponding to the heterogeneous graph using the features in the heterogeneous graph.

[0087] For example, the global features corresponding to a heterogeneous graph can be used to characterize the global information corresponding to the heterogeneous graph.

[0088] In one example, a master node can be set in a heterogeneous graph, which captures global information by interacting with multiple other nodes in the graph. For example, Figure 7 Figure 7 illustrates the architecture of a master node according to an embodiment of this application. By interacting with other nodes, the average amount of information can be obtained. and weight parameters Among them, the average information content It can be calculated using the following formula (6):

[0089] in, Indicates the average information content. It is a picture The number of nodes in the system.

[0090] S152. Group the graph network corresponding to each feature vector, the heterogeneous graph corresponding to each graph network group, and the global feature corresponding to each heterogeneous graph to obtain multiple cue graph groups.

[0091] For example, a clue graph group can be obtained by grouping graph networks, heterogeneous graphs, and global features. The clue graph group includes at least one of graph networks, heterogeneous graphs, and global features.

[0092] For example, each cue graph group may consist of a first cue including a graph network, a second cue including a heterogeneous graph, and a third cue including global features.

[0093] In one example, the Grouped Readout Scheme (GRS) can be used to group the graph network corresponding to each feature vector, the heterogeneous graph corresponding to each graph network group, and the global features corresponding to each heterogeneous graph, resulting in multiple cue graph groups. For example, a cue graph group can be represented as U={G, H, M}, where G represents cue captured from graph network nodes, H represents cue captured from heterogeneous graph nodes, and M represents the set of master nodes in the heterogeneous graph, representing global cue. This yields K cue graph groups. For example, a cue graph group can include... in, ={G}; ={H}; ={M}.

[0094] S153. For each clue group, the maximum value and average value of the features in the clue group are fused to obtain the fused clue features.

[0095] For example, for each clue in each clue group, the maximum value and average value of the feature corresponding to the clue can be determined, and the maximum value and average value corresponding to all clues in the clue group can be fused to obtain the fused clue feature.

[0096] In some optional embodiments, the features in the clue graph group include features corresponding to the graph network or features corresponding to the heterogeneous graph. For each graph network or each heterogeneous graph in the clue graph group, a score is calculated for each node based on its features and weight parameters. Based on the scores of each node, the nodes in the graph network or heterogeneous graph are filtered to obtain filtered nodes. Pooling information is calculated for the filtered nodes based on their features and weights. The maximum value and the average value of the pooling information are fused to obtain the features corresponding to the graph network or the fused clue features corresponding to the heterogeneous graph.

[0097] For example, when the clues are graph networks or heterogeneous graphs, the score corresponding to a node can be calculated based on the features corresponding to each node in the graph network or heterogeneous graph and the weight parameters corresponding to that node.

[0098] In one example, the score corresponding to a node can be calculated using the following formula (7):

[0099] Among them, the j-th graph in the k-th group can be represented as ;in, Include Each node. Furthermore, a learnable weight parameter can be used. Calculated by dot product The score corresponding to each node .

[0100] For example, nodes in a graph network or heterogeneous graph can be filtered based on the score corresponding to each node to obtain the filtered nodes. That is, nodes in a graph network or heterogeneous graph can be sorted according to the size of the score, and the sorted nodes can be filtered using a preset number P as the filtering condition. For example, the first P nodes can be used as the filtered nodes.

[0101] Furthermore, for the filtered nodes, the pooling information of the filtered nodes can be calculated based on their characteristics and corresponding weights.

[0102] In one example, the pooling information corresponding to the graph network or heterogeneous graph can be calculated using the following formula (8):

[0103] in, This refers to pooling information corresponding to graph networks or heterogeneous graphs. and Let represent the p-th node and its corresponding score.

[0104] Furthermore, the maximum value and average value of the pooling information corresponding to each node in the graph network or heterogeneous graph can be fused to obtain the features corresponding to the graph network or the fused cue features corresponding to the heterogeneous graph.

[0105] In one example, the pooling information of each graph in the clue can be concatenated using the following formula (9) to obtain the fused clue features corresponding to each graph:

[0106] in, For the kth group Pooling information corresponding to each graph; It is a series operation; and These represent the node maximization and averaging operation functions, respectively.

[0107] In this embodiment, by determining the features and weight parameters corresponding to each node in a graph network or heterogeneous graph, a score is calculated for that node. Based on the scores of each node, nodes in the graph network or heterogeneous graph are filtered to obtain filtered nodes. Then, based on the features and weights of the filtered nodes, pooling information is calculated for the filtered nodes, thereby achieving pooling of the graph network or heterogeneous graph and realizing information compression and dimensionality reduction. Furthermore, the maximum value and average value of the pooling information are fused to obtain the corresponding fusion cue features in the graph network or heterogeneous graph, enabling the capture of features of the graph network or heterogeneous graph from different aspects and improving the content of the fusion cue features.

[0108] S154. Input the fused clue features corresponding to each clue group into the audio detection neural network to determine the audio detection result.

[0109] For example, for each clue graph group, the fused clue features corresponding to all clues can be spliced ​​together, and the spliced ​​fused clue features can be input into the audio detection neural network to determine the audio detection result.

[0110] In one example, the concatenated fused cue features can be represented as: .

[0111] In the audio detection neural network, MLP can be used as the decision function to classify the spliced ​​fused cue features and output the audio detection results.

[0112] Figure 6 In the illustrated embodiment, for each heterogeneous graph corresponding to a graph network group, global features corresponding to that heterogeneous graph are calculated using the features in that heterogeneous graph. The graph network, heterogeneous graph, and global features are then grouped to obtain multiple clue graph groups. For each clue graph group, the maximum and average values ​​of the features in that group are fused to obtain fused clue features. The fused clue features corresponding to each clue graph group are input into the audio detection neural network to determine the audio detection result. It can be understood that the features in the heterogeneous graph are used to calculate the global features corresponding to that heterogeneous graph, thereby macroscopically establishing the overall coordination between the graph networks corresponding to the heterogeneous graphs, providing a general detection direction for audio inspection. Furthermore, by grouping the graph network, heterogeneous graph, and global features to construct clue graph groups, information is avoided from being mixed together indiscriminately, thus preventing mutual interference between clues. Furthermore, by using the maximum and average values ​​of features in the clue graph group to reflect the most prominent features and the overall coordination of features, and fusing them with the maximum and average values, this ensures that during audio detection, neither an extreme, local forgery trace is missed, nor is the overall coordination ignored due to a local anomaly, thus making a more comprehensive judgment.

[0113] In some alternative embodiments, the audio detection results may also include the name of the algorithm used to create the fake audio, as well as the scores for the graph network and the heterogeneous graph.

[0114] For example, if it is determined that the audio to be detected is fake audio, the name of the algorithm used to create the fake audio can also be determined. For example, the audio detection result can be expressed as: the audio to be detected is fake audio, which was created using algorithm X.

[0115] For example, the audio detection results may also include scores for graph networks and scores for heterogeneous graphs.

[0116] The scores of graph networks and heterogeneous graphs can provide interpretive evidence for audio detection results. For example, the main basis for determining that the audio to be detected is fake audio is that the acoustic feature map network detected a high anomaly (score of 0.95), which highly matches the known features of 'Algorithm X'. However, the text sentiment feature map network performed normally (score of 0.10), indicating that the forgery may only target acoustic characteristics.

[0117] Furthermore, when constructing corresponding graph networks and heterogeneous graphs through graph attention mechanisms and heterogeneous graph fusion mechanisms, the contribution of different graphs can be adjusted by the scores of the graph network and the heterogeneous graphs. When fusing through multilayer perception mechanisms, different graph fusion strategies can be formulated based on the scores of different graphs.

[0118] In this embodiment, the name of the algorithm used to create the fake audio, along with the scores for the graph network and the heterogeneous graph, are displayed in the audio detection results. That is, by identifying the fake audio algorithm, the source of the attack algorithm is traced, providing direction for improving corresponding defense strategies. Furthermore, by displaying the graph network score and the heterogeneous graph score, the confidence in the decision-making process is quantified, allowing for dynamic adjustment of the weights of different graphs based on the scores, thereby achieving more accurate detection.

[0119] Figure 1 In the illustrated embodiment, at least two of the following are obtained: text feature vector, emotion feature vector, and sound feature vector corresponding to the audio to be detected. For each feature vector, a graph attention mechanism is used to aggregate the features in the feature vector to obtain a graph network corresponding to each feature vector. Further, the graph networks corresponding to each feature vector are combined to obtain multiple graph network groups. For each graph network group, a heterogeneous graph fusion mechanism is used to fuse the features in the graph network group to obtain a heterogeneous graph corresponding to that graph network group. The audio detection result is determined based on the graph network corresponding to each feature vector and the heterogeneous graph corresponding to each graph network group. It is understood that this application achieves the ability to capture audio forgery clues from multiple dimensions by integrating multiple modal information such as sound, emotion, and text. This multimodal fusion method significantly improves the accuracy and generalization ability of forgery detection. Furthermore, by utilizing graph attention and heterogeneous graph fusion mechanisms to perform detection within each graph and between different graphs, multi-angle cross-validation of the audio to be detected is achieved, improving the audio detection accuracy.

[0120] To avoid privacy leaks, as another implementation of this application, this application also provides another implementation of the audio detection method, as detailed in the following embodiments.

[0121] Figure 8 A flowchart illustrating an embodiment of the audio detection method provided in this application is shown. Figure 8 As shown, the audio detection method includes the following steps S810-S820: S810. Input the audio to be detected into the text feature recognition model, and extract the text features corresponding to the audio to be detected through the text feature recognition model to obtain the text feature vector.

[0122] For example, the text feature recognition model can be a pre-trained model used to extract text features from the audio to be detected.

[0123] In one example, the text feature recognition model can extract text features from the audio to be detected using Automatic Speech Recognition (ASR).

[0124] S820: Input the audio to be detected into the timbre feature recognition model, capture the timbre features corresponding to the audio to be detected through the timbre feature recognition model, and determine the emotional feature vector and the sound feature vector through the timbre features.

[0125] For example, the timbre feature recognition model can be a pre-trained model used to extract emotional features and vocal features from the audio to be detected.

[0126] The timbre feature recognition model can capture the timbre features in the audio to be detected, and further, determine the emotional features and voice features corresponding to the audio to be detected through the timbre features.

[0127] Furthermore, in some optional embodiments, the timbre feature may include a first timbre feature and a second timbre feature. The first timbre feature includes preset sensitive information, while the second timbre feature does not. Further, the timbre feature can be separated to obtain the second timbre feature by minimizing the mutual information between the first and second timbre features. Further, an emotional feature vector and a vocal feature vector are determined based on the second timbre feature.

[0128] For example, the preset sensitive information may be information that could potentially lead to privacy breaches. For instance, the preset sensitive information may include acoustic features such as voiceprints. Furthermore, the preset sensitive information can be pre-set by a technician.

[0129] For example, the mutual information (MI) between the first timbre feature and the second timbre feature can be used to measure the amount of information shared between them. That is, if the first timbre feature and the second timbre feature are independent of each other, they have no shared information, and the mutual information is 0.

[0130] It is understood that in this embodiment of the application, by minimizing the mutual information between the first timbre feature and the second timbre feature, it is ensured that the separated second timbre feature does not contain preset sensitive information, thereby avoiding privacy leakage.

[0131] The following is combined Figure 9 The following examples illustrate the audio detection method.

[0132] Figure 9This illustration shows a flowchart of determining emotion feature vectors and voice feature vectors according to an embodiment of this application. Figure 9 As shown, in step S901, MFCC features can be extracted from the original audio.

[0133] In one example, the original speech data z can be input into the timbre feature recognition model, and the Mel-Frequency Cepstral Coefficients (MFCC) feature, i.e., the timbre feature, can be extracted through the timbre feature recognition model.

[0134] In step S902, the MFCC features can be separated to obtain a first feature vector containing the sensitive elements. and the second feature vector that does not contain sensitive elements .

[0135] In one example, a multi-head attention mechanism was added during the feature vector extraction process. The multi-head attention mechanism obtains a set of weights through the following formula (10), so that the model can pay more attention to the feature information related to the current task, while ignoring information that is irrelevant to the task.

[0136]

[0137] in, This represents the attention weight between the i-th query (Q) vector and the j-th key (K) vector in a specific attention head. The dimension of the key vector is represented by the number of elements in the vector. Prevent To prevent gradient explosion or vanishing problems caused by excessively large values, exp(.) converts the dot product result to a positive number, allowing normalization via the softmax function. h represents the number of attention heads.

[0138] Multi-head attention mechanisms concatenate the outputs of all attention heads and multiply them by a weight matrix. As shown in the following formula (11), the final output vector is obtained, where, It is a trainable weight matrix; V represents the eigenvalues.

[0139]

[0140] After the above steps The output is a feature vector for each attribute, by and express.

[0141] Furthermore, in S903, the first eigenvector is minimized. With the second eigenvector The mutual information between the two elements is used as the target for iteration to obtain the first feature vector after separation. The second eigenvector after separation .

[0142] In one example, the first eigenvector is minimized by minimizing mutual information (MI). With the second eigenvector The correlation between them, so as to separate the second feature vector as much as possible. The mutual information (MI) between x and y is represented as follows: The calculation formula (12) is shown below:

[0143] The goal of minimizing mutual information (MI) is achieved by minimizing formula (12), thereby reducing the correlation between features. Furthermore, the total mutual information loss can be expressed as formula (13):

[0144] Where V, E, and A represent the voiceprint, emotion, and audio attributes in the original speech data, respectively. I(V, E), I(E, A), and I(V, A) represent the mutual information (MI) calculations between these three pairs of attributes.

[0145] Understandable, Figure 8 In the illustrated embodiment, incorporating privacy protection during the feature extraction stage of the audio to be detected enhances user trust in how their data is processed. Adding a privacy protection mechanism at this stage prevents unauthorized access or leakage of data during subsequent processing, thereby improving overall data security. Furthermore, integrating the privacy protection mechanism with feature extraction improves the efficiency and effectiveness of data processing.

[0146] The following is combined Figure 10 , Figure 11 The following examples illustrate the audio detection method.

[0147] Figure 10 A schematic diagram of the architecture of an audio detection method provided in one embodiment of this application is shown; Figure 11 A flowchart illustrating an embodiment of the audio detection method provided in this application is shown.

[0148] like Figure 10As shown, the audio detection method may include a feature extraction module 1010, which may include a text feature extraction module for extracting text feature vectors; the feature extraction module may also include a timbre feature extraction module for extracting emotional features and voice features. Figure 11 The step S1101 shown involves obtaining text features by transing the voice recordings of a call or conference through a text feature extraction module, and determining emotional and vocal features through a timbre feature extraction module.

[0149] Furthermore, such as Figure 10 As shown, the audio detection method includes a multi-view detection module 1020, which comprises an in-view aggregation module 1021, a cross-view aggregation module 1022, and a multi-view fusion module 1023. Figure 11 In step S1102, within the view aggregation module, each feature can be aggregated using IGAM to obtain the graph network corresponding to each feature. In step S1103, across the view aggregation module, multiple graph networks can be fused using HGFM to obtain multiple heterogeneous graphs. In step S1104, in the multi-view fusion module, group fusion is performed using GRS, and the fused features are input into an MLP function to obtain audio detection results. These audio detection results may include a judgment on whether the call or conference audio is fake audio, the name of the algorithm used to create the fake audio, and scores for the graph network and the heterogeneous graph.

[0150] Based on the audio detection method provided in the above embodiments, this application also provides a specific implementation of the audio detection device 1200. Please refer to the following embodiments.

[0151] First see Figure 12 The audio detection device provided in this application includes the following modules: The acquisition module 1201 is used to acquire multiple feature vectors corresponding to the audio to be detected. The multiple feature vectors include at least two of the following: text feature vectors, emotion feature vectors, and sound feature vectors. The first determining module 1202 is used to aggregate the features in the feature vector through a graph attention mechanism for each feature vector to obtain a graph network corresponding to each feature vector. The graph network corresponding to each feature vector includes at least two of the following: text feature graph network, emotion feature graph network and sound feature graph network. The second determining module 1203 is used to combine the graph networks corresponding to each feature vector to obtain multiple graph network groups; The fusion module 1204 is used to fuse the features in each graph network group through a heterogeneous graph fusion mechanism to obtain the corresponding heterogeneous graph for the graph network group. The detection module 1205 is used to determine the audio detection result based on the graph network corresponding to each feature vector and the heterogeneous graph corresponding to each graph network group.

[0152] In one embodiment, the first determining module 1202 aggregates the features in each feature vector using a graph attention mechanism to obtain a graph network corresponding to each feature vector in the following manner: For each feature vector, multiple features in the feature vector are used as features corresponding to each node in the initial graph network; the strength relationship between each node in the initial graph network is calculated based on the features corresponding to each node; the aggregation information corresponding to each node in the initial graph network is calculated based on the strength relationship between each node and the features corresponding to each node; for each node, a dual projection mapping is performed based on the aggregation information and the features corresponding to the node to obtain the target feature; the initial graph network is updated based on the target feature corresponding to each node to obtain the graph network corresponding to the feature vector.

[0153] In one embodiment, the graph network group includes at least a first graph network and a second graph network, wherein the first graph network is any one of a text feature graph network, an emotion feature graph network, and an audio feature graph network, and the second graph network is one of the text feature graph network, emotion feature network, and audio feature graph network that is different from the first graph network; the second determining module 1203 uses the following method to fuse the features in each graph network group through a heterogeneous graph fusion mechanism to obtain a heterogeneous graph corresponding to the graph network group: according to a preset first transformation matrix, the features corresponding to each node in the first graph network are transformed to obtain a transformed first graph network, the preset first transformation matrix being determined according to the features corresponding to the nodes in the first graph network; according to a preset second transformation matrix, the features corresponding to each node in the second graph network are transformed to obtain a transformed second graph network, the preset second transformation matrix being determined according to the features corresponding to the nodes in the second graph network; and each node in the transformed first graph network is connected to each node in the transformed second graph network to obtain a heterogeneous graph.

[0154] In one embodiment, the detection module 1205 determines the audio detection result based on the graph network corresponding to each feature vector and the heterogeneous graph corresponding to each graph network group as follows: For the heterogeneous graph corresponding to each graph network group, the global feature corresponding to the heterogeneous graph is calculated using the features in the heterogeneous graph; the graph network corresponding to each feature vector, the heterogeneous graph corresponding to each graph network group, and the global feature corresponding to each heterogeneous graph are grouped to obtain multiple cue graph groups; for each cue graph group, the maximum value and average value of the features in the cue graph group are fused to obtain fused cue features; the fused cue features corresponding to each cue graph group are input into the audio detection neural network to determine the audio detection result.

[0155] In one embodiment, the features in the clue graph group include features corresponding to the graph network or features corresponding to the heterogeneous graph. The detection module 1205 fuses the maximum and average values ​​of the features in each clue graph group to obtain fused clue features in the following manner: For each graph network or heterogeneous graph in the clue graph group, the score corresponding to each node is calculated based on the features corresponding to each node in the graph network or heterogeneous graph and the weight parameters corresponding to that node; the nodes in the graph network or heterogeneous graph are filtered based on the scores corresponding to each node to obtain filtered nodes; the pooling information of the filtered nodes is calculated based on the features of the filtered nodes and the weights corresponding to the filtered nodes; the maximum value and the average value of the pooling information are fused to obtain the features corresponding to the graph network or the fused clue features corresponding to the heterogeneous graph.

[0156] In one embodiment, the acquisition module 1201 acquires multiple feature vectors corresponding to the audio to be detected in the following manner: the audio to be detected is input into a text feature recognition model, and the text features corresponding to the audio to be detected are extracted through the text feature recognition model to obtain a text feature vector; the audio to be detected is input into a timbre feature recognition model, and the timbre features corresponding to the audio to be detected are captured through the timbre feature recognition model, and the emotional feature vector and the voice feature vector are determined through the timbre features.

[0157] In one embodiment, the acquisition module 1201 inputs the audio to be detected into the timbre feature recognition model in the following manner, captures the timbre features corresponding to the audio to be detected through the timbre feature recognition model, and determines the emotional feature vector and the voice feature vector through the timbre features: the timbre features are separated to obtain the second timbre feature by minimizing the mutual information between the first timbre feature and the second timbre feature, wherein the first timbre feature contains preset sensitive information, and the second timbre feature does not contain preset sensitive information; the emotional feature vector and the voice feature vector are determined based on the second timbre feature.

[0158] In one implementation, the audio detection results also include the name of the algorithm used to create the fake audio, as well as the scores for the graph network and the heterogeneous graph.

[0159] Figure 13 A schematic diagram of the hardware structure of the audio detection device provided in an embodiment of this application is shown.

[0160] The audio detection device may include a processor 1301 and a memory 1302 storing computer program instructions.

[0161] Specifically, the processor 1301 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0162] Memory 1302 may include mass storage for data or instructions. For example, and not limitingly, memory 1302 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 1302 may include removable or non-removable (or fixed) media. Where appropriate, memory 1302 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 1302 is non-volatile solid-state memory.

[0163] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.

[0164] The processor 1301 implements any of the audio detection methods described in the above embodiments by reading and executing computer program instructions stored in the memory 1302.

[0165] In one example, the audio detection device may also include a communication interface 1303 and a bus 1310. For example, Figure 13 As shown, the processor 1301, memory 1302, and communication interface 1303 are connected through bus 1310 and complete communication with each other.

[0166] The communication interface 1303 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0167] Bus 1310 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 1310 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0168] The audio detection device can execute the audio detection method in the embodiments of this application based on the audio to be detected, thereby achieving a combination of Figure 1 The method described is for audio detection.

[0169] Furthermore, in conjunction with the audio detection methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the audio detection methods in the above embodiments.

[0170] This application also provides a computer program product, including a computer program, which, when executed, implements any of the audio detection methods described in the above embodiments.

[0171] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0172] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0173] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0174] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0175] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. An audio detection method, characterized in that, include: Obtain multiple feature vectors corresponding to the audio to be detected, wherein the multiple feature vectors include at least two of the following: text feature vectors, emotion feature vectors, and sound feature vectors; For each of the aforementioned feature vectors, the features in the feature vectors are aggregated through a graph attention mechanism to obtain a graph network corresponding to each feature vector. The graph network corresponding to each feature vector includes at least two of the following: a text feature graph network, an emotion feature graph network, and a sound feature graph network. The graph networks corresponding to each feature vector are combined to obtain multiple graph network groups; For each graph network group, the features in the graph network group are fused through a heterogeneous graph fusion mechanism to obtain the corresponding heterogeneous graph for the graph network group; The audio detection result is determined based on the graph network corresponding to each feature vector and the heterogeneous graph corresponding to each graph network group.

2. The method according to claim 1, characterized in that, The step of aggregating features in each feature vector using a graph attention mechanism to obtain a graph network corresponding to each feature vector includes: For each feature vector, multiple features in that feature vector are used as the features corresponding to each node in the initial graph network; Based on the characteristics corresponding to each node, the strength relationship between each node in the initial graph network is calculated respectively; Based on the strength relationships between the nodes and the features corresponding to each node, the aggregation information corresponding to each node in the initial graph network is calculated. For each node, a dual projection mapping is performed based on the aggregation information and features corresponding to that node to obtain the target features; The initial graph network is updated based on the target features corresponding to each node to obtain the graph network corresponding to the feature vector.

3. The method according to claim 1, characterized in that, The graph network group includes at least a first graph network and a second graph network, wherein the first graph network is any one of the text feature graph network, the emotion feature graph network, and the voice feature graph network, and the second graph network is one of the text feature graph network, the emotion feature network, and the voice feature network that is different from the first graph network; For each graph network group, the features in the graph network group are fused through a heterogeneous graph fusion mechanism to obtain the corresponding heterogeneous graph, including: According to a preset first transformation matrix, the features corresponding to each node in the first graph network are transformed to obtain a transformed first graph network. The preset first transformation matrix is ​​determined according to the features corresponding to the nodes of the first graph network. According to a preset second transformation matrix, the features corresponding to each node in the second graph network are transformed to obtain the transformed second graph network. The preset second transformation matrix is ​​determined according to the features corresponding to the nodes of the second graph network. The heterogeneous graph is obtained by connecting each node in the transformed first graph network with each node in the transformed second graph network.

4. The method according to claim 1, characterized in that, The step of determining the audio detection result based on the graph network corresponding to each feature vector and the heterogeneous graph corresponding to each graph network group includes: For each heterogeneous graph corresponding to a graph network group, the global features corresponding to the heterogeneous graph are calculated using the features in the heterogeneous graph. The graph network corresponding to each feature vector, the heterogeneous graph corresponding to each graph network group, and the global feature corresponding to each heterogeneous graph are grouped to obtain multiple cue graph groups; For each of the clue graph groups, the maximum value and average value of the features in the clue graph group are fused to obtain fused clue features; The fused cue features corresponding to each cue graph group are input into the audio detection neural network to determine the audio detection result.

5. The method according to claim 4, characterized in that, The features in the clue graph group include the features corresponding to the graph network or the features corresponding to the heterogeneous graph; For each of the clue graph groups, the maximum and average values ​​of the features in the clue graph group are fused to obtain fused clue features, including: For each graph network or heterogeneous graph in the clue graph group, the score corresponding to the node is calculated based on the features corresponding to each node in the graph network or heterogeneous graph and the weight parameters corresponding to the node. Based on the score corresponding to each node, the nodes in the graph network or the heterogeneous graph are filtered to obtain the filtered nodes; Based on the characteristics of the filtered nodes and the weights corresponding to the filtered nodes, the pooling information of the filtered nodes is calculated. The maximum value and the average value of the pooling information are fused to obtain the feature corresponding to the graph network or the fused cue feature corresponding to the heterogeneous graph.

6. The method according to claim 1, characterized in that, The acquisition of multiple feature vectors corresponding to the audio to be detected includes: The audio to be detected is input into a text feature recognition model, and the text feature recognition model extracts the text features corresponding to the audio to be detected to obtain the text feature vector. The audio to be detected is input into the timbre feature recognition model, which captures the timbre features corresponding to the audio to be detected and determines the emotional feature vector and the voice feature vector based on the timbre features.

7. The method according to claim 6, characterized in that, The step of inputting the audio to be detected into the timbre feature recognition model, capturing the timbre features corresponding to the audio to be detected through the timbre feature recognition model, and determining the emotion feature vector and the voice feature vector through the timbre features includes: The timbre features are separated to obtain the second timbre feature by minimizing the mutual information between the first timbre feature and the second timbre feature. The first timbre feature contains preset sensitive information, while the second timbre feature does not contain preset sensitive information. The emotional feature vector and the voice feature vector are determined based on the second timbre feature.

8. The method according to claim 1, characterized in that, The audio detection results also include the name of the algorithm used to create fake audio, as well as the score of the graph network and the score of the heterogeneous graph.

9. An audio detection device, characterized in that, The device includes: The acquisition module is used to acquire multiple feature vectors corresponding to the audio to be detected, wherein the multiple feature vectors include at least two of the following: text feature vectors, emotion feature vectors, and sound feature vectors. The first determining module is used to aggregate the features in each feature vector through a graph attention mechanism to obtain a graph network corresponding to each feature vector. The graph network corresponding to each feature vector includes at least two of the following: a text feature graph network, an emotion feature graph network, and a sound feature graph network. The second determining module is used to combine the graph networks corresponding to each feature vector to obtain multiple graph network groups; The fusion module is used to fuse the features in each graph network group through a heterogeneous graph fusion mechanism to obtain the heterogeneous graph corresponding to the graph network group. The detection module is used to determine the audio detection result based on the graph network corresponding to each feature vector and the heterogeneous graph corresponding to each graph network group.

10. An audio detection device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the audio detection method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the audio detection method as described in any one of claims 1-8.

12. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the audio detection method as described in any one of claims 1-8.

Images