Counterfeit speech recognition method and device, electronic equipment and computer storage medium

Abstract

The invention relates to a forged voice recognition method and device, electronic equipment and a computer storage medium, and belongs to the technical field of voice security, and the forged voice recognition method comprises the steps: extracting a time domain feature map and a frequency domain feature map of an audio signal of a to-be-recognized voice; fusing the time domain feature graph and the frequency domain feature graph based on a heterogeneous stacked graph attention mechanism to obtain a feature fusion graph, and extracting a global feature vector of the feature fusion graph; and determining the authenticity of the to-be-recognized speech based on a mapping relationship between the feature vector and the authenticity of the to-be-recognized speech by using a feature classifier comprising a plurality of hidden layers. According to the invention, the authenticity identification precision of the forged voice can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of voice security, and in particular to a fake voice recognition method and device, electronic equipment and computer storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, speech generation technology based on deep neural networks has made breakthrough progress. Text-to-speech and speech conversion technologies can generate highly realistic synthetic speech, which is almost indistinguishable from human speech in terms of sound quality, naturalness and personalized features.

[0003] Existing audio deep fake detection technology mainly relies on artificially designed acoustic features such as mel-frequency cepstral coefficients and linear predictive cepstral coefficients, combined with support vector machines and random forests for detection. It is difficult to capture dynamic time-domain anomalies in AI-generated speech, and the generalization ability of manual features is limited. When faced with new fake technologies (such as speech generated by diffusion models), the detection accuracy decreases significantly.

[0004] Therefore, the prior art cannot accurately identify fake speech. SUMMARY

[0005] Therefore, it is necessary to provide a fake voice recognition method, device, electronic equipment and computer storage medium to solve the problem that the prior art cannot accurately identify fake speech.

[0006] To solve the above problems, in a first aspect, the present application provides a fake voice recognition method, comprising: extracting a time-domain feature map and a frequency-domain feature map of an audio signal of the voice to be identified; fusing the time-domain feature map and the frequency-domain feature map based on a heterogeneous stack graph attention mechanism to obtain a feature fusion map, and extracting a global feature vector of the feature fusion map; using a feature classifier containing multiple hidden layers to determine the authenticity of the voice to be identified based on the mapping relationship between the feature vector and the authenticity of the voice to be identified.

[0007] In one possible implementation, the time-domain feature map and the frequency-domain feature map of the audio signal of the voice to be identified are extracted, comprising: extracting the time-domain feature and the frequency-domain feature of the audio signal of the voice to be identified through a convolution module; constructing the time-domain feature map and the frequency-domain feature map of the audio signal based on the time-domain feature and the frequency-domain feature using a feature map construction function, wherein the feature map construction function is:

[0008]

[0009] wherein, is a frequency domain feature map, is a time domain feature map, is a time domain feature, is a frequency domain feature, is a pooling operation combined with an attention mechanism, is a max operation, abs is an average operation.

[0010] In a possible implementation, the time domain feature map and the frequency domain feature map are fused based on a heterogeneous stacked graph attention mechanism to obtain a feature fusion graph, including: connecting nodes in the time domain feature map and the frequency domain feature map and adding a stacked node, the nodes in the time domain feature map and the frequency domain feature map are connected to the stacked node to obtain an intermediate feature fusion graph including a plurality of node pairs; calculating attention weights between the node pairs based on types of the node pairs by using different projection methods to obtain the feature fusion graph including the attention weights.

[0011] In a possible implementation, after obtaining the feature fusion graph, including: constructing an adjacency matrix of nodes in the feature fusion graph, the adjacency matrix is used to represent connection relationships between the nodes in the feature fusion graph and attention weights of the connection relationships between the nodes; updating node features in the feature fusion graph based on the connection relationships between the nodes and the attention weights of the connection relationships between the nodes in the adjacency matrix.

[0012] In a possible implementation, a global feature vector of the feature fusion graph is extracted, including: calculating global attention weights of the nodes in the feature fusion graph by using a preset global attention weight calculation formula, the global attention weight calculation formula is:

[0013] wherein, is a global attention weight of an i th node in the feature fusion graph, is a preset activation function, is a feature vector of the i th node in the feature fusion graph, j is a j th neighbor node of the i th node in the feature fusion graph, is a neighbor node set of the i th node in the feature fusion graph, is a feature vector of the j th neighbor node of the i th node in the feature fusion graph, W is a learnable weight matrix, is a transpose of an attention weight vector of a connection relationship between the i th node and the j th neighbor node in the feature fusion graph; The global feature vector of the feature fusion graph is calculated based on the node feature vectors of each node and the global attention weights of each node. The formula for calculating the global feature vector is as follows:

[0014] in, is the global feature vector of the feature fusion graph, and N is the number of nodes in the feature fusion graph.

[0015] In one possible implementation, a feature classifier comprising multiple hidden layers is used to determine the authenticity of the speech to be recognized based on the mapping relationship between feature vectors and the authenticity of the speech to be recognized, including: Multiple hidden layers are used to extract high-level features from the feature vector, and fully connected layers are used to connect the high-level features to obtain the feature output vector; An activation function is used to calculate the probability that the speech to be identified belongs to real speech or fake speech based on the feature output vector, and the authenticity of the speech to be identified is determined based on the probability.

[0016] In one possible implementation, the probability is calculated using the following formula:

[0017] in, Let be the probability that the speech to be recognized belongs to the i-th speech type. The unnormalized score for the speech to be recognized to belong to the i-th speech type. The speech to be recognized belongs to the first... k Unnormalized scores for various speech types K The total number of voice types; Determining the authenticity of a speech object based on probability includes: The speech category with the highest probability and greater than a preset threshold is used as the result of identifying whether the speech is genuine or fake.

[0018] Secondly, the present invention also provides a device for spoofing voice recognition, comprising: The feature map extraction module is used to extract the time-domain feature map and frequency-domain feature map of the audio signal of the speech to be recognized; The feature map fusion module is used to fuse the time-domain feature map and the frequency-domain feature map based on the heterogeneous stacked graph attention mechanism to obtain a feature fusion map, and extract the global feature vector of the feature fusion map; The classification and recognition module is used to determine the authenticity of the speech to be recognized based on the mapping relationship between the feature vector and the authenticity of the speech to be recognized by a feature classifier containing multiple hidden layers.

[0019] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein, The memory is configured to store a program. The processor is coupled to the memory and is configured to execute the program stored in the memory to implement the steps in the method for recognizing counterfeit speech according to any one of the embodiments.

[0020] In a fourth aspect, the present application provides a computer readable storage medium configured to store a program or instructions, which, when executed by a processor, can implement the steps in the method for recognizing counterfeit speech according to any one of the embodiments.

[0021] The method for recognizing counterfeit speech provided by the present application can focus on the time domain features and the frequency domain features of the audio signal of the speech to be recognized, fuse the time domain feature map and the frequency domain feature map based on a heterogeneous stacked graph attention mechanism, obtain a feature fusion map, effectively fuse the differences between the time domain features and the frequency domain features of the audio signal, improve the accuracy of feature extraction of the audio signal, extract a global feature vector of the feature fusion map, determine the authenticity of the speech to be recognized based on the mapping relationship between the feature vector and the authenticity of the speech to be recognized by using a feature classifier containing multiple hidden layers, focus on the global features of the feature fusion map, and identify more mapping relationships between the feature vector and the authenticity of the speech to be recognized by using the classifier with multiple hidden layers, so that the authenticity of the speech to be recognized is more accurately identified. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 A flowchart of a method for recognizing counterfeit speech provided by an embodiment of the present application; Figure 2 A flowchart of a feature fusion method provided by an embodiment of the present application; Figure 3 A feature fusion diagram provided by an embodiment of the present application; Figure 4 A flowchart of a feature updating method provided by an embodiment of the present application; Figure 5 A flowchart of a possible implementation method of S103 provided by an embodiment of the present application; Figure 6A structural schematic diagram of a counterfeit speech recognition device provided by an embodiment of the present application is shown. Figure 7 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] The preferred embodiments of the present application will be described in detail with reference to the drawings, in which the same or similar components are denoted by the same reference numerals, and the principles of the present application are explained with the embodiments of the present application, which are not intended to limit the scope of the present application.

[0025] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more. The association relationship of the associated objects is described by "and / or", which means that there can be three relationships, for example: A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone.

[0026] The "first", "second", and the like described in the embodiments of the present application are only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the technical features limited by "first" and "second" can explicitly or implicitly include at least one of the features.

[0027] In this document, the term "embodiment" means that the specific features, structures, or characteristics described in connection with the embodiment can be included in at least one embodiment of the present application. The phrase appears at various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily independent or alternative embodiments to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0028] One specific embodiment of the present application, as shown in Figure 1 A counterfeit speech recognition method is disclosed, comprising: S101, extracting the time domain feature map and the frequency domain feature map of the audio signal of the speech to be recognized.

[0029] In the embodiment of the present application, the counterfeit voice recognition method provided can be realized through a voice detection model. The present application mainly describes the working principle of the voice detection model. The voice detection model needs to be trained before use. The data set used for training can first obtain real audio, extract the timbre features of the real audio, and synthesize counterfeit audio. The synthesized audio is highly similar to the real audio in terms of sound quality and timbre. During the synthesis process, the real audio needs to be denoised. An adaptive filtering algorithm is used to remove background noise and interference signals in the audio. After obtaining the synthesized audio, the synthesized audio is normalized. The volume, amplitude and other features of the audio are unified to a specific range to ensure the consistency and stability of the data. For the synthesized audio used for model training, data augmentation is also performed. Through random speed, pitch, and adding different types of noise operations on the audio, the diversity of the counterfeit voice data is expanded, the richness and complexity of the data are increased, and the effect and generalization ability of subsequent model training are improved. Specifically, a plurality of individuals with different gender, age, region and accent backgrounds can be selected as the voice collection objects. These individuals record voices in a plurality of different scenarios, covering a wide range of daily communication situations. In terms of daily language scenarios, common social and life scenarios such as family gatherings, work meetings, friend chats, shopping and consumption are simulated. In the poetry recitation scenario, participants are required to read classic poems of different styles and different ages, including Tang poetry, Song lyrics, and modern poetry, to reflect the diversity of voice in literary expression. In order to ensure that the collected voice data is representative, a plurality of different types of recording devices are used for recording. These devices include professional recording microphones, smart phone built-in recording functions, tablet computer recording applications, and some common portable recording devices. Different recording devices have different audio characteristics and sampling rates, which helps to simulate the use of various devices for voice interaction in real life. These audio data are processed in depth, including loading audio data, extracting relevant features, and performing preprocessing operations such as data cleaning, labeling and sorting, so that they meet the input requirements of the subsequent model. Then the data is divided into training set, validation set and test set, which are used for model training, parameter tuning and performance evaluation respectively.

[0030] In the embodiment of the present application, when the trained voice detection model is used to identify the authenticity of the voice to be identified, the time domain feature map and the frequency domain feature map of the audio signal of the voice to be identified need to be extracted first. For example, the Fourier algorithm is used to convert the audio signal to obtain the time domain signal and the frequency domain signal, and then the time domain feature map of the time domain signal and the frequency domain feature map of the frequency domain signal are extracted.

[0031] In the embodiment of the present application, when the trained voice detection model is used to identify the authenticity of the voice to be identified, the time domain feature map and the frequency domain feature map of the audio signal of the voice to be identified need to be extracted first. For example, the Fourier algorithm is used to convert the audio signal to obtain the time domain signal and the frequency domain signal, and then the time domain feature map of the time domain signal and the frequency domain feature map of the frequency domain signal are extracted.

[0032] In the embodiment of the present application, after the time domain feature map and the frequency domain feature map of the audio signal of the to-be-identified speech are obtained, the time domain feature map and the frequency domain feature map need to be fused to obtain a feature fusion map, considering the difference between the time domain feature map and the frequency domain feature map. Specifically, the time domain feature map and the frequency domain feature map can be fused by a heterogeneous stacked graph attention mechanism. The heterogeneous stacked graph attention mechanism can fuse and unify the node information of the feature map, and can process the cross-domain (time domain and frequency domain) relationship between the nodes of the feature map. The specific fusion process of the time domain feature map and the frequency domain feature map will be described in detail later in the present application. After the feature fusion map is obtained, the global feature vector of the fusion feature map needs to be extracted, considering the global feature of the to-be-identified speech, which is used to represent the global feature of the audio signal of the to-be-identified speech. The extraction method of the global feature vector will be described in detail later in the present application.

[0033] S103, determining the authenticity of the to-be-identified speech based on the mapping relationship between the feature vector and the authenticity of the to-be-identified speech by using a feature classifier containing multiple hidden layers.

[0034] In the embodiment of the present application, after the global feature vector of the fusion feature map is extracted, the global feature vector is taken as the input of the feature classifier containing multiple hidden layers to judge the authenticity of the to-be-identified speech. A suitable classifier can accurately learn the mapping relationship between the feature vector and the authenticity of the speech, thereby outputting a reliable classification result. By reasonably designing the structure and parameters of the classifier, the classification accuracy and stability of the model can be improved.

[0035] The counterfeit speech recognition method provided by the present application can focus on the time domain features and frequency domain features of the audio signal by extracting the time domain feature map and the frequency domain feature map of the audio signal of the to-be-identified speech, and can effectively fuse the differences between the time domain features and the frequency domain features of the audio signal by fusing the time domain feature map and the frequency domain feature map based on the heterogeneous stacked graph attention mechanism, thereby improving the accuracy of feature extraction of the audio signal. The global feature vector of the feature fusion map is extracted. A feature classifier containing multiple hidden layers is used to determine the authenticity of the to-be-identified speech based on the mapping relationship between the feature vector and the authenticity of the to-be-identified speech. By focusing on the global feature of the feature fusion map, the classifier with multiple hidden layers can identify more mapping relationships between the feature vector and the authenticity of the to-be-identified speech, and the authenticity of the to-be-identified speech can be identified more accurately.

[0036] In some possible implementations, the time domain feature map and the frequency domain feature map of the audio signal of the to-be-identified speech are extracted, including: extracting the time domain features and the frequency domain features of the audio signal of the to-be-identified speech by a convolution module; The time-domain feature map and the frequency-domain feature map of the audio signal are constructed based on the time-domain feature and the frequency-domain feature by using a feature map construction function, wherein the feature map construction function is:

[0037]

[0038] wherein, is the frequency-domain feature map, is the time-domain feature map, is the time-domain feature, is the frequency-domain feature, is a pooling operation combined with an attention mechanism, is a max operation, abs is an average operation.

[0039] In the embodiment of the application, the time-domain feature and the frequency-domain feature of the audio signal to be recognized are extracted by an encoder of a speech detection model, the encoder extracts representative feature information from the audio signal through a series of convolution, pooling and other operations, the encoder includes a plurality of convolution operations, can extract the time-domain feature and the frequency-domain feature of the audio signal to be recognized, and then a feature map construction function is used to construct the time-domain feature map and the frequency-domain feature map of the audio signal based on the time-domain feature and the frequency-domain feature, The operation processes the frequency-domain feature, captures the association between different frequency features through a graph attention mechanism, and then optimizes the graph structure through a graph pooling layer to highlight the key spectral features. Similarly, when constructing the time-domain feature map, The same operation is performed on the time-domain feature to focus on the change feature of the audio in the time dimension.

[0040] The embodiment of the application can capture the association between different frequency features and the change feature in the time dimension of the audio signal by extracting the time-domain feature and the frequency-domain feature of the audio signal and constructing the time-domain feature map and the frequency-domain feature map, and ensures the accuracy of the feature extraction of the audio signal.

[0041] In some possible embodiments of the application, as shown in Figure 2 The time-domain feature map and the frequency-domain feature map are fused based on a heterogeneous stacked graph attention mechanism to obtain a feature fusion graph, including: S201, connecting nodes in the time-domain feature map and the frequency-domain feature map and adding a stacked node, the nodes in the time-domain feature map and the frequency-domain feature map are connected to the stacked node to obtain an intermediate feature fusion graph containing a plurality of node pairs; S202, calculating the attention weight between the node pairs based on the type of the node pairs by using different projection methods to obtain a feature fusion graph containing the attention weight.

[0042] In this embodiment of the invention, both the frequency domain feature map and the time domain feature map contain multiple nodes. When fusing the frequency domain feature map and the time domain feature map, as follows: Figure 3 As shown, nodes in the time-domain feature map and the frequency-domain feature map can be connected. Specifically, this is done by adding edges between each node in both maps. These newly added edges allow for the calculation of attention weights across the time and frequency domains through an attention mechanism between heterogeneous node pairs. In the actual calculation of attention weights, different projection vectors are used to calculate attention weights for different types of edges (e.g., edges between nodes within the time-domain feature map, edges between the time-domain and frequency-domain feature maps, and edges between nodes within the frequency-domain feature map). For example, for edges between nodes within the frequency-domain feature map, a specific projection vector is used to calculate the attention weights, enabling the model to capture the relationships between nodes in the frequency domain. For edges between the time-domain and frequency-domain feature maps, the projection vector is suitable for bidirectional edges. The graph attention layer performs element-wise multiplication on the two nodes instead of concatenation, ensuring the attention weights are symmetric and effectively fusing time-domain and frequency-domain information. Furthermore, the heterogeneous stacked graph attention mechanism first projects the temporal and frequency domain feature maps into another latent space through two fully connected layers, unifying the dimensions of the nodes in the temporal and frequency domain feature maps. Then, it projects different node connection edges using different projection methods, calculates projection vectors, and determines the attention weights between node pairs based on these projection vectors. Another key aspect of the heterogeneous stacked graph attention mechanism is the use of stacked nodes. The role of stacked nodes is to aggregate heterogeneous information, especially information and relationships between the frequency and temporal domains. Stacked nodes are connected to all nodes from the temporal and frequency domain feature maps, and these connections are unidirectional (from other nodes to the stacked node), thus preserving temporal and frequency domain information while preventing information from propagating to other nodes. When multiple heterogeneous stacked graph attention layers are sequentially connected, stacked nodes in upper layers can be passed to lower layers. This results in a feature fusion map containing attention weights.

[0043] This invention fuses the time-domain feature map and the frequency-domain feature map of an audio signal to obtain a fused feature map that includes attention weights between node pairs. While capturing time-domain and frequency-domain features, it also considers the weights associated with the time-domain and frequency-domain features, thereby further improving the accuracy of feature extraction.

[0044] In some possible embodiments of the present invention, such as Figure 4 As shown, after obtaining the feature fusion map, it includes: S401, Construct the adjacency matrix of the nodes in the feature fusion graph. The adjacency matrix is ​​used to represent the connection relationship between each node in the feature fusion graph and the attention weight of the connection relationship between each node. S402, update the node features in the feature fusion graph based on the attention weights of the connection relationships between nodes in the adjacency matrix and the connection relationships between nodes.

[0045] In this embodiment of the invention, considering that the fused feature map only extracts the time-domain and frequency-domain features of the audio signal and does not represent the relationships between nodes in the fused feature map, a graph convolutional network is used to process the fused feature map. Graph convolutional networks are powerful tools for processing graph-structured data; they can effectively propagate and aggregate features on graph networks, thereby uncovering deeper relationships between speech features. Through the processing of the graph convolutional network, the local feature information of the nodes can be integrated, enabling the model to learn more representative global features, providing stronger support for subsequent speech authenticity judgment.

[0046] Graph convolutional networks (GCNNs) are used for feature propagation and aggregation in graph networks. Each node updates its own feature representation based on the features of its neighbors and the weights of the edges. Nodes aggregate the features of their neighbors through a weighted summation, ensuring that each node's features include information from its local neighborhood. Let A be the adjacency matrix of the graph and X be the feature matrix. The feature matrix after passing through one layer of a graph convolutional network is... Then we have:

[0047] in, For activation function, , I It is the identity matrix. yes The degree matrix, W The weight matrix is ​​learnable. To enhance feature propagation, a multi-layer graph convolutional network (GCNN) structure is employed. Each layer of the GCNN further updates and abstracts the node features, enabling nodes to capture a wider range of speech feature information. In the multi-layer GCNN, the receptive field gradually increases across different layers, allowing for the analysis of speech features at different scales. After processing each layer of the GCNN, a non-linear activation function is applied to non-linearly transform the node features. Commonly used non-linear activation functions include ReLU, which introduces non-linearity and enhances the model's expressive power. Through non-linear activation, the model can learn more complex speech feature patterns, improving its ability to judge the authenticity of speech. Non-linear activation functions can overcome the limitations of linear transformations, allowing the model to fit more complex decision boundaries, thus more accurately distinguishing between real and fake speech.

[0048] The embodiments of the present invention further process the fused feature map through graph convolutional networks, which can learn more complex speech feature patterns and improve the ability to judge the authenticity of speech.

[0049] In some possible embodiments of the present application, the global feature vector of the feature fusion graph is extracted, including: The global attention weight of each node in the feature fusion graph is calculated by using a preset global attention weight calculation formula, and the global attention weight calculation formula is:

[0050] wherein, is the global attention weight of the i-th node in the feature fusion graph, is a preset activation function, is the feature vector of the i-th node in the feature fusion graph, and j is the j-th neighbor node of the i-th node in the feature fusion graph, is the neighbor node set of the i-th node in the feature fusion graph, is the feature vector of the j-th neighbor node of the i-th node in the feature fusion graph, and W is a learnable weight matrix, is the transpose of the attention weight vector of the connection relationship between the i-th node and the j-th neighbor node in the feature fusion graph; The global feature vector of the feature fusion graph is calculated based on the node feature vector of each node in the feature fusion graph and the global attention weight of each node, and the calculation formula of the global feature vector is:

[0051] wherein, is the global feature vector of the feature fusion graph, and N is the number of nodes in the feature fusion graph.

[0052] In the embodiment of the present application, the foregoing embodiment only extracts the frequency domain features and time domain features of the speech features, without considering the global features of the audio signal, therefore, a global attention mechanism is introduced to calculate the importance weight of each node in the entire graph network. The global attention mechanism considers the mutual relationship between all nodes, assigns a global attention weight to each node by calculating the correlation between the node features, and the global attention weight of each node in the feature fusion graph can be calculated through the global attention weight calculation formula, which reflects the importance of the node in the entire speech detection task, so that the model can pay more attention to the nodes that have important influence on the speech authenticity judgment. The global attention mechanism can grasp the information distribution of the graph network from a macro perspective and highlight the role of key nodes. Then, the global feature vector calculation formula is used to calculate the global feature vector of the feature fusion graph based on the node feature vector of each node in the feature fusion graph and the global attention weight of each node. The global feature vector integrates the information of all nodes in the graph network and highlights the features of important nodes, providing more valuable information for subsequent classification decision. Through feature fusion, the result of the attention mechanism can be integrated into the final feature representation, making the classification decision more accurate.

[0053] In the embodiment of the present application, the global attention mechanism is used to extract the global features of the fusion feature graph, which can effectively capture the global features of the audio signal and ensure the accuracy of the audio information authenticity recognition.

[0054] In some possible embodiments of the present application, as shown in Figure 5 The authenticity of the speech to be recognized is determined based on the mapping relationship between the feature vector and the authenticity of the speech to be recognized by using a feature classifier comprising multiple hidden layers, including: S501, high-level features of the feature vector are extracted by using multiple hidden layers, and the high-level features are connected by using a fully connected layer to obtain a feature output vector; S502, an activation function is used to calculate the probability that the speech to be recognized belongs to real speech and fake speech based on the feature output vector, and the authenticity of the speech to be recognized is determined based on the probability.

[0055] In the embodiment of the present application, the classifier is the final decision module of the entire model, which judges the authenticity of the input speech according to the fusion feature vector obtained in the previous steps. A suitable classifier can accurately learn the mapping relationship between the feature vector and the authenticity of the speech, so as to output reliable classification results. By reasonably designing the structure and parameters of the classifier, the classification accuracy and stability of the model can be improved.

[0056] In this embodiment of the invention, a fully connected neural network is used as the classifier, and the global feature vector from the aforementioned embodiment is used as the input to the classifier. The fully connected neural network contains multiple hidden layers, each of which further transforms and abstracts the input features. The output of the hidden layer of the l-th layer is:

[0057] in, For the first l The output of the hidden layer, For the first l -1 hidden layer output, For activation function, For the first l The weight matrix of the hidden layer, For the first l The bias vector of the hidden layer, when l When =1, .

[0058] Through multi-layered transformations, the classifier can learn the complex mapping relationship between feature vectors and speech authenticity. The multi-layered structure of a fully connected neural network can progressively extract high-level features from the feature vectors, thereby making better classification decisions. In the output layer of the fully connected neural network, an activation function is used to transform the output into a probability distribution, obtaining the probability that the audio signal belongs to each audio category, and then determining the authenticity of the speech to be recognized based on the probability.

[0059] This invention uses a classifier to predict the probability that the speech to be identified belongs to real speech or fake speech based on global feature vectors, thereby achieving the identification of the authenticity of the speech to be identified.

[0060] In some possible embodiments of the present invention, the probability is calculated using the following formula:

[0061] in, Let be the probability that the speech to be recognized belongs to the i-th speech type. The unnormalized score for the speech to be recognized to belong to the i-th speech type. The speech to be recognized belongs to the first... k Unnormalized scores for various speech types K The total number of voice types; Determining the authenticity of a speech object based on probability includes: The speech category with the highest probability and greater than a preset threshold is selected as the speech to be recognized.

[0062] In the embodiment of the present application, the probability formula can be used to calculate the probability of the to-be-identified speech belonging to real speech and fake speech, and according to the probability distribution, the category with the maximum probability is selected as the final classification result, that is, it is judged whether the input speech is real speech or AI-generated fake speech. Meanwhile, in order to improve the accuracy and reliability of the classification, a confidence threshold can be set, and only when the probability of the classification result exceeds the threshold, the classification result is considered to be reliable. By setting the confidence threshold, the misjudgment situation can be reduced, and the robustness of the model can be improved.

[0063] In order to better implement the fake speech recognition method in the embodiment of the present application, on the basis of the fake speech recognition method, as shown in Figure 6 The present application also provides a fake speech recognition device, as shown in The feature map extraction module 601 is configured to extract the time domain feature map and the frequency domain feature map of the audio signal of the to-be-identified speech. The feature map fusion module 602 is configured to fuse the time domain feature map and the frequency domain feature map based on the heterogeneous stack graph attention mechanism, obtain a feature fusion graph, and extract a global feature vector of the feature fusion graph. The classification and recognition module 603 is configured to determine the authenticity of the to-be-identified speech based on the mapping relationship between the feature vector and the authenticity of the to-be-identified speech by using a feature classifier containing multiple hidden layers.

[0064] The fake speech recognition device 600 provided in the above embodiment can implement the technical solutions described in the fake speech recognition method embodiments described above, and the principles of the specific implementation of each module or unit can be referred to the corresponding content in the fake speech recognition method embodiments described above, which will not be described here.

[0065] As shown in Figure 7 The present application also provides an electronic device 700. The electronic device 700 includes a processor 701, a memory 702 and a display 703. Figure 7 Only part of the components of the electronic device 700 are shown, but it should be understood that all the shown components are not required, and more or fewer components can be alternatively implemented.

[0066] The processor 701 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, used to run the program code or process data stored in the memory 702, such as the fake speech recognition method in the present application.

[0067] In some embodiments, the processor 701 can be a single server or a group of servers. The group of servers can be centralized or distributed. In some embodiments, the processor 701 can be local or remote. In some embodiments, the processor 701 can be implemented in a cloud platform. In some embodiments, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination thereof.

[0068] The memory 702 can be an internal storage unit of the electronic device 700, such as a hard disk or a memory of the electronic device 700, in some embodiments. The memory 702 can also be an external storage device of the electronic device 700, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., equipped on the electronic device 700, in other embodiments.

[0069] Further, the memory 702 can include both an internal storage unit and an external storage device of the electronic device 700. The memory 702 is used to store application software and various data installed on the electronic device 700.

[0070] The display 703 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc., in some embodiments. The display 703 is used to display information of the electronic device 700 and to display a visualized user interface. The components 701-703 of the electronic device 700 communicate with each other through a system bus.

[0071] In some embodiments, when the processor 701 executes the counterfeit speech recognition program in the memory 702, the following steps can be implemented: extracting a time domain feature map and a frequency domain feature map of an audio signal of the speech to be recognized; fusing the time domain feature map and the frequency domain feature map based on a heterogeneous stacked graph attention mechanism to obtain a feature fusion map, and extracting a global feature vector of the feature fusion map; adopting a feature classifier containing multiple hidden layers to determine the authenticity of the speech to be recognized based on a mapping relationship between the feature vector and the authenticity of the speech to be recognized.

[0072] It should be understood that, in addition to the above functions, the processor 701 can also implement other functions when executing the counterfeit speech recognition program in the memory 702. For specific descriptions, reference can be made to the descriptions of the corresponding method embodiments.

[0073] Correspondingly, the embodiment of the present application further provides a computer readable storage medium for storing computer readable programs or instructions, which can realize the steps or functions in the counterfeit speech recognition method provided by the above-mentioned method embodiments when executed by a processor.

[0074] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. The computer readable storage medium includes a magnetic disk, an optical disk, a read-only memory, a random access memory, etc.

[0075] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any changes or replacements within the technical range disclosed by the present application can be easily thought of by those skilled in the art, and should be covered within the protection scope of the present application.

Claims

1. A method of spoof speech recognition, the method comprising: The method comprises the following steps: extracting a time domain feature map and a frequency domain feature map of an audio signal of a speech to be identified; fusing the time domain feature map and the frequency domain feature map based on a heterogeneous stacked graph attention mechanism to obtain a feature fusion graph, and extracting a global feature vector of the feature fusion graph; determining the authenticity of the speech to be identified based on a mapping relationship between the feature vector and the authenticity of the speech to be identified by using a feature classifier comprising multiple hidden layers.

2. The method of spoof speech recognition according to claim 1, characterized in that, The step of extracting the time domain feature map and the frequency domain feature map of the audio signal of the speech to be identified comprises the following steps: extracting a time domain feature and a frequency domain feature of the audio signal of the speech to be identified by using a convolution module; constructing the time domain feature map and the frequency domain feature map of the audio signal based on the time domain feature and the frequency domain feature by using a feature map construction function, wherein the feature map construction function is as follows: wherein, is a frequency domain feature map, is a time domain feature map, is a time domain feature, is a frequency domain feature, is a pooling operation combined with attention mechanism, is a max operation, abs is an average operation.

3. The method of claim 1, wherein, The step of fusing the time domain feature map and the frequency domain feature map based on the heterogeneous stacked graph attention mechanism to obtain the feature fusion graph comprises the following steps: connecting nodes in the time domain feature map and the frequency domain feature map and adding a stacked node, wherein the nodes in the time domain feature map and the frequency domain feature map are all connected to the stacked node to obtain an intermediate feature fusion graph comprising multiple node pairs; calculating attention weights between the node pairs based on types of the node pairs by using different projection methods to obtain a feature fusion graph comprising attention weights.

4. The method of recognizing a counterfeit voice according to claim 3, characterized by, After the step of obtaining the feature fusion graph, the following steps are included: constructing an adjacency matrix of nodes in the feature fusion graph, wherein the adjacency matrix is used to represent connection relationships between the nodes in the feature fusion graph and attention weights of the connection relationships between the nodes; updating node features in the feature fusion graph based on the connection relationships of the nodes in the adjacency matrix and the attention weights of the connection relationships between the nodes.

5. The method of recognizing a counterfeit voice according to claim 4, characterized by, The step of extracting the global feature vector of the feature fusion graph comprises the following steps: calculating global attention weights of the nodes in the feature fusion graph by using a preset global attention weight calculation formula, wherein the global attention weight calculation formula is as follows: wherein, is a global attention weight of the i-th node in the feature fusion graph, is a preset activation function, is a feature vector of the i-th node in the feature fusion graph, j is a j-th neighbor node of the i-th node in the feature fusion graph, is a neighbor node set of the i-th node in the feature fusion graph, is a feature vector of the j-th neighbor node of the i-th node in the feature fusion graph, W is a learnable weight matrix, is a transpose of an attention weight vector of a connection relationship between the i-th node and the j-th neighbor node in the feature fusion graph. calculating the global feature vector of the feature fusion graph based on node feature vectors of the nodes in the feature fusion graph and the global attention weights of the nodes, wherein a calculation formula of the global feature vector is as follows: wherein, is a global feature vector of the feature fusion graph, and N is the number of nodes in the feature fusion graph.

6. The method of spoof speech recognition according to claim 5, characterized in that, The step of determining the authenticity of the speech to be identified based on the mapping relationship between the feature vector and the authenticity of the speech to be identified by using the feature classifier comprising multiple hidden layers comprises the following steps: extracting high-level features of the feature vector by using multiple hidden layers, and connecting the high-level features by using a fully connected layer to obtain a feature output vector; calculating probabilities that the speech to be identified belongs to real speech and fake speech based on the feature output vector by using an activation function, and determining the authenticity of the speech to be identified based on the probabilities.

7. The method of spoof speech recognition according to claim 6, characterized in that, A calculation formula of the probabilities is as follows: wherein, is a probability that the speech to be recognized belongs to the i-th i speech type, is a non-normalized score that the speech to be recognized belongs to the i-th i speech type, is a non-normalized score that the speech to be recognized belongs to the i-th k speech type, K is a total number of speech types; The step of determining the authenticity of the speech to be identified based on the probabilities comprises the following steps: taking a speech category with the maximum probability and greater than a preset threshold as a real or fake identification result of the speech to be identified.

8. A spoofed voice recognition apparatus characterized by, The method comprises the following steps: a feature map extraction module is configured to extract a time domain feature map and a frequency domain feature map of an audio signal of a speech to be identified; The feature map fusion module is configured to fuse the time domain feature map and the frequency domain feature map based on a heterogeneous stacked graph attention mechanism, to obtain a feature fusion map, and to extract a global feature vector of the feature fusion map; The classification recognition module is configured to determine the authenticity of the to-be-recognized speech based on a mapping relationship between the feature vector and the to-be-recognized speech by using a feature classifier comprising multiple hidden layers.

9. An electronic device, comprising: comprising a memory and a processor, wherein, The memory is configured to store a program. The processor, coupled with the memory, is configured to execute the program stored in the memory to implement the steps in the counterfeit speech recognition method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer readable program or instruction is stored, and the program or instruction is executed by a processor to implement the steps in the counterfeit speech recognition method of any one of claims 1 to 7.

Images