Methods, devices, equipment, and media for synthesized audio recognition based on persona characteristics

Abstract

This disclosure provides a method, apparatus, device, and medium for synthetic audio recognition based on persona features. The method includes: acquiring multimodal user data to be recognized; wherein the multimodal user data includes at least audio data; extracting features from the audio data using a corresponding encoder to obtain a first audio feature vector; extracting features from the user data of the corresponding modality using an encoder adjusted through contrastive learning to obtain feature vectors corresponding to each modality, wherein the feature vectors include a second audio feature vector; concatenating the other feature vectors except the second audio feature vector to obtain a persona feature vector; calculating the similarity between the second audio feature vector and the persona feature vector; and determining whether the audio data is natural audio or synthetic audio based on the similarity and the first audio feature vector. This invention can improve the accuracy of synthetic audio recognition.

Description

Technical Field

[0001] This invention relates to the field of synthesized speech, and more specifically to a method, apparatus, device, and medium for synthesized audio recognition based on human characteristics. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence, speech synthesis technology has matured and has been widely applied in fields such as human-computer interaction, smart homes, entertainment, and education. Current speech synthesis technology produces results that closely resemble real speech in terms of clarity and naturalness, and the acoustic parameters of the synthesized audio are also very close to real speech, capable of generating speech comparable to that of a human. While the development of speech synthesis technology has brought convenience to people, its misuse has also caused harm to the public and society. Therefore, the corresponding technology for distinguishing between real and fake speech has received widespread attention.

[0003] In prior art, the detection of synthesized speech mainly relies on simple audio for identification, which not only has limited accuracy but is also difficult to improve. Summary of the Invention

[0004] In view of this, the present disclosure provides a method, apparatus, device and medium for synthetic audio recognition based on persona features, which at least partially solves the problems existing in the prior art.

[0005] This invention provides a synthetic audio recognition method based on persona features, comprising:

[0006] Acquire multimodal user data to be identified; wherein, the multimodal user data includes at least audio data;

[0007] The first audio feature vector is obtained by extracting features from the audio data using a corresponding encoder.

[0008] Using an encoder adjusted through contrastive learning, features are extracted from user data of the corresponding modality to obtain feature vectors corresponding to each modality, wherein the feature vectors include a second audio feature vector.

[0009] Concatenate the feature vectors other than the second audio feature vector to obtain the persona feature vector;

[0010] Calculate the similarity between the second audio feature vector and the persona feature vector;

[0011] Based on the similarity and the first audio feature vector, the audio data is determined to be either natural audio or synthetic audio.

[0012] Preferably, the multimodal user data further includes the user's image data and text data describing the persona; then the encoder includes a Wav2vec2.0 encoder for feature extraction of audio data, a ViT encoder for feature extraction of images, and a BERT encoder for feature extraction of text data.

[0013] Preferably, the textual data describing the persona includes age, gender, personality, and emotions.

[0014] Preferably, before using the encoder adjusted through contrastive learning to extract features from user data of the corresponding modality and obtain feature vectors corresponding to each modality, the feature vectors include the following steps before including audio feature vectors:

[0015] Acquire multiple sets of multimodal user data for training;

[0016] For each set of multimodal user data, extract the feature vector corresponding to each modality of user data;

[0017] Concatenate the feature vectors other than the audio feature vector to obtain the persona feature vector;

[0018] The parameters of each encoder are adjusted by comparing the persona feature vector with the audio feature vector to make the paired feature vectors more similar in the feature space, while the unpaired feature vectors are as far apart as possible in the vector space. During the comparative learning, the persona feature vector and its paired audio feature vector are used as positive examples, and the unpaired persona feature vector and audio feature vector are used as negative examples. For a given example... and its corresponding positive example and negative samples Using Euclidean distance We calculate the distance between vectors; at the same time, we define a contrastive loss to narrow the distance between positive samples. ,in The distance is Euclidean distance, and m is used to control the offset between positive and negative sample distances, so that the model does not need to consider optimizing overly simple negative samples.

[0019] Preferably, determining whether the audio data is natural audio or synthetic audio based on the similarity and the first audio feature vector specifically includes:

[0020] The similarity is directly encoded and then input into a Transformer neural network to obtain the first degree of consistency between the second audio feature vector and the persona feature vector; wherein, the similarity is represented by the Euclidean distance between the second audio feature vector and the persona feature vector;

[0021] The first audio feature vector is input into the Transformer neural network to obtain the second degree of consistency between the first audio feature vector and the synthesized audio.

[0022] Based on the first degree of conformity and the second degree of conformity and their corresponding weights, the audio data is determined to be either natural audio or synthetic audio.

[0023] Preferably, the Transformer neural network uses cross-entropy loss L2, and the total target loss calculation function of the neural network is:

[0024] in and For weights.

[0025] Preferably, it further includes:

[0026] An 18-layer residual network is used to automatically generate the optimal... and Weights.

[0027] This invention also provides a synthetic audio recognition device based on persona features, comprising:

[0028] The user data acquisition unit is used to acquire multimodal user data to be identified; wherein, the multimodal user data includes at least audio data;

[0029] The first feature extraction unit is used to extract features from the audio data using a corresponding encoder to obtain a first audio feature vector.

[0030] The second feature extraction unit is used to extract features from user data of the corresponding modality using the encoder adjusted through contrastive learning, and to obtain feature vectors corresponding to each modality, wherein the feature vectors include the second audio feature vector.

[0031] The splicing unit is used to splice the feature vectors other than the second audio feature vector to obtain the persona feature vector;

[0032] A similarity calculation unit is used to calculate the similarity between the second audio feature vector and the persona feature vector;

[0033] The judgment unit is used to determine whether the audio data is natural audio or synthetic audio based on the similarity and the first audio feature vector.

[0034] This invention also provides a synthetic audio recognition device based on persona features, which includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement the synthetic audio recognition method based on persona features as described above.

[0035] This invention also provides a computer-readable storage medium storing a computer program that can be executed by a processor of the device in which the computer-readable storage medium is located, to implement the synthetic audio recognition method based on person-features as described above.

[0036] In summary, this embodiment, while taking into account the acoustic features of the audio itself, can assist in audio synthesis detection by fusing multiple modalities outside of audio (such as image information containing human appearance, and text information describing personality traits including age, gender, personality, and emotions). This effectively addresses the problem that current synthesized audio is becoming increasingly similar to real audio in its acoustic features, making it difficult for models to improve audio synthesis detection performance. Furthermore, the method in this embodiment is robust, capable of performing audio synthesis detection normally even when relying solely on audio or when some modalities are missing. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the synthetic audio recognition method based on persona features according to the first embodiment of the present invention.

[0039] Figure 2 This is a schematic diagram illustrating the adjustment of the encoder using contrastive learning;

[0040] Figure 3 This is a schematic diagram illustrating how a neural network is used to determine whether an audio file is synthesized speech.

[0041] Figure 4 This is a schematic diagram of the structure of the synthetic audio recognition device based on human characteristics according to the second embodiment of the present invention. Detailed Implementation

[0042] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0043] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0044] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0045] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0046] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0047] Please see Figure 1 The first embodiment of the present invention provides a synthetic audio recognition method based on persona features, which can be executed by a synthetic audio recognition device based on persona features (hereinafter referred to as the recognition device), specifically, by one or more processors within the recognition device, to achieve the following steps:

[0048] S101, acquire the multimodal user data to be identified; wherein, the multimodal user data includes at least audio data.

[0049] In this embodiment, the identification device may be a terminal device with computing and processing capabilities, such as a computer, workstation, or server; the present invention does not impose any specific limitations.

[0050] In this embodiment, before identification, it is necessary to acquire multimodal user data. Different forms of existence or information sources can be referred to as a modality. Data composed of two or more modalities is called multimodal data (multimodal is used to represent different forms of data, or different formats of the same form, generally representing text, images, audio, video, and mixed data).

[0051] In this embodiment, the multimodal user data specifically includes image data depicting a person's appearance, text data describing personality traits, such as text information including elements like age, gender, personality, and emotions, and corresponding audio data. Of course, it may also include other modal user data, such as fingerprint data, etc., and this invention does not specifically limit it.

[0052] S102, the first audio feature vector is obtained by extracting features from the audio data using the corresponding encoder.

[0053] S103, using the encoder adjusted through contrastive learning, features are extracted from user data of the corresponding modality to obtain feature vectors corresponding to each modality, the feature vectors including the second audio feature vector.

[0054] In this embodiment, as Figure 2 As shown, the encoder includes a Wav2vec2.0 encoder for feature extraction of audio data, a ViT encoder for feature extraction of images, and a BERT encoder for feature extraction of text data.

[0055] Specifically, the Wav2vec2.0 encoder before contrastive learning adjustment was used when extracting the first audio feature vector, while the Wav2vec2.0 encoder after contrastive learning adjustment was used when extracting the second audio feature vector.

[0056] Specifically, contrastive learning focuses on learning the common features among similar instances and distinguishing the differences between dissimilar instances. Compared to generative learning, contrastive learning does not need to focus on the tedious details of instances; it only needs to learn to distinguish data at the abstract semantic level of feature space. Therefore, the model and its optimization become simpler, and its generalization ability is stronger. Contrastive learning aims to reduce the distance with positive samples and increase the distance with negative samples, making the distance between positive samples and anchor points much smaller than the distance between negative samples and anchor points (or making the similarity between positive samples and anchor points much greater than the similarity between negative samples and anchor points), thereby achieving the true distance of their original spatial distribution.

[0057] Among them, such as Figure 2 As shown, in order to obtain the encoder adjusted by contrastive learning, then:

[0058] First, acquire multiple sets of multimodal user data for training.

[0059] The multi-modal user data here includes user data from different users.

[0060] Then, for each group of multimodal user data, the feature vectors corresponding to each modality of user data are extracted.

[0061] Next, the feature vectors other than the audio feature vector are concatenated to obtain the persona feature vector.

[0062] Taking the above as an example, the image feature vector and the text feature vector can be concatenated to obtain the persona feature vector (if the image feature vector or the text feature vector is missing, it is padded with zeros).

[0063] Finally, the parameters of each encoder are adjusted by comparing and learning the persona feature vector with the audio feature vector, so that the paired feature vectors encoded by each encoder are closer in the feature space, while the unpaired feature vectors are as far apart as possible in the vector space.

[0064] In contrastive learning, the persona feature vector and its paired audio feature vector are used as positive examples, while unpaired persona feature vectors and audio feature vectors are used as negative examples; for a given example and its corresponding positive example and negative samples Using Euclidean distance We calculate the distance between vectors; at the same time, we define a contrastive loss to narrow the distance between positive samples. ,in The distance is Euclidean distance, and m is used to control the offset between positive and negative sample distances, so that the model does not need to consider optimizing overly simple negative samples.

[0065] In this embodiment, by continuously adjusting the encoder parameters, the paired feature vectors are made closer in the middle of the features, while the unpaired feature vectors are as far apart as possible in the vector space, thus obtaining the encoder adjusted by contrastive learning.

[0066] Furthermore, during the comparative learning process, the feature points of the various modalities differ after concatenation. Therefore, the network focuses on different aspects during training, leading to faster convergence of the dominant modality and a decrease in shared weights, which hinders the optimization of other modalities. To fully utilize the information from different modalities, an automatic feature weight optimization mechanism based on task-specific multi-attribute evaluation is employed to automatically optimize gradients and ensure that the convergence speeds of each feature are similar. Specifically:

[0067]

[0068] Among them, the dominant mode v will have ρ v > 1, which will lead to faster convergence.

[0069]

[0070] The learning speed of the dominant mode can be slowed down by artificially reducing the overall gradient of the dominant mode.

[0071] S104, concatenate the feature vectors other than the second audio feature vector to obtain the persona feature vector.

[0072] S105, calculate the similarity between the second audio feature vector and the persona feature vector.

[0073] The similarity between the second audio feature vector and the persona feature vector can be obtained by calculating the Euclidean distance between them.

[0074] S106, Based on the similarity and the first audio feature vector, determine whether the audio data is natural audio or synthetic audio.

[0075] Specifically:

[0076] First, the similarity is directly encoded and input into a Transformer neural network to obtain the first degree of consistency between the second audio feature vector and the persona feature vector; wherein, the similarity is represented by the Euclidean distance between the second audio feature vector and the persona feature vector;

[0077] Then, the first audio feature vector is input into the Transformer neural network to obtain the second degree of consistency between the first audio feature vector and the synthesized audio;

[0078] Finally, based on the first degree of conformity and the second degree of conformity and their corresponding weights, the audio data is determined to be either natural audio or synthetic audio.

[0079] Among them, such as Figure 3 As shown, in this embodiment, the Transformer neural network needs to go through two processes to determine whether the audio data is synthesized speech.

[0080] First, based on the acoustic features of the audio data, it is determined whether the audio is synthesized. The audio is converted into a feature vector using the original, untuned Wav2vec 2.0 encoder, which contains many acoustic features. This feature vector is then input into a Transformer neural network for audio synthesis detection. The loss function is cross-entropy loss, denoted as L1.

[0081] Second, it calculates whether the second audio feature vector and the persona feature vector match. The text feature vector and image feature vector are concatenated, and the Euclidean distance is calculated between them and the second audio feature vector. This Euclidean distance is then directly encoded and input into the Transformer neural network to determine if they match. The loss function used is cross-entropy loss L2.

[0082] The Transformer neural network model requires the calculation of two losses; the total objective loss calculation function is as follows:

[0083] in and To improve model performance, this embodiment employs an additional 18-layer residual network for automatically generating optimal weights. and Weights are determined by using a new neural network to make decisions, which in turn determine the impact of each factor on the final judgment.

[0084] In summary, this embodiment, while taking into account the acoustic features of the audio itself, can assist in audio synthesis detection by fusing multiple modalities outside of audio (such as image information containing human appearance, and text information describing personality traits including age, gender, personality, and emotions). This effectively addresses the problem that current synthesized audio is becoming increasingly similar to real audio in its acoustic features, making it difficult for models to improve audio synthesis detection performance. Furthermore, the method in this embodiment is robust, capable of performing audio synthesis detection normally even when relying solely on audio or when some modalities are missing.

[0085] Please see Figure 4 The second embodiment of the present invention also provides a synthetic audio recognition device based on persona features, which includes:

[0086] User data acquisition unit 210 is used to acquire multimodal user data to be identified; wherein, the multimodal user data includes at least audio data;

[0087] The first feature extraction unit 220 is used to extract features from the audio data using a corresponding encoder to obtain a first audio feature vector.

[0088] The second feature extraction unit 230 is used to extract features from user data of the corresponding modality using the encoder adjusted through contrastive learning, and to obtain feature vectors corresponding to each modality, wherein the feature vectors include the second audio feature vector.

[0089] The splicing unit 240 is used to splice the feature vectors other than the second audio feature vector to obtain the persona feature vector;

[0090] The similarity calculation unit 250 is used to calculate the similarity between the second audio feature vector and the persona feature vector;

[0091] The judgment unit 260 is used to determine whether the audio data is natural audio or synthetic audio based on the similarity and the first audio feature vector.

[0092] The third embodiment of the present invention also provides a synthetic audio recognition device based on persona features, which includes a memory and a processor. The memory stores a computer program, which can be executed by the processor to realize the synthetic audio recognition method based on persona features as described above.

[0093] The fourth embodiment of the present invention also provides a computer-readable storage medium storing a computer program that can be executed by a processor of the device in which the computer-readable storage medium is located, so as to implement the synthetic audio recognition method based on human characteristics as described above.

[0094] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for synthesized audio recognition based on persona features, characterized in that, include: Acquire multimodal user data to be identified; wherein, the multimodal user data includes at least audio data; The first audio feature vector is obtained by extracting features from the audio data using a corresponding encoder. Using an encoder adjusted through contrastive learning, features are extracted from user data of the corresponding modality to obtain feature vectors corresponding to each modality, wherein the feature vectors include a second audio feature vector. The feature vectors other than the first and second audio feature vectors are concatenated to obtain the persona feature vector. Calculate the similarity between the second audio feature vector and the persona feature vector; Based on the similarity and the first audio feature vector, the audio data is determined to be either natural audio or synthetic audio.

2. The synthetic audio recognition method based on persona features according to claim 1, characterized in that, The multimodal user data also includes user image data and text data describing the user persona; the encoder includes a Wav2vec2.0 encoder for feature extraction of audio data, a ViT encoder for feature extraction of images, and a BERT encoder for feature extraction of text data.

3. The synthetic audio recognition method based on persona features according to claim 1, characterized in that, The textual data describing a persona includes age, gender, personality, and emotions.

4. The synthetic audio recognition method based on persona features according to claim 1, characterized in that, Using an encoder adjusted through contrastive learning, features are extracted from user data of corresponding modalities to obtain feature vectors corresponding to each modality. These feature vectors include, before the audio feature vectors, the following: Acquire multiple sets of multimodal user data for training; For each set of multimodal user data, extract the feature vector corresponding to each modality of user data; Concatenate the feature vectors other than the audio feature vector to obtain the persona feature vector; The character feature vector and the audio feature vector are compared and learned to adjust the parameters of each encoder, so that the paired feature vectors encoded by each encoder are closer in the feature space, while the unpaired feature vectors are as far apart as possible in the vector space. During the comparison learning, the character feature vector and its paired audio feature vector are used as positive examples. Unpaired persona feature vectors and audio feature vectors are used as negative samples. For a given example and its corresponding positive example and negative samples Euclidean distance is used; We calculate the distance between vectors; at the same time, we define a contrastive loss to narrow the distance between positive samples. ,in The distance is Euclidean distance, and m is used to control the offset between positive and negative sample distances, so that the model does not need to consider optimizing overly simple negative samples.

5. The synthetic audio recognition method based on persona features according to claim 4, characterized in that, Determining whether the audio data is natural or synthetic based on the similarity and the first audio feature vector specifically includes: The similarity is directly encoded and then input into a Transformer neural network to obtain the first degree of consistency between the second audio feature vector and the persona feature vector; wherein, the similarity is represented by the Euclidean distance between the second audio feature vector and the persona feature vector; the loss of the Transformer neural network is the cross-entropy loss, denoted as L2; The first audio feature vector is input into the Transformer neural network to obtain the second degree of consistency between the first audio feature vector and the synthesized audio; the loss of the Transformer neural network adopts the cross-entropy loss, denoted as L1; Based on the first degree of conformity and the second degree of conformity and their corresponding weights, the audio data is determined to be either natural audio or synthetic audio.

6. The synthetic audio recognition method based on persona features according to claim 5, characterized in that, The overall target loss function for the two Transformer neural networks is: ;in and For weights.

7. The synthetic audio recognition method based on persona features according to claim 6, characterized in that, Also includes: An 18-layer residual network is used to automatically generate the optimal... and Weights.

8. A synthetic audio recognition device based on persona characteristics, characterized in that, include: The user data acquisition unit is used to acquire multimodal user data to be identified; wherein, the multimodal user data includes at least audio data; The first feature extraction unit is used to extract features from the audio data using a corresponding encoder to obtain a first audio feature vector. The second feature extraction unit is used to extract features from user data of the corresponding modality using the encoder adjusted through contrastive learning, and to obtain feature vectors corresponding to each modality, wherein the feature vectors include the second audio feature vector. The splicing unit is used to splice the feature vectors other than the second audio feature vector to obtain the persona feature vector; A similarity calculation unit is used to calculate the similarity between the second audio feature vector and the persona feature vector; The judgment unit is used to determine whether the audio data is natural audio or synthetic audio based on the similarity and the first audio feature vector.

9. A synthesized audio recognition device based on persona characteristics, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program that can be executed by the processor to implement the synthetic audio recognition method based on person-features as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that can be executed by a processor of the device on which the computer-readable storage medium is located, to implement the synthetic audio recognition method based on person-features as described in any one of claims 1 to 7.

Images