Deep counterfeit audio identification method based on hybrid enhancement and contrast loss function

Abstract

The invention discloses a deep forged audio identification method based on hybrid enhancement and a contrast loss function. The method comprises the following steps: step 1, preprocessing an original audio training data set by using a sliding window technology and a hybrid enhancement technology; step 2, for the preprocessed original audio training data set, using a contrast loss function to constrain and forge feature vector spacing of real audio training data in a high-dimensional space; and step 3, using a sliding window to divide the audio to be detected into a plurality of audio clips to be detected with the total length of 64600, and if any audio clip to be detected is detected to be forged, determining that the audio to be detected is forged. According to the method, the analysis and identification capability of the algorithm on weak and local counterfeit traces is improved, the accuracy, stability and generalization of the deep counterfeit audio identification algorithm are ensured, the extraction and analysis capability of the algorithm on essential counterfeit traces is improved, and the interference of environmental noise is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and deep fake identification, and relates to a deep fake audio identification method based on hybrid enhancement and contrast loss function. Background Art

[0002] With the rapid development of deep learning, its application scenarios have become more and more extensive. Some researchers have used deep learning to develop intelligent audio synthesis tools that can produce sounds comparable to real human voices. However, although these tools were originally created to help people, they have also been used to spread false audio information on the Internet, and their malicious use has led to people's fear of audio deep fakes. Audio deep fakes, also known as audio manipulation, can be easily used through mobile devices or personal computers to generate false audio and spread it on the Internet. This has led to global public concerns about network security caused by the use of audio deep fakes. Regarding audio deep fakes, the academic community has studied many detection methods to distinguish fake audio from real audio. These methods use different strategies to detect fake audio. First, the audio is preprocessed and converted into appropriate audio features, such as Mel spectrograms. Then, these features are input into the detection model, and the detection model performs some necessary operations, such as feature extraction process. The output of the detection model is then input into a fully connected layer with an activation function to produce the final prediction probability of fake and real. These methods generally adopt various neural network models. The more parameters a model has, the stronger its expressive power is. The more information the model stores, the greater the amount of information, which may even lead to information overload. The main problem with these current methods is that they cannot locate forged key information, and they are not capable of analyzing and identifying weak and local forgery traces, which leads to major problems in accuracy and stability.

[0003] At the same time, for deep fake audio identification, it is also crucial to ignore irrelevant factors such as environmental noise to solve the negative impact of irrelevant factors on the accuracy of the detection system and improve the generalization of the system. However, in the field of audio deep fake identification research, there is a lack of research on noise. At the same time, most related work is focused on a certain data set, and the richness of training data is relatively lacking, which also affects the generalization of the identification method. When the data set is changed, the performance of the detection system may be greatly impacted. Audio deep fake technology is developing rapidly, and how to adapt to new methods and generalize to new fake audio that has not been seen is an urgent problem to be solved. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a deep fake audio identification method based on hybrid enhancement and contrast loss function. The information entropy and learning difficulty of training data are improved through hybrid enhancement technology, and the algorithm's ability to analyze and identify weak and local fake traces is improved. The contrast loss function is used to constrain the feature vector spacing of fake and real audio in high-dimensional space, thereby improving the algorithm's ability to extract and analyze essential fake traces and reducing the interference of environmental noise.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: A deep fake audio identification method based on hybrid enhancement and contrast loss function, characterized by comprising the following steps: Step 1: Use sliding window technology and hybrid enhancement technology to preprocess the original audio training data set: first, use the sliding window technology to segment the real audio and fake audio in the original audio training data set to obtain multiple real audio segments and multiple fake audio segments, and the sliding window size is 64600; then randomly cut and extract fake audio with a total length of 0.5s-1.5s from the multiple fake audio segments and put them into the fake audio library, and take multiple real audio segments to construct a real audio window library; finally, extract an element from the fake audio library and the real audio window library respectively, and embed the fake audio element into the random position of the real audio element for mixing, so as to ensure that the audio segment window size after mixing is still 64600; Step 2: for the preprocessed original audio training data set, constrain the feature vector spacing of the forged and real audio training data in a high-dimensional space based on the contrast loss function, and train the forged audio identification network; Step 3: Use a sliding window to divide the audio to be tested into multiple audio segments with a total length of 64,600 for detection, and put them into the trained forged audio identification network. If any audio segment to be tested is detected as forged, the audio to be tested is forged.

[0006] Furthermore, the step 1 comprises: Use a sliding window to cut the real audio and fake audio in the original audio training data set respectively. If the size of the audio data V1 obtained after cutting is smaller than the sliding window size, the self-replication technology is used to supplement the size of the audio data V1 obtained after cutting to the sliding window size.

[0007] Furthermore, the step 2 comprises: The preprocessed original audio training data set is randomly divided into two groups for training. For each group of two audio training data, features are extracted and represented as feature vectors. The distance between the feature vectors of two audio training data is defined as : , in, represents the Euclidean distance operation, Represent the feature vectors of two audio training data respectively, and Represent two different audio training data respectively; when performing the comparative prediction task on the paired feature vectors of N audio training data, let Y be the binary label of each pair of feature vectors. and When they belong to the same category, Y = 0. and When they belong to different categories, Y = 1, and the category includes real audio training data and forged audio training data, then the contrast loss function is for: , Among them, the boundary m is a positive number, when the distance between the feature vectors of two audio training data When it is smaller than m, different pairs of feature vectors result in contrastive loss.

[0008] The beneficial effects of the present invention are: (1) This paper proposes a data enhancement method based on hybrid enhancement. While using a sliding window to cut the original audio to ensure the richness of the training data, the hybrid enhancement technology is used to embed the forged segments into the real audio, which greatly improves the information entropy and learning difficulty of the training data, and improves the algorithm's ability to analyze and identify weak and local forgery traces, ensuring the accuracy, stability and generalization of the deep forgery audio identification algorithm.

[0009] (2) The present invention proposes a contrastive loss function method to process audio features. By using the contrastive loss function on sample feature pairs, the features of speech data of the same category are kept close, while the features of speech data of different categories are separated. By constraining the feature vector spacing between forged and real audio in high-dimensional space, the algorithm's ability to extract and analyze essential forgery traces is improved, the interference of environmental noise is reduced, and generalization is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flow chart of the deep fake audio identification method based on hybrid enhancement and contrast loss function of the present invention; Figure 2 Schematic diagram of a method for preprocessing audio training data according to the present invention. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0012] The deep fake audio identification method based on hybrid enhancement and contrast loss function proposed in the present invention mainly includes two parts: hybrid enhancement pre-processing audio training data and feature processing based on contrast loss optimization. The specific implementation method is as follows: Figure 1 The flowchart of the deep fake audio identification method based on hybrid enhancement and contrast loss function of the present invention is shown, which mainly includes the following steps: Step 1: Mixing and enhancing preprocessed audio training data In order to increase the information entropy and learning difficulty of training data, improve the richness of training data, and improve the accuracy and stability of the method, in the specific process of audio deep fake identification, the audio training data is first preprocessed using hybrid enhancement technology and sliding window method.

[0013] Hybrid enhancement technology is a technology in image processing. Based on its principle, the present invention studies hybrid enhancement technology applied to audio. This is also the first time that similar technology appears in the field of audio identification. The specific processing process is as follows: like Figure 2 As shown in the figure, the forged audio and real audio in the training set are first cut using a sliding window. The sliding window size is 64600, and the cut audio clips are about 4 seconds long. When cutting the audio, if the obtained audio is less than 64600, the audio length is supplemented to 64600 by self-copying to ensure the consistency of the audio length. At the same time, the audio sampling rate needs to be 16000.

[0014] Then, using hybrid enhancement technology, for all fake audio clips of window size, randomly cut and extract fake audio of 0.5s-1.5s in length, and then add it to the fake audio library; for half of the real audio in the training set, the real audio data of the window size obtained after cutting are all added to the real audio window library; then extract one element from the fake audio library and the real audio window library respectively, embed the fake audio element into the random position of the real audio window element, and ensure that the sample obtained after embedding is still 64600 in size, and the obtained sample is converted into a fake sample. The number of fake samples obtained by this method is set to half of the number of real audio samples. All the fake samples obtained are put into the training set for subsequent training.

[0015] In this way, the data in the test set also needs to be cut using a sliding window. When calculating the results, all window results of an audio piece need to be fused and calculated. If one window is judged to be forged, the entire audio piece is considered forged.

[0016] Step 2: Optimize feature processing based on contrast loss In order to further improve the algorithm's ability to extract and analyze essential forgery traces, reduce the interference of environmental noise, and improve generalization, in the specific feature processing process, contrast loss is used to constrain the feature vector spacing of forged and real audio in high-dimensional space, thereby enhancing the algorithm's ability to identify forged audio.

[0017] Contrastive loss is a commonly used dimensionality reduction learning method and is widely used in other fields. The present invention introduces contrastive loss into audio identification for the first time. Meanwhile, its use in the training phase is also different, and the audio is grouped before training.

[0018] In the specific implementation process, during the training phase, the training set obtained by the previous preprocessing is randomly divided into two groups for training. First, the features of the two audio segments are obtained through feature extraction, and then the contrast loss is used to ensure that the distance between the two audio segments of the same category is close to 0, and the distance between the two audio segments of different categories is as large as possible without exceeding a certain threshold. In this task, the same category means that the two recordings are either from real speech or from forged speech, and different means that one recording is from real speech and the other recording is from forged speech. The features of the two audio segments can be represented by vectors, and the distance between the two representation vectors is defined as , the specific formula is: , in, represents the Euclidean distance.

[0019] When performing a comparative prediction task on paired features of N samples, let Y be the binary label of each pair of features. and When they belong to the same class, Y = 0. and When they belong to different classes, Y = 1. Then the contrast loss function formula is: , Among them, the boundary m is a positive number. Different feature pairs will only cause contrast loss when the distance between two sample features is less than m. In this way, feature representation learning is promoted, the features of speech data of the same category are kept close, while the features of speech data of different categories are separated, and the feature vector spacing of forged and real audio is constrained in high-dimensional space. The forged audio identification network is trained based on this contrast loss function and the preprocessed original audio training dataset; For example, for a pair of real sample features, Y=0. The greater the distance between the pair of sample features, the greater the loss. The loss constraint can shorten the distance between the pair of sample features. For a pair of features consisting of a real sample feature and a forged sample feature, Y=1. The greater the distance between the pair of sample features, the smaller the loss. The loss constraint can shorten the distance between the pair of sample features. At the same time, the distance also has a maximum limit. If it exceeds the maximum limit, the loss will not be calculated.

[0020] Step 3: Audio Deep Fake Detection Finally, for the detection process of the audio test set data, it is also necessary to first use a sliding window of size 64600 to cut the test set data, and then put each cut audio segment into the trained forged audio identification network for detection. When calculating the results, it is necessary to merge all the cut audio segments of an audio. If one window is judged to be forged, the entire audio is forged.

[0021] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A deep fake audio identification method based on hybrid enhancement and contrast loss function, characterized in that: The steps include: Step 1: Use sliding window technology and hybrid enhancement technology to preprocess the original audio training data set: first, use the sliding window technology to segment the real audio and fake audio in the original audio training data set to obtain multiple real audio segments and multiple fake audio segments, and the sliding window size is 64600; then randomly cut and extract fake audio with a total length of 0.5s-1.5s from the multiple fake audio segments and put them into the fake audio library, and take multiple real audio segments to construct a real audio window library; finally, extract an element from the fake audio library and the real audio window library respectively, and embed the fake audio element into the random position of the real audio element for mixing, so as to ensure that the audio segment window size after mixing is still 64600; Step 2: for the preprocessed original audio training data set, constrain the feature vector spacing of the forged and real audio training data in a high-dimensional space based on the contrast loss function, and train the forged audio identification network; Step 3: Use a sliding window to divide the audio to be tested into multiple audio segments with a total length of 64,600 for detection, and put them into the trained forged audio identification network. If any audio segment to be tested is detected as forged, the audio to be tested is forged.

2. According to claim 1, a deep fake audio identification method based on hybrid enhancement and contrast loss function is characterized in that: The step 1 comprises: Use a sliding window to cut the real audio and fake audio in the original audio training data set respectively. If the size of the audio data V1 obtained after cutting is smaller than the sliding window size, the self-replication technology is used to supplement the size of the audio data V1 obtained after cutting to the sliding window size.

3. A deep fake audio identification method based on hybrid enhancement and contrast loss function according to claim 2, characterized in that: The step 2 comprises: The preprocessed original audio training data set is randomly divided into two groups for training. For each group of two audio training data, features are extracted and represented as feature vectors. The distance between the feature vectors of two audio training data is defined as : ， in, represents the Euclidean distance operation, Represent the feature vectors of two audio training data respectively, and Represent two different audio training data respectively; when performing the comparative prediction task on the paired feature vectors of N audio training data, let Y be the binary label of each pair of feature vectors. and When they belong to the same category, Y = 0. and When they belong to different categories, Y = 1, and the category includes real audio training data and forged audio training data, then the contrast loss function is for: , Among them, the boundary m is a positive number, when the distance between the feature vectors of two audio training data When it is smaller than m, different pairs of feature vectors result in contrastive loss.

Images