Method and system for identifying forged audio based on improved RawNet2

Abstract

The invention relates to the technical field of forged audio recognition, and provides a forged audio recognition method and system based on improved RawNet2. The method comprises the following steps: collecting multiple dimensions of forged audios in a call scene, and establishing a training data set; based on an improved RawNet2 neural network, constructing a forged audio recognition model, performing model training by using a training data set to obtain a trained forged audio recognition model, and splitting an original waveform of a call audio stream by introducing an orthogonal processing layer to extract more forged audio features; a time convolution network is introduced to enlarge the receptive field of a forged audio detection network so as to capture forged audio features; and intercepting an original call audio with a preset length from the call audio stream to be processed, and inputting the original call audio into the forged audio recognition model to obtain a recognition result so as to determine an early warning prompt in real time. According to the method, the equal error rate is remarkably reduced by 60%-70%, the model detection effect is improved, and forged audios can be recognized more effectively.

Description

Technical Field

[0001] The present invention relates to the technical field of forged audio recognition, and in particular to a forged audio recognition method and system based on improved RawNet2. Background Art

[0002] In recent years, the mobile information service industry based on mobile communication networks has been welcomed by the majority of users due to its convenience and fast service. However, with the rapid development of information technology, the deep fake call audio synthesized by AI has increased rapidly. These fake calls involving fraud and harassment by AI, which are of various types, difficult to distinguish, and large in volume, have caused great interference to the work and life of the majority of mobile phone users, had a negative impact on the corporate image of various operators, and also brought great difficulties to the supervision of relevant departments.

[0003] However, due to the real-time nature of call scenarios and the diversity of methods of forged audio, existing forged audio recognition technologies find it difficult to distinguish normal calls from forged audio calls in real time, and have poor recognition effects on AI harassment and fraud calls, which seriously affects the user experience.

[0004] In the prior art, traditional methods such as call behavior analysis and blacklist management are usually used to handle fake audio calls, and the accuracy of identifying and warning fake audio calls is low. In addition, there is usually a lag in judging whether a call contains fake audio, and the judgment result can often be determined some time after the user's call ends. It is impossible to conduct real-time warning, interception and other operations on fake audio fraud, harassment and other activities by criminals. In addition, the deep fake audio detection capability of the prior art is weak, the fake detection coverage is small, the detection accuracy is low, and it can usually only cover one or two fake methods, which cannot meet the needs of identifying fake audio in actual business calls. In addition, the prior art lacks call scene-related data sets, the total amount of training data is small, and the fake methods covered are few. It cannot meet the current training needs of multiple fake audio methods, and lacks data such as environmental noise, signal noise, music songs, etc. contained in actual call scenes, resulting in poor model effects for call scene training.

[0005] Therefore, the present invention provides a forged audio recognition method based on improved RawNet2 to solve the above problems. Summary of the invention

[0006] The present invention intends to provide a forged audio recognition method and system based on improved RawNet2 to solve the problems in the prior art that it is difficult to distinguish normal calls from forged audio calls in real time, the recognition effect of AI harassment and fraud calls is poor, resulting in poor user experience, and the inability to meet the training requirements of various current forged audio methods. There is a lack of data such as environmental noise, signal noise, music songs, etc. contained in actual call scenarios, resulting in poor model effect for call scenario training. The technical problems to be solved by the present invention are achieved through the following technical solutions.

[0007] The first aspect of the present invention proposes a method for identifying forged audio based on improved RawNet2, including: collecting forged audio of multiple dimensions in call scenarios to establish a training data set, wherein the multiple dimensions include real customer service calls, synthesized forged audio, call environment noise, signal noise, and music songs; constructing a forged audio recognition model based on the improved RawNet2 neural network, and using the training data set to train the forged audio recognition model to obtain a trained forged audio recognition model, which specifically includes: introducing an orthogonal processing layer to split the original waveform of the call audio stream to extract more forged audio features; introducing a temporal convolutional network to expand the receptive field of the forged audio detection network to capture forged audio features; intercepting a preset length of original call audio from the call audio stream to be processed to input the forged audio recognition model to obtain a recognition result; and determining an early warning prompt in real time based on the obtained recognition result.

[0008] The second aspect of the present invention proposes a forged audio recognition system based on improved RawNet2, which adopts the forged audio recognition method based on improved RawNet2 described in the first aspect of the present invention. The forged audio recognition system includes: a data collection module, which is used to collect forged audio of multiple dimensions in call scenarios and establish a training data set, wherein the multiple dimensions include real customer service calls, synthesized forged audio, call environment noise, signal noise, and music songs; a model construction module, which constructs a forged audio recognition model based on the neural network of improved RawNet2, and uses the training data set to train the forged audio recognition model to obtain a trained forged audio recognition model, which specifically includes: introducing an orthogonal processing layer to split the original waveform of the call audio stream to extract more forged audio features; introducing a time convolutional network to expand the receptive field of the forged audio detection network to capture the forged audio features; a determination module, which is used to intercept the original call audio of a preset length from the call audio stream to be processed to input the forged audio recognition model to obtain a recognition result; and an early warning module, which determines an early warning prompt in real time according to the obtained recognition result.

[0009] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the forged audio identification method based on improved RawNet2 described in the first aspect of the present invention.

[0010] A fourth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the forged audio identification method based on improved RawNet2 described in the first aspect of the present invention is implemented.

[0011] The embodiments of the present invention include the following advantages:

[0012] Compared with the prior art, the present invention establishes a training data set by collecting forged audio of multiple dimensions in call scenarios, wherein the multiple dimensions include real customer service calls, synthesized forged audio, call environment noise, signal noise, and music songs; constructs a forged audio recognition model based on an improved RawNet2 neural network, and uses the training data set to train the forged audio recognition model to obtain a trained forged audio recognition model, which can retain more dimensions of original audio information, enable the model network to learn low-dimensional embedding for specific tasks, and improve the model's recognition accuracy for forged audio; intercepts original call audio of a preset length from the call audio stream to be processed to input the forged audio recognition model to obtain a recognition result, and can directly process the original waveform, so that the network of the forged audio recognition model can learn forged audio features customized for forged audio detection tasks, significantly reducing the equal error rate by 60%-70%, thereby improving the model detection effect, and can more effectively identify forged audio; based on the obtained recognition result, it is possible to determine early warning prompts in real time.

[0013] In addition, by introducing an orthogonal processing layer in the fake audio recognition model, orthogonal constraints are used to prevent parameters from being too complex, which can effectively avoid gradient explosion and reduce overfitting. At the same time, the orthogonal constraints can also enable the model network to have stronger separation capabilities in each feature dimension, thereby improving the generalization ability of the fake audio recognition model in the face of various call scenarios.

[0014] In addition, the residual convolutional layer of the forged audio recognition model ensures that the model can capture sufficient temporal context information to detect forged speech, expands the receptive field of the forged audio detection network, enhances the model's ability to extract long-term forged audio features, and captures forgery traces throughout the speech in fewer stacked layers. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1is a flowchart of an example of a method for identifying forged audio based on improved RawNet2 of the present invention;

[0016] Figure 2 is a schematic diagram of a streaming link in an application example of a forged audio identification method based on improved RawNet2 of the present invention;

[0017] Figure 3 It is a schematic diagram of the principle of a forged audio recognition model in the forged audio recognition method based on improved RawNet2 of the present invention;

[0018] Figure 4 It is a partial principle schematic diagram of an orthogonal processing layer in a forged audio recognition model in the forged audio recognition method based on improved RawNet2 of the present invention;

[0019] Figure 5 This is an example diagram of a warning pop-up window of the forged audio identification method based on improved RawNet2 of the present invention;

[0020] Figure 6 It is a structural block diagram of the forged audio recognition system based on improved RawNet2 of the present invention;

[0021] Figure 7 is a schematic structural diagram of an electronic device embodiment according to the present invention;

[0022] Figure 8 It is a schematic diagram of the structure of a computer-readable medium embodiment according to the present invention. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In view of the above problems, the present invention proposes a method and system for identifying forged audio based on improved RawNet2. The method collects forged audio of multiple dimensions in call scenarios to establish a training data set, wherein the multiple dimensions include live customer service calls, synthesized forged audio, call environment noise, signal noise, and music songs; based on the improved RawNet2 neural network, a forged audio recognition model is constructed, and the forged audio recognition model is trained using the training data set to obtain a trained forged audio recognition model, which can retain more dimensional audio original information, so that the model network learns low-dimensional embedding for specific tasks, and can improve the recognition accuracy of the model for forged audio; the original call audio of a preset length is intercepted from the call audio stream to be processed to input the forged audio recognition model to obtain a recognition result, and the forged audio recognition model can directly process the original waveform, so that the network of the forged audio recognition model can learn the forged audio features customized for the forged audio detection task, which can effectively improve the model detection effect; according to the obtained recognition result, it is possible to determine the early warning prompt in real time.

[0025] Example 1

[0026] Refer to the following Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 , the contents of the present invention will be described in detail.

[0027] Figure 1 It is a flowchart of an example of the method for identifying forged audio based on improved RawNet2 of the present invention.

[0028] like Figure 1 As shown, in step S101, forged audio of various dimensions in call scenarios are collected to establish a training data set, wherein the various dimensions include real customer service calls, synthesized forged audio, call environment noise, signal noise, and music songs.

[0029] In a specific application example, the method of the present invention is applied to a forged audio recognition system, which can be connected to the call messaging platform of each operator, that is, the forged audio recognition system can access the audio stream from the call messaging platform of each operator. A streaming connection is established with the forged audio recognition system via the RTMP (Real-Time Messaging Protocol) audio streaming protocol. For details, see Figure 2 Specifically, the forged audio identification system receives the call audio stream from the call message platform, pre-processes the call audio stream, and then performs model recognition to determine whether to send forged call audio warning information to the user.

[0030] Specifically, forged audio of various dimensions in call scenarios are collected to establish a training data set. The various dimensions include real customer service calls, synthesized forged audio, call environment noise, signal noise, and music songs.

[0031] In a specific implementation, data from multiple dimensions, such as live customer service calls, fake audio synthesized by 19 fake audio methods, call environment noise, signal noise, and music songs, are collected in call scenarios to establish a training data set. The specific fake audio methods include: a model that directly generates audio waveforms using a neural network, an audio encoder, a method for synthesizing audio using the generative adversarial network (GAN) and vocoder technology, an iterative algorithm for reconstructing audio waveforms from spectrograms, and so on.

[0032] In addition, we also collect fake audios publicly available online to build training data sets, such as ASVspoof2019, Aishell, Kespeech, Codefake, and other data.

[0033] Specifically, corresponding fake / real audio file names are created respectively, bonafide is used to represent real call audio, and Spoof is used to represent fake audio.

[0034] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0035] Next, in step S102, a forged audio recognition model is constructed based on the improved RawNet2 neural network, and the forged audio recognition model is trained using the training data set to obtain a trained forged audio recognition model, specifically including: introducing an orthogonal processing layer to split the original waveform of the call audio stream to extract more forged audio features; introducing a temporal convolutional network to expand the receptive field of the forged audio detection network to capture forged audio features.

[0036] Based on the improved RawNet2 neural network, a fake audio recognition model is constructed. For details, see Figure 3 and Figure 4 .

[0037] In order to retain more dimensions of original audio information, enable the model network to learn low-dimensional embedding for specific tasks (specifically, forged audio recognition tasks), and improve the model's recognition accuracy for forged audio, this paper improves the RawNet2 network by introducing orthogonal constraints and TCN temporal convolutional networks in RawNet2 to improve the model's ability to distinguish forged audio.

[0038] Specific improvements include: introducing an orthogonal processing layer to split the original waveform of the call audio stream to extract more forged audio features; introducing a temporal convolutional network to expand the receptive field of the forged audio detection network to capture forged audio features.

[0039] like Figure 3 As shown, the forged audio recognition model includes an orthogonal processing layer (corresponding to the convolution layer in Table 1 below), a residual convolution layer (including a temporal convolution network) and an output layer, wherein the orthogonal processing layer includes at least three channels, each channel corresponds to a convolution part, each channel is orthogonally processed, and then the feature data of the three channels are aggregated as the output of the orthogonal processing layer; the residual convolution layer includes a convolution layer, a first residual layer (corresponding to residual layer 1 in Table 1 below), a second residual layer (corresponding to residual layer 2 in Table 1 below), and a gated recurrent unit in sequence. The convolution layer in the residual convolution layer includes a convolution module, a maximum pooling layer, a batch normalization layer, and an activation function layer. Each residual layer includes a batch normalization layer, an activation function layer, a convolution layer, a batch normalization layer, a maximum pooling layer, and a fully connected layer. For details, please refer to Table 1 below.

[0040] It should be noted that in the present invention, batch normalization (BN) in the forged audio recognition model is a part used to accelerate neural network training and improve stability. Specifically, by normalizing the input in each layer of the network, the gradient vanishing and gradient exploding problems of each network layer are further alleviated, and at the same time, each network layer is allowed to use a higher learning rate, which can further improve the recognition accuracy of the forged audio recognition model.

[0041] Table 1

[0042]

[0043]

[0044] Table 1 is an example showing a specific implementation of the residual convolution layer of the fake audio recognition model.

[0045] The forged audio recognition model can directly process the original waveform, so that the network of the forged audio recognition model can learn forged audio features customized for the forged audio detection task, which can effectively improve the model detection effect.

[0046] It should be noted that RawNet2 is a deep learning model specifically designed for speech processing tasks, especially speaker verification and speaker recognition, and has demonstrated excellent performance in multiple benchmarks.

[0047] Specifically, an orthogonal processing layer is introduced to split the original waveform of the call audio stream to extract more forged audio features. The number of channels of the orthogonal processing layer is a multiple of three, specifically including three and six.

[0048] In this example, the orthogonal processing layer has three channels, which split the original waveform of the call audio stream into a low-frequency part, a mid-frequency part, and a high-frequency part.

[0049] In the orthogonal processing layer, orthogonal processing is performed in the process of splitting the original waveform of the call audio stream.

[0050] In an optional embodiment, in the orthogonal processing layer of the forged audio recognition model, the convolution operation is regarded as a matrix-vector multiplication by applying the orthogonality property of the matrix to the orthogonal learning process of the convolution kernel, wherein the kernel matrix M is generated by the convolution kernel K. Specifically, the convolution expression Conv(K,X) is converted into a DBT matrix vector representation by a matrix construction method, that is, the convolution output of the orthogonal processing layer is represented as:

[0051] Y=Mx

[0052] Among them, Y represents the convolution represented by the DBT matrix-vector; M is the DBT matrix, and the shape of the DBT matrix is ​​determined according to the number of input channels, the number of output channels, the length of the input feature map, and the length of the output feature map; x and y represent the input tensor and the output tensor, respectively, and the shape of the DBT matrix, and the rows of the DBT matrix are orthogonalized. After the low-frequency part, the medium-frequency part, and the high-frequency part of the call audio stream are orthogonalized, three matrices corresponding to the low-frequency part, the medium-frequency part, and the high-frequency part are obtained.

[0053] The low-frequency part, the intermediate-frequency part, and the high-frequency part are output from the orthogonal processing layer after being convolved, orthogonally processed, and aggregated (specifically, the three matrices are merged) in their respective channels in the orthogonal processing layer.

[0054] By introducing an orthogonal processing layer into the fake audio recognition model, orthogonal constraints are used to prevent parameters from being too complex, which can effectively avoid gradient explosion and reduce overfitting. At the same time, the orthogonal constraints can also make the model network have stronger separation capabilities in each feature dimension (specifically, the feature dimensions of the low-frequency part, the medium-frequency part, and the high-frequency part), thereby improving the generalization ability of the fake audio recognition model in the face of various call scenarios.

[0055] Furthermore, the output of the orthogonal processing layer is used as the input of the residual convolution layer.

[0056] Specifically, the forged audio recognition system receives real-time audio stream data generated by the user during a call, and intercepts the audio data stream within a specified time period from the real-time audio stream data as the input of the forged audio recognition model. Specifically, the intercepted audio data stream within the specified time period is input into the orthogonal processing layer, split into a low-frequency part, a medium-frequency part, and a high-frequency part through at least three channels, and different forged audio features are extracted therefrom, and the different forged audio features are merged as the output of the orthogonal processing layer.

[0057] The output of the orthogonal processing layer is input into the residual convolution layer, and a specific convolution is used to adjust the output channel of the residual convolution layer to match the input channel. After extracting the forged audio features in the deep time dimension, the output of the residual convolution layer is obtained.

[0058] It should be noted that in the invention, in the network structure of the forged audio recognition model, specifically in the convolutional deep neural network, depth usually refers to the number of convolutional layers or the number of layers of the network, and can also be used to describe the number of channels of the feature map (i.e. the third dimension of the tensor, usually referred to as the number of channels). Specifically, the structure of the residual convolutional layer is modified on the basis of the original network, and the first residual layer is stacked twice and the second residual layer is stacked four times to increase the network depth. The optimized network depth is 1+2*2+4*2=13 layers, and the output layer size in the original network is increased from 1*128 to 1*1024 to capture longer audio dependencies, that is, to capture forged audio features of more dimensions.

[0059] The temporal convolutional network of the residual convolutional layer (specifically the TCN temporal convolutional network) can capture longer-term audio dependencies by increasing the network depth, which can enhance the modeling ability of the forged audio recognition model for long-term call data. And because the TCN temporal convolutional network can process data in parallel, it significantly improves the recognition speed of forged audio in call scenarios, thereby quickly and in real time identifying forged call risks.

[0060] In another optional implementation, an orthogonal processing layer is introduced to split the original waveform of the call audio stream to extract more forged audio features. The orthogonal processing in the orthogonal processing layer is an optimized orthogonal processing. In the orthogonal processing layer, at least three channels are split into a low-frequency part, a mid-frequency part, and a high-frequency part, and different forged audio features are extracted from them, and the different forged audio features are merged as the output of the orthogonal processing layer. Three channels correspond to one convolution part and three orthogonal processing parts.

[0061] Specifically, in the orthogonal processing layer (specifically, in the orthogonal processing part of the orthogonal processing layer), in the process of splitting the original waveform of the call audio stream, the following expression is used to perform orthogonal calculation to obtain the convolution output of the orthogonal processing layer:

[0062] Y'=Conv(K,K,padding=P, stride=S)=I r0

[0063] Among them, Y' represents the convolution output obtained by orthogonal calculation in the orthogonal processing layer; K represents the size of the convolution kernel of the orthogonal processing layer, S represents the step size, and P represents padding; Ir0 is a tensor parameter, the center of which is an n×n unit matrix, and the rest is filled with zeros.

[0064] Further comprising the step of optimizing the orthogonal calculation. By minimizing the difference between Y'=Conv(K, K, padding=P, stride=S) and Ir0, an orthogonal convolution can be obtained, and the following expression is used to represent the loss function of the obtained orthogonal convolution:

[0065] min k L orth =││Y'-I r0 ││ 2 F

[0066] Among them, min k L orth Represents the difference between Y'=Conv(K,K,padding=P,stride=S) and Ir0, Y'=Conv(K,K,padding=P,stride=S), Y' represents the convolution output obtained by orthogonal calculation in the orthogonal processing layer, K represents the size of the convolution kernel of the orthogonal processing layer, S represents the step size, and P represents padding; Ir0 is a tensor parameter, the center of which is an n×n unit matrix, and the rest is filled with zeros, n is a positive integer; F represents the Frobenius norm, which is a matrix norm used to measure the square root of the sum of the squares of all elements of the matrix, that is, the "size" of the matrix. By calculating the square of the Frobenius norm, the orthogonal loss quantifies the degree to which Y' deviates from orthogonality, where the calculated min k L orth The smaller it is, the closer the convolution output Y' is to the orthogonal characteristic.

[0067] After the low frequency part, the intermediate frequency part, and the high frequency part of the call audio stream are orthogonalized (specifically, orthogonal calculation), three matrices corresponding to the low frequency part, the intermediate frequency part, and the high frequency part are obtained. After the low frequency part, the intermediate frequency part, and the high frequency part are convolved, orthogonalized (specifically, orthogonal calculation) and aggregated (specifically, merging the three matrices) in their respective channels in the orthogonal processing layer, they are output from the orthogonal processing layer.

[0068] After the orthogonal processing layer, the output of the orthogonal processing layer is used as the input of the residual convolution layer. The output of the residual convolution layer is then obtained to further obtain the output of the forged audio recognition model.

[0069] It should be noted that the residual convolution layer (including TCN temporal convolutional network) is introduced on the original RawNet network structure. The convolution kernel size in the residual convolution layer is 1*3. The receptive field of the model for forged audio is improved by expanding the convolution. Therefore, the model's ability to extract long-term forged audio features can be enhanced.

[0070] The residual convolution layer first uses batch normalization and ReLU activation function, followed by dilated convolution. Next, the output channel is adjusted and reshaped using 1x1 convolution operations to further accelerate convergence and facilitate the training of deeper models. The residual convolution layer ensures that the model can capture enough temporal context information to detect forged speech. It expands the receptive field of the forged audio detection network, enhances the model's ability to extract long-term forged audio features, and captures forgery traces throughout the speech in fewer stacked layers.

[0071] When the forged audio recognition model is trained using the training data set established in step S101, the bonafide value is replaced with 0, the Spoof is replaced with 1, and the loss function is calculated using the result between 0 and 1 calculated by the forged audio recognition model and the true value of the training data.

[0072] Specifically, when the model training rounds are between 150 and 260 and the model accuracy exceeds 90%, the model iteration process, that is, the model training process, is terminated.

[0073] Preferably, when the model training rounds are 200 and the model accuracy exceeds 90%, the model iteration process ends.

[0074] Thus, a trained forged audio recognition model is obtained.

[0075] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0076] Next, in step S103, the original call audio of a preset length is intercepted from the call audio stream to be processed to input into the forged audio recognition model to obtain a recognition result.

[0077] Specifically, the forged audio identification system receives a call audio stream to be processed from a call message platform of operator a.

[0078] Through the preset transmission block (chunk) and message (message) size, a specified time period (for example, 5s) in the preset length of the call audio stream to be processed is intercepted as the model input, and input into the forged audio recognition model to obtain the recognition result.

[0079] Optionally, the specified time period includes 1s to 5s, specifically 1s, 2s, 3s, 4s and 5s.

[0080] The recognition result is whether the audio is forged, whether it is forged, or the category of forged audio.

[0081] Specifically, the forged audio recognition model outputs the following recognition results corresponding to the call audio stream to be processed: whether it is forged audio and the category of forged audio.

[0082] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0083] Next, in step S104, a warning prompt is determined in real time according to the obtained recognition result.

[0084] Specifically, according to the output recognition result, an early warning method is selected for early warning processing. When the recognition result obtained by the model is forged audio, the forged audio recognition system determines that the user corresponding to the audio stream to be processed is making a forged audio call. Then a mobile phone pop-up window, voice broadcast, etc. are selected to send early warning information to the user. For example, it includes a pop-up window for fraudulent message warning, ** reminding you that the call is a fraudulent act of AI forged audio call, please be vigilant and protect the safety of personal property, etc. The warning information can be found in Figure 5 .

[0085] By providing real-time warnings and intercepting fraud and harassing calls generated by forged audio, the user experience can be improved.

[0086] In order to verify the forged audio recognition effect of the method of the present invention, a forged audio recognition detection test was conducted. The international challenge ASVspoof2019 dataset was used for testing. For example, the ASVspoof2019 dataset is a dataset specifically used for research and development to combat voice forgery attacks, aiming to promote the development of automatic speaker verification systems to combat forgery threats. The forged audio recognition model of the present invention is tested in the ASVspoof2019 dataset.

[0087] The verification results are shown in Table 2 below:

[0088] Table 2

[0089]

[0090] Table 2 shows the comparison results of ablation tests between the method of the present invention and other comparative methods for improving the RawNet2 model.

[0091] As can be seen from Table 2, both orthogonal regularization and temporal convolutional network (TCN) have a positive effect on the processing of speech signals. Experimental comparison shows that the equal error rate of the method of the present invention is 1.23%, which is the lowest among all methods. Therefore, the improved RawNet2 fake audio recognition model has the best performance. This is because the orthogonal processing layer can reduce the correlation between filters, thereby improving the generalization ability of the model. The TCN temporal network can capture long-term dependencies in speech signals. All models tested performed better than the baseline RawNet model. Therefore, by introducing orthogonal constraints and TCN temporal convolutional networks in RawNet2 to construct a fake audio recognition model based on the RawNet2 network, the discrimination ability of the model can be effectively improved, and the equal error rate is significantly reduced by 60%-70% compared with other models.

[0092] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0093] In addition, the drawings are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the drawings do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0094] Compared with the prior art, the present invention establishes a training data set by collecting forged audio of multiple dimensions in call scenarios, wherein the multiple dimensions include real customer service calls, synthesized forged audio, call environment noise, signal noise, and music songs; constructs a forged audio recognition model based on an improved RawNet2 neural network, and uses the training data set to train the forged audio recognition model to obtain a trained forged audio recognition model, which can retain more dimensions of original audio information, so that the model network learns low-dimensional embedding for specific tasks, significantly reducing the equal error rate by 60%-70%, thereby improving the model's recognition accuracy for forged audio; intercepts original call audio of a preset length from the call audio stream to be processed to input the forged audio recognition model to obtain a recognition result, and can directly process the original waveform, so that the network of the forged audio recognition model can learn forged audio features customized for forged audio detection tasks, which can effectively improve the model detection effect and can more effectively identify forged audio; according to the obtained recognition result, real-time determination of early warning prompts can be achieved.

[0095] In addition, by introducing an orthogonal processing layer in the fake audio recognition model, orthogonal constraints are used to prevent parameters from being too complex, which can effectively avoid gradient explosion and reduce overfitting. At the same time, the orthogonal constraints can also enable the model network to have stronger separation capabilities in each feature dimension, thereby improving the generalization ability of the fake audio recognition model in the face of various call scenarios.

[0096] In addition, the residual convolutional layer of the forged audio recognition model ensures that the model can capture sufficient temporal context information to detect forged speech, expands the receptive field of the forged audio detection network, enhances the model's ability to extract long-term forged audio features, and captures forgery traces throughout the speech in fewer stacked layers.

[0097] Example 2

[0098] The following is a system embodiment of the present invention, which can be used to implement the method embodiment of the present invention. For details not disclosed in the system embodiment of the present invention, please refer to the method embodiment of the present invention.

[0099] Figure 6 It is a structural diagram of an example of a forged audio recognition system based on improved RawNet2 of the present invention.

[0100] Reference Figure 6 The second aspect of the present disclosure provides a forged audio recognition system based on improved RawNet2, which adopts the forged audio recognition method based on improved RawNet2 described in the first aspect of the present disclosure. The forged audio recognition system 400 includes a data collection module 410, a model building module 420, a determination module 430 and an early warning module 440.

[0101] Specifically, the data collection module 410 is used to collect forged audio of various dimensions in call scenarios and establish a training data set, wherein the various dimensions include live customer service calls, synthesized forged audio, call environment noise, signal noise, and music songs. The model construction module 420 constructs a forged audio recognition model based on the improved RawNet2 neural network, and uses the training data set to train the forged audio recognition model to obtain a trained forged audio recognition model, which specifically includes: introducing an orthogonal processing layer to split the original waveform of the call audio stream to extract more forged audio features; introducing a time convolutional network to expand the receptive field of the forged audio detection network to capture forged audio features. The determination module 430 is used to intercept the original call audio of a preset length from the call audio stream to be processed to input the forged audio recognition model to obtain a recognition result; the warning module 440 determines the warning prompt in real time based on the obtained recognition result.

[0102] According to an optional implementation, the original waveform of the call audio stream is split by introducing an orthogonal processing layer to extract more forged audio features, including: in the orthogonal processing layer, by applying the orthogonality property of the matrix to the orthogonal learning process of the convolution kernel, the convolution operation is regarded as a matrix-vector multiplication, wherein the kernel matrix M is generated by the convolution kernel K, and specifically the convolution expression Conv(K,X) is converted into a DBT matrix vector representation by a matrix construction method, that is, the convolution output of the orthogonal processing layer is represented as:

[0103] Y=Mx

[0104] Among them, Y represents the convolution represented by the DBT matrix-vector; M is the DBT matrix, and the shape of the DBT matrix is ​​determined according to the number of input channels, the number of output channels, the length of the input feature map, and the length of the output feature map; x and y represent the input tensor and the output tensor respectively, among which, the shape of the DBT matrix, and the rows of the DBT matrix are orthogonalized.

[0105] According to an optional implementation, the original waveform of the call audio stream is split by introducing an orthogonal processing layer to extract more forged audio features.

[0106] In the orthogonal processing layer, the following expression is used to perform orthogonal calculation to obtain the convolution output of the orthogonal processing layer:

[0107] Y'=Conv(K,K,padding=P, stride=S)=I r0

[0108] Among them, Y' represents the convolution output obtained by orthogonal calculation in the orthogonal processing layer; K represents the size of the convolution kernel of the orthogonal processing layer, S represents the step size, and P represents padding; Ir0 is a tensor parameter, the center of which is an n×n unit matrix, and the rest is filled with zeros, and n is a positive integer.

[0109] According to an optional implementation manner, the step of optimizing orthogonal calculation is further included, specifically including:

[0110] By minimizing the difference between Y'=Conv(K,K,padding=P,stride=S) and Ir0, an orthogonal convolution can be obtained. The following expression is used to represent the loss function of the obtained orthogonal convolution:

[0111] min k L orth =││Y'-I r0 ││ 2 F

[0112] Among them, min k L orthrepresents the difference between Y'=Conv(K,K,padding=P,stride=S) and Ir0, Y'=Conv(K,K,padding=P,stride=S), Y' represents the convolution output obtained by orthogonal calculation in the orthogonal processing layer, K represents the size of the convolution kernel of the orthogonal processing layer, S represents the step size, and P represents padding; Ir0 is a tensor parameter, the center of which is an n×n unit matrix, and the rest is filled with zeros; F represents the Frobenius norm, which is a matrix norm used to measure the square root of the sum of the squares of all elements of the matrix, that is, the "size" of the matrix. By calculating the square of the Frobenius norm, the orthogonal loss quantifies the degree to which Y' deviates from orthogonality.

[0113] According to an optional implementation, it further includes: the forged audio recognition system receives real-time audio stream data generated by the user during a call, and intercepts the audio data stream within a specified time period from the real-time audio stream data as the input of the forged audio recognition model, specifically inputting the intercepted audio data stream within the specified time period into the orthogonal processing layer, splitting it into a low-frequency part, a medium-frequency part, and a high-frequency part through at least three channels, and extracting different forged audio features therefrom, merging the different forged audio features as the output of the orthogonal processing layer; inputting the output of the orthogonal processing layer into the residual convolution layer, using a specific convolution to adjust the output channel of the residual convolution layer to match the input channel, and obtaining the output of the residual convolution layer after extracting the forged audio features in a deep time dimension.

[0114] According to an optional implementation manner, the specified time period includes 1s to 5s.

[0115] The number of channels in the orthogonal processing layer is a multiple of three, specifically including three and six.

[0116] According to an optional implementation, it further includes: the forged audio recognition model outputs the following recognition results corresponding to the call audio stream to be processed: whether it is forged audio, and the category of forged audio.

[0117] According to the output recognition results, select the warning method for warning processing

[0118] It should be noted that the forged audio recognition method based on improved RawNet2 executed by the forged audio recognition system based on improved RawNet2 in Example 2 is the same as the forged audio recognition method based on improved RawNet2 in Example 1, and therefore, the description of the same parts is omitted.

[0119] Compared with the prior art, the present invention establishes a training data set by collecting forged audio of multiple dimensions in call scenarios, wherein the multiple dimensions include real customer service calls, synthesized forged audio, call environment noise, signal noise, and music songs; constructs a forged audio recognition model based on an improved RawNet2 neural network, and uses the training data set to train the forged audio recognition model to obtain a trained forged audio recognition model, which can retain more dimensions of original audio information, so that the model network learns low-dimensional embedding for specific tasks, significantly reducing the equal error rate by 60%-70%, thereby improving the model's recognition accuracy for forged audio; intercepts original call audio of a preset length from the call audio stream to be processed to input the forged audio recognition model to obtain a recognition result, and can directly process the original waveform, so that the network of the forged audio recognition model can learn forged audio features customized for forged audio detection tasks, which can effectively improve the model detection effect and can more effectively identify forged audio; according to the obtained recognition result, real-time determination of early warning prompts can be achieved.

[0120] In addition, by introducing an orthogonal processing layer in the fake audio recognition model, orthogonal constraints are used to prevent parameters from being too complex, which can effectively avoid gradient explosion and reduce overfitting. At the same time, the orthogonal constraints can also enable the model network to have stronger separation capabilities in each feature dimension, thereby improving the generalization ability of the fake audio recognition model in the face of various call scenarios.

[0121] In addition, the residual convolutional layer of the forged audio recognition model ensures that the model can capture sufficient temporal context information to detect forged speech, expands the receptive field of the forged audio detection network, enhances the model's ability to extract long-term forged audio features, and captures forgery traces throughout the speech in fewer stacked layers.

[0122] Figure 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

[0123] like Figure 7 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0124] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0125] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0126] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0127] It should be understood that Figure 7 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0128] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Figure 8 As shown, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of commands to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.

[0129] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0130] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0131] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0132] The computer-readable medium carries one or more programs (eg, computer-executable programs). When the one or more programs are executed by a device, the computer-readable medium implements the method of the present disclosure.

[0133] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0134] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several commands to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.

[0135] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structure, configuration or implementation method described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A forged audio recognition method based on improved RawNet2, characterized in that: include: Collect fake audio of various dimensions in call scenarios to establish a training data set. The multiple dimensions include real customer service calls, synthesized fake audio, call environment noise, signal noise, and music songs; Based on the improved RawNet2 neural network, a forged audio recognition model is constructed, and the forged audio recognition model is trained using the training data set to obtain a trained forged audio recognition model, specifically including: introducing an orthogonal processing layer to split the original waveform of the call audio stream to extract more forged audio features; introducing a temporal convolutional network to expand the receptive field of the forged audio detection network to capture the forged audio features; Intercepting a preset length of original call audio from the call audio stream to be processed to input the forged audio recognition model to obtain a recognition result; According to the obtained recognition results, early warning prompts are determined in real time.

2. The forged audio identification method based on improved RawNet2 according to claim 1 is characterized in that: The orthogonal processing layer is introduced to split the original waveform of the call audio stream to extract more forged audio features, including: In the orthogonal processing layer, by applying the orthogonality property of the matrix to the orthogonal learning process of the convolution kernel, the convolution operation is regarded as a matrix-vector multiplication, where the kernel matrix M is generated by the convolution kernel K. Specifically, the matrix construction method is used to convert the convolution expression Conv(K,X) into a DBT matrix vector representation, that is, the convolution output of the orthogonal processing layer is expressed as: Y=Mx Among them, Y represents the convolution represented by the DBT matrix-vector; M is the DBT matrix, and the shape of the DBT matrix is ​​determined according to the number of input channels, the number of output channels, the length of the input feature map, and the length of the output feature map; x and y represent the input tensor and the output tensor respectively, among which, the shape of the DBT matrix, and the rows of the DBT matrix are orthogonalized.

3. The forged audio identification method based on improved RawNet2 according to claim 1, characterized in that: The orthogonal processing layer is introduced to split the original waveform of the call audio stream to extract more forged audio features, including: In the orthogonal processing layer, the following expression is used to perform orthogonal calculation to obtain the convolution output of the orthogonal processing layer: Y’=Conv(K,K,padding=P,stride=S)=I r0 Among them, Y' represents the convolution output obtained by orthogonal calculation in the orthogonal processing layer; K represents the size of the convolution kernel of the orthogonal processing layer, S represents the step size, and P represents padding; Ir0 is a tensor parameter, the center of which is an n×n unit matrix, and the rest is filled with zeros, and n is a positive integer.

4. The forged audio identification method based on improved RawNet2 according to claim 3 is characterized in that: The method further includes the steps of optimizing the orthogonal calculation, specifically including: By minimizing the difference between Y'=Conv(K,K,padding=P,stride=S) and Ir0, an orthogonal convolution can be obtained. The following expression is used to represent the loss function of the obtained orthogonal convolution: my k L orth =││Y'-I r0 ││ 2 F Among them, min k L orth represents the difference between Y'=Conv(K,K,padding=P,stride=S) and Ir0, Y'=Conv(K,K,padding=P,stride=S), Y' represents the convolution output obtained by orthogonal calculation in the orthogonal processing layer, K represents the size of the convolution kernel of the orthogonal processing layer, S represents the step size, and P represents padding; Ir0 is a tensor parameter, the center of which is an n×n unit matrix, and the rest is filled with zeros, n is a positive integer; F represents the Frobenius norm, which is a matrix norm used to measure the square root of the sum of the squares of all elements of the matrix, that is, the "size" of the matrix. By calculating the square of the Frobenius norm, the orthogonal loss quantifies the degree to which Y' deviates from orthogonality.

5. The forged audio identification method based on improved RawNet2 according to claim 2 or 3, characterized in that: Further including: The forged audio recognition system receives the real-time audio stream data generated by the user during the call, and intercepts the audio data stream within a specified time period from the real-time audio stream data as the input of the forged audio recognition model, specifically inputs the intercepted audio data stream within the specified time period into the orthogonal processing layer, splits it into a low-frequency part, a medium-frequency part, and a high-frequency part through at least three channels, and extracts different forged audio features from them, and merges the different forged audio features as the output of the orthogonal processing layer; The output of the orthogonal processing layer is input into the residual convolution layer, and a specific convolution is used to adjust the output channel of the residual convolution layer to match the input channel. After extracting the forged audio features in the deep time dimension, the output of the residual convolution layer is obtained.

6. The forged audio identification method based on improved RawNet2 according to claim 5 is characterized in that: include: The specified time period includes 1s to 5s; The number of channels in the orthogonal processing layer is a multiple of three, specifically including three and six.

7. The forged audio identification method based on improved RawNet2 according to claim 5 is characterized in that: Further including: The forged audio recognition model outputs the following recognition results corresponding to the call audio stream to be processed: whether it is forged audio and the forged audio category; According to the output recognition result, a warning method is selected to perform warning processing.

8. A forged audio recognition system based on improved RawNet2, characterized in that: It implements the forged audio recognition method based on improved RawNet2 as described in any one of claims 1 to 7, and the forged audio recognition system comprises: A data collection module is used to collect forged audio of various dimensions in call scenarios and establish a training data set. The multiple dimensions include real customer service calls, synthesized forged audio, call environment noise, signal noise, and music songs; The model building module builds a forged audio recognition model based on the improved RawNet2 neural network, and uses the training data set to train the forged audio recognition model to obtain a trained forged audio recognition model, specifically including: introducing an orthogonal processing layer to split the original waveform of the call audio stream to extract more forged audio features; introducing a temporal convolutional network to expand the receptive field of the forged audio detection network to capture the forged audio features; A determination module, used to intercept the original call audio of a preset length from the call audio stream to be processed, so as to input the forged audio recognition model and obtain a recognition result; The early warning module determines early warning prompts in real time based on the obtained recognition results.

9. The forged audio identification system based on improved RawNet2 according to claim 8, characterized in that: include: In the orthogonal processing layer, by applying the orthogonality property of the matrix to the orthogonal learning process of the convolution kernel, the convolution operation is regarded as a matrix-vector multiplication, where the kernel matrix M is generated by the convolution kernel K. Specifically, the matrix construction method is used to convert the convolution expression Conv(K,X) into a DBT matrix vector representation, that is, the convolution output of the orthogonal processing layer is expressed as: Y=Mx Among them, Y represents the convolution represented by the DBT matrix-vector; M is the DBT matrix, and the shape of the DBT matrix is ​​determined according to the number of input channels, the number of output channels, the length of the input feature map, and the length of the output feature map; x and y represent the input tensor and the output tensor respectively, among which, the shape of the DBT matrix, and the rows of the DBT matrix are orthogonalized.

10. The forged audio identification system based on improved RawNet2 according to claim 8, characterized in that: include: In the orthogonal processing layer, the following expression is used to perform orthogonal calculation to obtain the convolution output of the orthogonal processing layer: Y’=Conv(K,K,padding=P,stride=S)=I r0 Among them, Y' represents the convolution output obtained by orthogonal calculation in the orthogonal processing layer; K represents the size of the convolution kernel of the orthogonal processing layer, S represents the step size, and P represents padding; Ir0 is a tensor parameter, the center of which is an n×n identity matrix, and the rest is filled with zeros, and n is a positive integer.

Images