Sound authenticity identification method and system based on multi-modal feature deep interactive fusion

Abstract

The invention discloses a sound authenticity identification method based on multi-modal feature deep interactive fusion. According to the method, a double-flow architecture is adopted, and a pre-trained BEATs model and a CNN14 network are respectively utilized to extract Transform sequence features and convolution time-frequency embedding features of an audio; an adaptive MobileFormer fusion device is innovatively proposed, an original MobileFormer structure used in the image field is transformed into a bidirectional cross-modal interaction module suitable for one-dimensional time sequence audio features, dynamic complementary modeling of local details and global semantic features is achieved through a cross attention mechanism, and dynamic nonlinearity is introduced to activate and enhance the expression ability; and the fused enhanced features are input into a hierarchical graph attention network ASSIST, and high-order semantic modeling and authenticity classification are completed by combining spectrogram and time sequence double-flow reasoning. Experimental results show that the performance of the method on an ASVspoof2021LA data set is superior to that of an existing baseline model. The method can effectively detect AI generation voice, replay attack and other forged voice, and is suitable for a voice authentication system.

Description

Technical Field

[0001] This invention relates to the fields of voice security, multimodal machine learning, and artificial intelligence content authentication, and in particular to a method and system for voice authenticity recognition based on a cross-modal deep interactive fusion mechanism. This method innovatively modifies the advanced visual architecture MobileFormer to achieve one-dimensional sequence-level deep fusion of CNN and Transformer modalities in audio, making it suitable for high-security voice identity authentication, media content authentication, and financial anti-fraud scenarios. Background Technology

[0002] With the development of deep generative models (such as WaveNet, Tacotron, Voice Conversion, and Speech Diffusion), the ability to generate high-quality forged speech has significantly improved, posing a security challenge to traditional voiceprint recognition systems. This type of forged speech closely resembles real speech in terms of semantic coherence and auditory naturalness, and can circumvent detection mechanisms based on traditional acoustic features such as spectrum, fundamental frequency, and formants, thus posing a threat to the security of voice authentication systems.

[0003] Existing methods for identifying the authenticity of audio mainly fall into two categories:

[0004] Methods based on handcrafted features, such as Cochleagram Cepstral Coefficients (CQCC) and Linear Frequency Cepstral Coefficients (LFCC), rely on manually designed features to distinguish between genuine and fake data.

[0005] Deep learning-based end-to-end models, such as ResNet, EfficientNet, and RawNet, learn discriminative representations directly from the original waveforms or spectrograms.

[0006] However, the above method has the following limitations:

[0007] Single-modal modeling capabilities are limited: AI-generated speech may be continuous and natural in the time domain, but high-frequency artifacts or local inconsistencies may still exist in the frequency domain or time-frequency joint space. A single modality is difficult to fully capture such traces.

[0008] The multimodal fusion mechanism is superficial: existing fusion methods mostly use feature splicing or weighted averaging, lacking deep interaction mechanisms, resulting in the semantic differences between different modalities not being effectively bridged;

[0009] Insufficient long-term dependency modeling: Some models have a weak ability to model global contextual relationships across frames, making it difficult to identify widely distributed forgery patterns.

[0010] In recent years, although some studies have attempted to use dual-stream structures (such as CNN+Transformer) for multimodal modeling, their fusion methods have mostly remained at the feature-level concatenation or decision-level weighting, lacking dynamic interaction mechanisms at the intermediate representation level. For example, some studies have used CNN to extract Mel spectrogram features, while using Transformer to process the original waveform, and finally fused them at the classification layer (reference: Penghui Wen et al., "Robust Audio Anti-Spoofing..."). Such methods are essentially still late fusion strategies, with shallow intermodal information interaction levels, making it difficult to bridge the "semantic gap" between local details and global semantics.

[0011] Furthermore, although some architectures supporting deep cross-modal interaction have emerged in recent years (such as MobileFormer), their original design is geared towards two-dimensional image spatial feature maps. Directly applying them to one-dimensional audio time series will lead to problems such as dimension mismatch and position modeling failure, making it impossible to achieve effective fusion.

[0012] Therefore, designing a deep cross-modal interaction mechanism suitable for one-dimensional time-series signals to achieve complementary integration of CNN's local perception capability and Transformer's global modeling capability has become a key technical bottleneck in improving the performance of sound authenticity recognition. Summary of the Invention

[0013] This invention provides a method and system for identifying the authenticity of audio based on deep interactive fusion of multimodal features, aiming to solve the problems of shallow multimodal fusion mechanisms, insufficient interaction, and limited discrimination capabilities in existing technologies.

[0014] To achieve the above objectives, this invention is the first to migrate the MobileFormer architecture, originally used in the image field, to the audio authenticity detection task, and to make key structural modifications to it, proposing an "AdaptedMobileFormerBlock" to support bidirectional deep cross-modal interaction of one-dimensional temporal audio features.

[0015] This invention discloses a method for identifying authentic and fake audio based on deep interactive fusion of multimodal features, comprising:

[0016] Step 1: Construct a dual-stream feature extraction path for the input audio signal, and extract the Transformer sequence features and convolutional time-frequency embedding features of the audio using the pre-trained BEATs model and the CNN14 network respectively;

[0017] Step 2: Input the two types of features into the Adapted MobileFormer fusion engine and fuse them through a bidirectional cross-attention mechanism. During the fusion stage, the two modal features interact bidirectionally through multiple AdaptedMobileFormerBlocks to generate an enhanced sequence representation. This Adapted MobileFormer fusion engine is the first to modify the MobileFormer architecture, which was originally used in the image domain, and apply it to a one-dimensional temporal audio feature fusion task.

[0018] Step 3: The fused features are fed into the hierarchical graph attention classification module ASSIST. Through spectrogram and temporal dual-stream inference, master node aggregation and multi-granularity pooling, the binary classification result of audio authenticity is output.

[0019] Further, the method in step 1 includes:

[0020] The original waveform is processed using the BEATs model, and context-aware sequence features are output. And mapped to a 128-dimensional sequence X by linear projection. trans =X ssl W L +b L Meanwhile, frame-level embedding features are extracted using a CNN14 network. And compressed into a 128-dimensional sequence X by a projection layer. cnn =X cnn14 W p .

[0021] Furthermore, the AdaptedMobileFormerBlock process in step 2 includes:

[0022] First, the Mobile_Former mechanism, within the Mobile_Former module, uses X... trans As for query Q, X cnn After linear transformation, the keys K and values ​​V are used as follows:

[0023] Q=q_proj(z),K=k_proj(x),V=v_proj(x)

[0024] Attention output is:

[0025]

[0026] The final updated global sequence is:

[0027] z cross =norm(z+out_proj(A))

[0028] Then there's the Former mechanism, within the Former module, the updated global sequence z. cross Its internal representation is further enhanced through a self-attention mechanism. The self-attention calculation method is the same as that of the standard Transformer, including multi-head self-attention and a feedforward network;

[0029] Then perform dynamic MLP, using the average value of the global sequence, token0 =

[0030] mean(z out (dim=1) controls the dynamic nonlinear activation function DynamicReLU. Specifically, the local vector x is first expanded by two dimensions through a linear layer linear1, and then through...

[0031] DynamicReLU is activated, and then the system returns to the initial dimension via a linear layer (linear2). The parameters of DynamicReLU are generated by an MLP from the control vector token0, in the following form:

[0032] f(x) = max(α·x, β·x)

[0033] α and β are learnable parameters dynamically generated from the mean of the input features, which enhance the nonlinear expressive power.

[0034] Finally, the Former_Mobile mechanism: In the Former_Mobile module, the updated local vector x mid As a query (Q), and the global sequence z out After a linear transformation, the values ​​are used as keys (K) and values ​​(V). The specific calculation is as follows:

[0035] Q = q_proj(x mid ),K=kv_proj(z out ) key V = kv_proj(z out ) value Attention output is:

[0036]

[0037] The final updated local sequence is:

[0038] x update =norm(x mid +out_proj(A))

[0039] In all the above formulas, "_proj" indicates different fully connected layers;

[0040] In the Adapted MobileFormer fusion engine, a bidirectional cross-attention mechanism is used to achieve dynamic complementary modeling of the features of two modalities. The above process is stacked and executed in multiple AdaptedMobileFormerBlocks to complete deep interactive fusion.

[0041] Furthermore, the ASSIST mentioned in step 3 includes:

[0042] First, the fused features are processed by a convolutional encoder to extract high-level representations. Then, x is transposed and reshaped into a two-dimensional form. After max pooling and batch normalization, it is input into a convolutional encoder consisting of multiple residual blocks to extract high-level time-frequency representations.

[0043] h = Encoder(BN(MaxPool2d(X)))

[0044] Then, spectral features are generated using an attention mechanism. and time series characteristics

[0045]

[0046] Then e s and e T Input the graph attention layer (GAT) separately, model the dependencies between nodes, and calculate the attention weights as follows:

[0047]

[0048] Where h i Let N(i) be the feature of node i, and N(i) be its set of neighbors.

[0049] The output after nonlinear transformation is:

[0050]

[0051] Then, a learnable master node m1 is introduced. respectively with gat s ,gat T Perform attention interactions and compress the number of nodes using a graph pooling layer:

[0052] out s =Pool s (gat s ),out T =Pool T (gat T )

[0053] A hierarchical graph attention network (HtrgGAT) is used for two-stream inference. Each stream contains two levels of cross-modal fusion:

[0054] First-class:

[0055]

[0056] Second-class:

[0057]

[0058] Finally, after applying Dropout to the two sets of enhanced outputs, they are merged using an element-wise maximization operation:

[0059]

[0060] To enhance the model's representation capabilities.

[0061] Furthermore, step 3 also includes:

[0062] Multi-granularity pooling is performed on the final graph representation to extract the maximum and average values ​​of the temporal and spectral branches:

[0063]

[0064] And concatenate the output m of the master node to form the final classification vector;

[0065] splicing vectors After Dropout, the input is fed into a fully connected layer, and the output is the classification probability:

[0066] y = Softmax(W out z+b out )

[0067] This invention also discloses a sound authenticity recognition system based on multimodal feature deep interactive fusion, used to implement the method described in any of the above, comprising:

[0068] The feature extraction module is used to construct a dual-stream feature extraction path for the input audio signal, and extracts the Transformer sequence features and convolutional time-frequency embedding features of the audio using the pre-trained BEATs model and the CNN14 network, respectively.

[0069] The feature fusion module is used to input the two types of features into the AdaptedMobileFormer fusion module and fuse them through bidirectional cross-attention and dynamic nonlinear activation mechanism. During the fusion stage, the two modal features interact bidirectionally through multiple AdaptedMobileFormerBlocks to generate an enhanced sequence representation.

[0070] The classification decision module feeds the fused features into the hierarchical graph attention classification module ASSIST, which outputs a binary classification result of audio authenticity through spectral and temporal dual-stream inference, master node aggregation, and multi-granularity pooling.

[0071] Because of the adoption of the above technical solution, the present invention has the following advantages:

[0072] 1. Multimodal deep interaction enhances feature discriminativeness. By using an adaptive MobileFormer fusion machine to achieve bidirectional cross-attention interaction between Transformer sequences and CNN embeddings, and by introducing a dynamic nonlinear activation mechanism, the semantic alignment and complementarity between modalities are significantly enhanced, overcoming the semantic gap problem of traditional post-fusion methods.

[0073] 2. High-order semantic modeling enhances robustness. A hierarchical graph attention network (HtrgGAT) is used to perform multi-granular reasoning on fused features. Combined with learnable master nodes and max / average pooling strategies, it effectively captures global forgery patterns across frames and frequency bands, improving the generalization ability to unknown attack types.

[0074] 3. The system has a complete structure and strong practicality. This invention provides dual protection of "method + system", which is suitable for high-security voice scenarios such as financial identity verification, smart speakers, and remote customer service. It can effectively defend against mainstream forgery methods such as AI-generated voice, recording and playback, and voice-changing attacks.

[0075] 4. The model balances efficiency and performance. Through feature dimension compression, modular design, and pre-training transfer, it achieves a balance between high performance and low inference latency, making it valuable for engineering applications. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0077] Figure 1 This is a schematic diagram of the overall architecture of the sound authenticity recognition system based on deep interactive fusion of multimodal features according to the present invention;

[0078] Figure 2 This is a schematic diagram of the structure of the dual-stream feature extraction and adaptation MobileFormer fusion generator of the present invention;

[0079] Figure 3 This is a structural design diagram of the ASSIST layered graph attention classification module of the present invention; Detailed Implementation

[0080] The present invention will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0081] This invention proposes a method and system for audio authenticity recognition based on deep interactive fusion of multimodal features, aiming to address the problems of shallow multimodal fusion mechanisms, large cross-modal semantic gaps, and limited discriminative performance in existing technologies. Based on a two-stream deep network architecture, this invention innovatively introduces an adaptive MobileFormer fusion engine to deeply fuse Transformer sequence features extracted by a pre-trained BEATs model with convolutional time-frequency embedding features output by a CNN14 network. The adaptive MobileFormer fusion engine, through key modifications to the original MobileFormer structure, makes it suitable for bidirectional cross-modal interaction of one-dimensional temporal audio features: in multiple AdaptedMobileFormerBlocks, a bidirectional cross-attention mechanism is used to achieve dynamic complementary modeling of local details and global semantics, and combined with a dynamic nonlinear activation function (DynamicReLU) based on global token mean adjustment, further enhancing feature representation capabilities. The fused enhanced sequence representation is fed into the hierarchical graph attention classification module ASSIST, where high-order semantic modeling is completed and a binary classification result of audio authenticity is output through spectrogram and temporal two-stream inference, master node aggregation, and multi-granularity pooling.

[0082] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0083] Figure 1 This is a schematic diagram of the overall architecture of the sound authenticity recognition system of this invention. The system includes a feature extraction module, a feature fusion module, and a classification decision module that communicate with each other. The input audio signal is simultaneously fed into a dual-stream path: one path extracts context-aware sequence features of the original waveform through a pre-trained BEATs model, and the other path extracts convolutional embedding features of the Mel spectrogram through a CNN14 network. The two types of features are deeply fused in the feature fusion module using an adaptive MobileFormer structure to generate an enhanced sequence representation. Subsequently, the fused features are fed into the classification decision module AASIST, where high-order semantic modeling is performed through a hierarchical graph attention network (HtrgGAT), and finally, a binary classification result of "real" or "fake" is output through multi-granularity pooling and fully connected layers.

[0084] Figure 2This is a schematic diagram of the structure of the dual-stream feature extraction and adaptive MobileFormer fusion processor. The fusion processor first processes the input data in the feature extraction stage. The BEATs model processes the original audio waveform with a sliding step of 20 milliseconds to generate a high-dimensional semantic sequence. And mapped to a 128-dimensional sequence X by linear projection. trans Meanwhile, the original waveform is processed frame-by-frame by a CNN14 network to extract the 2048-dimensional embedding X. cnn14 Then compressed into 128-dimensional X through a projection layer. cnn .

[0085] During the fusion phase, the two modal features undergo bidirectional cross-attention interaction across multiple AdaptedMobileFormerBlocks. Each block includes the following steps:

[0086] Mobile_Former mechanism: In the Mobile_Former module, X trans As for query Q, X cnn After linear transformation, the keys K and values ​​V are used as follows:

[0087] Q=q_proj(z),K=k_proj(x),V=v_proj(x)

[0088] Attention output is:

[0089]

[0090] The final updated global sequence is:

[0091] z cross =norm(z+out_proj(A))Former mechanism: In the Former module, the updated global sequence z cross Its internal representation is further enhanced through a self-attention mechanism. The self-attention is computed in the same way as the standard Transformer, including multi-head self-attention and a feedforward network.

[0092] Dynamic MLP: Using the global sequence average token0 = mean(z) out `dim=1` controls the dynamic nonlinear activation function `DynamicReLU`. Specifically, the local vector `x` is first expanded to twice its dimension through a linear layer `linear1`, then activated by `DynamicReLU`, and finally returned to its initial dimension through a linear layer `linear2`. The parameters of `DynamicReLU` are generated by the control vector `token0` using an MLP, in the form:

[0093] f(x) = max(α·x, β·x)

[0094] α and β are learnable parameters dynamically generated from the mean of the input features, which enhance the nonlinear expressive power.

[0095] Former_Mobile mechanism: In the Former_Mobile module, the updated local vector x mid As a query (Q), and the global sequence z out After a linear transformation, the values ​​are used as keys (K) and values ​​(V). The specific calculation is as follows:

[0096] Q = q_proj(x mid ),K=kv_proj(z out ) key V = kv_proj(z out ) value Attention output is:

[0097]

[0098] The final updated local sequence is:

[0099] x update =norm(x mid +out_proj(A))

[0100] In all the formulas above, "_proj" indicates different fully connected layers.

[0101] After multiple iterations of AdaptedMobileFormerBlock, the final local vectors and global sequences are restored to their original dimensions through linear layers to obtain the fused features. Figure 3 This is the structural design diagram of the hierarchical graph attention classification module ASSIST. After fusion, the features are first processed by a convolutional encoder to extract high-level representations, and then separated into spectral features e through an attention mechanism. s With time series characteristics e T :

[0102]

[0103] Both are input into the graph attention layer (GAT), and the attention weights between nodes are calculated as follows:

[0104]

[0105] Learnable master nodes m1 and m2 are introduced to participate in hierarchical inference. Two-layer HtrgGAT is used for dual-stream inference, generating two sets of enhanced outputs. and After Dropout processing, the maximum operation is used for fusion:

[0106]

[0107] Improve the model's adaptability.

[0108] Finally, for out T and out S Perform multi-granularity pooling:

[0109]

[0110] The final classification vector is formed by concatenating the output m of the master node:

[0111]

[0112] After Dropout, the input is fed into a fully connected layer, and the output is the classification probability:

[0113] y = Softmax(W out z+b out )

[0114] This invention provides an embodiment of a method for identifying the authenticity of audio based on deep interactive fusion of multimodal features, comprising:

[0115] Step 1: Construct a dual-stream feature extraction path for the input audio signal, and extract the Transformer sequence features and convolutional time-frequency embedding features of the audio using the pre-trained BEATs model and the CNN14 network respectively;

[0116] Step 2: Input the two types of features into the Adapted MobileFormer fusion engine and fuse them through a bidirectional cross-attention mechanism. During the fusion stage, the two modal features interact bidirectionally through multiple AdaptedMobileFormerBlocks to generate an enhanced sequence representation. This Adapted MobileFormer fusion engine is the first to modify the MobileFormer architecture, which was originally used in the image domain, and apply it to a one-dimensional temporal audio feature fusion task.

[0117] Step 3: The fused features are fed into the hierarchical graph attention classification module ASSIST. Through spectrogram and temporal dual-stream inference, master node aggregation and multi-granularity pooling, the binary classification result of audio authenticity is output.

[0118] In this embodiment, step 2 includes:

[0119] Deep feature fusion is achieved by performing bidirectional cross-attention and dynamic nonlinear activation in multiple AdaptedMobileFormerBlocks.

[0120] In this embodiment, step 3 specifically includes:

[0121] High-precision discrimination is achieved by using HtrgGAT for dual-flow graph attention inference, introducing learnable master nodes for hierarchical aggregation, and combining multi-granularity pooling and Softmax classification.

[0122] The present invention also provides an embodiment of a voice authenticity recognition system based on multimodal feature deep interactive fusion, for implementing the above method, comprising:

[0123] The feature extraction module is used to construct a dual-stream feature extraction path for the input audio signal, and extracts the Transformer sequence features and convolutional time-frequency embedding features of the audio using the pre-trained BEATs model and the CNN14 network, respectively.

[0124] The feature fusion module is used to input the two types of features into the adaptable MobileFormer fusion processor, and perform deep interactive fusion through bidirectional cross-attention and dynamic nonlinear activation mechanism to generate an enhanced sequence representation;

[0125] The classification decision module feeds the fused features into the hierarchical graph attention classification module, which outputs a binary classification result of audio authenticity through spectral and temporal dual-stream inference, master node aggregation, and multi-granularity pooling.

[0126] The present invention will be further described below with reference to embodiments:

[0127] On an Ubuntu 20.04 system, using an NVIDIA RTX 3090 GPU, the model was trained and tested using Python within the PyTorch framework. The experimental data used the ASVspoof2021 public dataset, which includes various types of forgeries such as real speech, AI-generated speech, audio playback, and voice changing. The training parameters were set as follows:

[0128]

[0129] The method was compared with baseline models such as ResNet18, RawNet2, and LCNN. The test results show that the method of this invention has an equal error rate (EER) of 1.2% on the ASVspoof2021 LA dataset, compared with 3.8% for ResNet18 and 2.5% for RawNet2, which is better than many baseline models.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for identifying the authenticity of speech based on deep interactive fusion, characterized in that, include: Step 1: Construct a dual-stream feature extraction path for the input audio signal, and extract the Transformer sequence features and convolutional time-frequency embedding features of the audio using the pre-trained BEATs model and the CNN14 network respectively; Step 2: Input the two types of features into the Adapted MobileFormer fusion engine and fuse them through a bidirectional cross-attention mechanism. During the fusion stage, the two modal features interact bidirectionally through multiple AdaptedMobileFormerBlocks to generate an enhanced sequence representation. This Adapted MobileFormer fusion engine is the first to modify the MobileFormer architecture, which was originally used in the image domain, and apply it to a one-dimensional temporal audio feature fusion task. Step 3: The fused features are fed into the hierarchical graph attention classification module ASSIST. Through spectrogram and temporal dual-stream inference, master node aggregation and multi-granularity pooling, the binary classification result of audio authenticity is output.

2. The voice authenticity recognition method based on deep interactive fusion according to claim 1, characterized in that, Step 1 includes: The original waveform is processed using the BEATs model, and context-aware sequence features are output. And mapped to a 128-dimensional sequence X by linear projection. trans =X ssl W L +b L Meanwhile, the original waveform is processed by a CNN14 network to extract frame-level embedding features. And compressed into a 128-dimensional sequence X by a projection layer. cnn =X cnn14 W p .

3. The method for voice authenticity recognition based on deep interactive fusion according to claim 1, characterized in that, The AdaptedMobileFormerBlock module in step 2 includes: First, the Mobile_Former mechanism, within the Mobile_Former module, uses X... trans As for query Q, X cnn After linear transformation, the keys K and value V are used as the following calculations: Q=q_proj(z),K=k_proj(x),V=v_proj(x) Attention output is: The final updated global sequence is: z cross =norm(z+out_proj(A)) Then there's the Former mechanism, within the Former module, the updated global sequence z. cross Its internal representation is further enhanced through a self-attention mechanism. The self-attention calculation method is the same as that of the standard Transformer, including multi-head self-attention and a feedforward network; Then, a dynamic MLP is used, employing the global sequence average token0 = mean(z). out `dim=1` controls the dynamic nonlinear activation function `DynamicReLU`. Specifically, the local vector `x` is first expanded to twice its dimension through a linear layer `linear1`, then activated by `DynamicReLU`, and finally returned to its initial dimension through a linear layer `linear2`. The parameters of `DynamicReLU` are generated by the control vector `token0` using an MLP, in the form: f(x) = max(α·x, β·x) α and β are learnable parameters dynamically generated from the mean of the input features, which enhance the nonlinear expressive power. Finally, the Former_Mobile mechanism: In the Former_Mobile module, the updated local vector x mid As a query (Q), and the global sequence z out After a linear transformation, the values ​​are used as keys (K) and values ​​(V). The specific calculation is as follows: Q=q_proj(x mid ),K=kv_proj(z out ) key ,V=kv_proj(z out ) value Attention output is: The final updated local sequence is: x update =norm(x mid +out_proj(A)) In all the above formulas, "_proj" indicates different fully connected layers; In the Adapted MobileFormer fusion engine, a bidirectional cross-attention mechanism is used to achieve dynamic complementary modeling of the features of two modalities. The above process is stacked and executed in multiple AdaptedMobileFormerBlocks to complete deep interactive fusion.

4. The method for identifying the authenticity of audio based on deep interactive fusion of multimodal features according to claim 1, characterized in that, Step 3 includes: First, the fused features are processed by a convolutional encoder to extract high-level representations. Then, x is transposed and reshaped into a two-dimensional form. After max pooling and batch normalization, it is input into a convolutional encoder consisting of multiple residual blocks to extract high-level time-frequency representations. h = Encoder(BN(MaxPool2d(X))) Then, spectral features are generated separately using an attention mechanism. and time series characteristics e s and e T Input the graph attention layer (GAT) separately, model the dependencies between nodes, and calculate the attention weights as follows: Where h i Let N(i) be the feature of node i, and N(i) be its set of neighbors; The output after nonlinear transformation is:

5. The method for identifying the authenticity of audio based on deep interactive fusion of multimodal features according to claim 4, characterized in that, Step 3 also includes: Introducing a learnable master node respectively with gat s ,gat T Perform attention interactions and compress the number of nodes using a graph pooling layer: out s =Pool s (gat s ),out T =Pool T (gat T ) Then, a hierarchical graph attention network (HtrgGAT) is used for two-stream inference. Each stream contains two levels of cross-modal fusion: First-class: Second-class:

6. The method for identifying the authenticity of audio based on deep interactive fusion of multimodal features according to claim 5, characterized in that, Step 3 also includes: After applying Dropout to the two sets of enhanced outputs respectively, they are merged using an element-wise maximization operation: To enhance the model's representation capabilities. Multi-granularity pooling is performed on the final graph representation to extract the maximum and average values ​​of the temporal and spectral branches: And concatenate the output m of the master node to form the final classification vector; splicing vectors After Dropout, the input is fed into a fully connected layer, and the output is the classification probability: y=Softmax(W out z+b out )。 7. The method for identifying the authenticity of audio based on deep interactive fusion of multimodal features according to claim 1, characterized in that, The method is applied to a voice anti-fraud system to detect forged audio such as AI-generated voice, recording playback, and voice-changing attacks. It is suitable for scenarios such as financial identity verification and smart device voiceprint authentication.

8. A voice authenticity recognition system based on deep interactive fusion, used to implement the voice authenticity recognition method based on multimodal feature deep interactive fusion as described in any one of claims 1-7, characterized in that, include: The feature extraction module is used to construct a dual-stream feature extraction path for the input audio signal, and extracts the Transformer sequence features and convolutional time-frequency embedding features of the audio using the pre-trained BEATs model and the CNN14 network, respectively. The feature fusion module inputs the two types of features into the AdaptedMobileFormer fusion module and fuses them through bidirectional cross-attention and dynamic nonlinear activation mechanisms. During the fusion phase, the two modal features interact bidirectionally through multiple AdaptedMobileFormerBlocks to generate an enhanced sequence representation. The classification decision module feeds the fused features into the hierarchical graph attention classification module ASSIST, which outputs a binary classification result of audio authenticity through spectral and temporal dual-stream inference, master node aggregation, and multi-granularity pooling.

Images