A speech deep spoofing detection method and device based on hybrid spectrum expert cooperation

Abstract

This invention discloses a method for detecting deep speech forgery using a hybrid spectrum expert collaboration approach, comprising the following steps: S1: acquiring the speech signal to be detected; S2: performing frequency domain transformation on the speech signal to be detected to obtain a low-level frequency domain feature representation; S3: inputting the speech signal to be detected into a pre-trained speech model, using a hybrid spectrum expert module to efficiently adapt the parameters of the pre-trained model, and combining multi-layer feature aggregation to obtain a high-level contextual feature representation; S4: performing time-dimensional alignment and cross-attention fusion on the low-level frequency domain feature representation and the high-level contextual feature representation to obtain a fused feature representation; S5: inputting the fused feature representation into a classification and discrimination module to output the deep speech forgery detection result. This invention can improve the detection accuracy, model robustness, and cross-scene generalization ability in deep speech forgery detection tasks.

Description

Technical Field

[0001] This invention belongs to the fields of speech signal processing, artificial intelligence and information security technology, and specifically relates to a method and device for deep speech forgery detection using hybrid spectrum expert collaboration. Background Technology

[0002] In recent years, with the rapid development of deep learning and generative artificial intelligence technologies, technologies such as text-to-speech, speech conversion, speech cloning, and neural vocoders have made continuous progress. The generated speech has reached a high level in terms of naturalness, fluency, and speaker similarity. At the same time, deep speech spoofing technology has also brought potential risks to application scenarios such as identity authentication, voice interaction, media content authenticity verification, and cyberspace security. Therefore, it is of great significance to study high-precision and high-robustness deep speech spoofing detection methods.

[0003] Existing methods for detecting deep speech forgery typically employ deep neural networks to model acoustic features, enabling binary classification between real and forged speech. With the development of self-supervised pre-trained models, pre-trained Transformer models such as wav2vec 2.0 and WavLM have been gradually introduced into this task, demonstrating strong high-level semantic representation capabilities and generalization potential. However, existing techniques still have the following shortcomings: First, the pre-training objectives of pre-trained models usually focus more on speech content and contextual semantic information, and their ability to perceive low-level fine-grained physical artifacts such as local phase anomalies, amplitude perturbations, and harmonic blurring in forged speech is relatively insufficient, which limits their ability to distinguish in complex forged scenarios. Second, some methods have attempted to introduce frequency domain priors into the detection framework, but most schemes mainly focus on amplitude spectrum modeling and lack explicit characterization of phase information, making it difficult to fully explore the microscopic phase disorder and local spectral inconsistency introduced by processes such as neural vocoders and waveform reconstruction. Third, in existing dual-branch methods, there are often significant differences in representation granularity between high-level pre-training context features and low-level frequency domain physical features, making feature space alignment difficult and resulting in the ineffective collaborative utilization of the two types of complementary information. Fourth, regarding the adaptation methods for pre-trained models, if parameter freezing is used, it is difficult to fully adapt to the forgery detection task; if full parameter fine-tuning is used, it is easy to bring large computational overhead and may destroy the existing general acoustic priors of the pre-trained model, affecting the model's stability and generalization ability.

[0004] Therefore, existing deep forgery detection methods for speech suffer from problems such as insufficient perception of underlying physical forgeries in complex forgery scenarios, limited task adaptability of pre-trained models, and difficulty in effectively fusing features of different granularities. Summary of the Invention

[0005] (1) Technical problems to be solved To address the shortcomings of existing technologies, the present invention aims to provide a method and apparatus for deep speech forgery detection using hybrid spectrum expert collaboration. This method addresses the problems of insufficient perception of underlying physical artifacts, limited task adaptability of pre-trained models, and difficulty in effectively fusing features of different granularities in existing deep speech forgery detection methods under complex forgery scenarios.

[0006] (2) Technical solution To address the aforementioned technical problems, this invention provides a method for detecting deep speech forgery through hybrid spectrum expert collaboration, comprising the following steps: S1: Acquire the speech signal to be detected; S2: Perform frequency domain transformation on the speech signal to be detected to obtain a complex-valued time-frequency representation; extract the logarithmic amplitude spectrum, sine phase component and cosine phase component based on the complex-valued time-frequency representation, and perform compression encoding, local texture modeling and dimension projection through a frequency domain audio encoder to obtain the underlying frequency domain feature representation; S3: Input the speech signal to be detected into the pre-trained speech model, use the hybrid spectrum expert module to efficiently adapt the parameters of the pre-trained model, and combine multi-layer feature aggregation to obtain a high-level context feature representation; S4: Perform temporal alignment and cross-attention fusion on the lower-level frequency domain feature representation and the higher-level context feature representation to obtain the fused feature representation; S5: Input the fused feature representation into the classification and discrimination module, and output the voice deep forgery detection result.

[0007] Preferably, step S2 specifically comprises the following steps: S201, Frequency Domain Transform Processing: Perform a short-time Fourier transform on the speech signal to be detected to obtain a complex-valued time-frequency representation; S202, Amplitude Information Construction: Perform a logarithmic transformation on the amplitude part of the complex-valued time-frequency representation to obtain the logarithmic amplitude spectrum, which characterizes the energy distribution features of the speech signal; S203. Phase Information Construction: Perform sine mapping and cosine mapping on the phase angle of the complex-valued time-frequency representation to obtain the sine phase component and cosine phase component, so as to avoid the discontinuity caused by phase jump and retain the complete phase information. S204, Multi-component splicing: The logarithmic amplitude spectrum, sine phase component and cosine phase component are spliced ​​along the channel dimension to form the original multimodal frequency domain features; S205, Bottleneck compression coding: The original multimodal frequency domain features are compressed and mapped through a bottleneck convolutional layer to reduce the feature dimension and filter key anti-spoofing information; S206, Local Texture Modeling: Local spectral-temporal texture modeling is performed on the compressed frequency domain features through a depthwise separable convolution module to extract fine-grained forgery traces such as harmonic continuity anomalies, local phase jump anomalies, and instantaneous frequency fluctuations. S207, Dimension Alignment Projection: The features modeled by local texture are mapped to the representation dimension consistent with the hidden layer dimension of the pre-trained speech model through a linear projection layer to obtain the underlying frequency domain feature representation.

[0008] Furthermore, step S3 specifically involves the following steps: S301. Pre-training weight decomposition: Perform singular value decomposition on the weight matrix of the feedforward network in the pre-trained speech model to obtain the singular vector basis matrix and singular value energy distribution. S302, Spectrum Expert Construction: Divide the multiple singular components after singular value decomposition into multiple spectrum experts. Each spectrum expert includes a frozen singular basis and corresponding trainable energy level parameters. S303, Spectrum Domain Modulation: By adjusting the energy level parameters in each spectrum expert, the mapping intensity of input features in different spectrum subspaces is adaptively adjusted to enhance the discriminative spectrum components related to forged speech; S304. Dynamic route allocation: Generate activation weights for multiple spectrum experts based on the global context information of the input features, and dynamically select each spectrum expert according to the activation weights. S305, Expert Collaboration and Fusion: The output results of multiple spectrum experts are weighted and summed according to their activation weights to obtain an adaptive incremental representation. The adaptive incremental representation is then fused with the original feedforward network output using residual fusion, thereby enhancing the adaptability of the pre-trained model to the forgery detection task without significantly increasing the parameter scale. S306, Cross-layer sharing and aggregation: Share at least some spectral expert parameters among multiple network layers of a pre-trained speech model, and perform weighted aggregation of the hidden states output by different layers to form a high-level contextual feature representation.

[0009] Furthermore, step S4 specifically involves the following steps: S401, Feature Alignment: Align the low-level frequency domain feature representation and the high-level context feature representation in the time dimension to ensure that the two types of features are consistent in time resolution; S402. A cross-attention mechanism is used for feature fusion, in which high-level contextual features are used as query features, and low-level frequency domain features are used as key and value features. This enables the model to dynamically retrieve and aggregate local physical clues that are highly relevant to the authenticity judgment from the low-level frequency domain information under the guidance of high-level semantics, thereby obtaining the fused multi-granularity feature representation.

[0010] Furthermore, the classification and discrimination module outputs a binary classification result based on the fused multi-granularity feature representation, which is used to determine whether the input speech is real speech or deepfake speech.

[0011] Furthermore, before feature fusion, the two types of features can be aligned in time dimension by interpolation, sampling or other time scale transformations to improve the matching and fusion stability between features of different granularities.

[0012] The present invention also provides a voice deepfake detection device based on hybrid spectrum expert collaboration, comprising: The input processing module is used to acquire the speech signal to be detected; The frequency domain feature construction module is used to perform frequency domain transformation on the speech signal to be detected and extract amplitude and phase information; The frequency domain coding module is used to jointly encode the amplitude information, the sinusoidal phase component and the cosine phase component to obtain the underlying frequency domain feature representation; The pre-trained model adaptation module is used to adapt the pre-trained speech model to a task through the hybrid spectrum expert module to obtain high-level contextual feature representations. The feature fusion module is used to fuse the low-level frequency domain feature representation and the high-level context feature representation; the classification and discrimination module is used to output the authenticity detection result of the speech signal to be detected.

[0013] Furthermore, the hybrid spectrum expert module includes: The singular value decomposition unit is used to decompose the weight matrix of the feedforward network in the pre-trained model and freeze the basis matrix. A spectrum expert unit is used to perform spectrum domain modulation based on trainable energy level parameters; The dynamic routing unit is used to generate activation weights for each spectrum expert based on input features and control multiple spectrum experts to collaboratively output high-level context feature representations.

[0014] Beneficial effects Compared with the prior art, the beneficial effects of the present invention are as follows: This invention enhances the model's ability to characterize microscopic forgery clues such as local phase disturbances, instantaneous frequency fluctuations, and spectral inconsistencies by explicitly and jointly modeling amplitude and phase information, especially by decomposing the phase information into sine and cosine phase components. Compared with methods that rely solely on waveform or single amplitude spectrum modeling, this invention can more fully characterize the underlying physical anomalies introduced by the deep speech forgery process.

[0015] The hybrid spectrum expert module proposed in this invention performs spectral domain adaptive modulation on the pre-trained speech model by freezing the singular value decomposition basis and learning only the energy level parameters. While preserving the general speech priors and stable representation structure of the pre-trained model as much as possible, it achieves efficient adaptation to the forgery detection task, taking into account parameter efficiency, model stability and detection performance.

[0016] This invention enables collaboration among multiple spectrum experts through a dynamic router, allowing different spectrum experts to adaptively divide tasks based on the distribution of forgery clues in different input samples, thereby improving the model's responsiveness to diverse forgery attack patterns. At the same time, the residual fusion method makes the adaptation process smoother, which helps to reduce excessive perturbation to the original pre-trained representation.

[0017] This invention effectively narrows the representation gap between high-level semantic features and low-level physical features by constructing a cross-attention fusion mechanism guided by high-level contextual semantics and supported by low-level frequency domain information. This enables information of different granularities to achieve complementary fusion in a unified feature space, thereby improving the comprehensive detection capability of complex deep fake speech.

[0018] This invention employs a dual-branch detection framework that organically combines low-level frequency domain physical cue modeling, pre-trained model task adaptation, and multi-granularity feature fusion. It can utilize the high-level representation capabilities formed by the pre-trained model on large-scale speech data, while also compensating for its insufficient perception of fine-grained forged physical cues. Therefore, it has good application potential in complex scenarios, unknown attacks, and cross-domain detection tasks.

[0019] This invention constructs a dual-branch detection framework that combines a frequency domain feature coding branch with a pre-trained speech model branch. While fully utilizing the high-level contextual semantic representation capabilities of the pre-trained model, it introduces explicit modeling of amplitude and phase information, and combines a hybrid spectrum expert module to efficiently adapt the parameters of the pre-trained model. Furthermore, it achieves effective alignment and fusion between the underlying physical spectrum features and the high-level contextual features through a cross-attention mechanism, thereby improving the detection accuracy, model robustness, and cross-scene generalization ability in speech deep forgery detection tasks. Attached Figure Description

[0020] Figure 1 The diagram above illustrates the overall framework of a hybrid spectrum expert collaborative speech deep forgery detection method provided in this embodiment of the invention. It shows the connection relationships between the frequency domain feature coding branch, the pre-trained speech model branch, the hybrid spectrum expert module, the feature fusion module, and the classification and discrimination module.

[0021] Figure 2 A bar chart comparing the performance of different fine-tuning strategies.

[0022] Figure 3 A logarithmic coordinate comparison of the number of trainable parameters for different fine-tuning strategies.

[0023] Figure 4 A chart comparing the percentage of parameters for different fine-tuning strategies.

[0024] Figure 5 The graph shows the impact of different numbers of spectrum experts K on detection performance.

[0025] Figure 6 The graph shows the impact of different cross-layer shared group sizes G on detection performance.

[0026] Figure 7 This is a graph showing the effect of the routing temperature parameter τ on detection performance. Detailed Implementation

[0027] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Various modifications or equivalent substitutions made to the present invention by those skilled in the art without departing from the spirit and substance of the present invention should fall within the scope of protection of the present invention.

[0028] This embodiment provides a method for deep speech forgery detection based on hybrid spectrum expert collaboration, the overall framework of which is as follows: Figure 1 As shown, this method generally adopts a dual-branch detection structure, including a frequency domain feature encoding branch and a pre-trained speech model branch. The frequency domain feature encoding branch is used to extract low-level physical spectrum cues from the input speech, while the pre-trained speech model branch is used to extract high-level contextual semantic features. Subsequently, a hybrid spectrum expert adaptation mechanism and a cross-attention fusion mechanism are used to achieve collaborative modeling of the two types of features, and finally, the authenticity judgment result of the detected speech is output.

[0029] Specifically, this embodiment provides a method for detecting deep speech forgery through hybrid spectrum expert collaboration, including the following steps: S1: Acquire the speech signal to be detected and use it as input audio; for the input audio, it is fed into the frequency domain feature encoding branch on one hand and the pre-trained speech model branch on the other hand, so as to extract the low-level physical features and high-level contextual features in parallel; this dual-branch structure can give full play to the complementary advantages of frequency domain explicit modeling and pre-trained representation learning in deep fake speech detection.

[0030] S2: In the frequency domain feature coding branch, frequency domain transformation is performed on the speech signal to be detected to obtain a complex-valued time-frequency representation; based on the complex-valued time-frequency representation, logarithmic amplitude spectrum, sine phase component and cosine phase component are extracted, and compression coding, local texture modeling and dimension projection are performed by a frequency domain audio encoder to obtain the underlying frequency domain feature representation; The specific steps of step S2 are as follows: S201, Frequency Domain Transform Processing: Perform a short-time Fourier transform on the speech signal to be detected to obtain a complex-valued time-frequency representation; S202, Amplitude Information Construction: Perform a logarithmic transformation on the amplitude part of the complex-valued time-frequency representation to obtain the logarithmic amplitude spectrum, which characterizes the energy distribution features of the speech signal; S203. Phase Information Construction: Perform sine mapping and cosine mapping on the phase angle of the complex-valued time-frequency representation to obtain the sine phase component and cosine phase component, so as to avoid the discontinuity caused by phase jump and retain the complete phase information. S204, Multi-component splicing: The logarithmic amplitude spectrum, sine phase component and cosine phase component are spliced ​​along the channel dimension to form the original multimodal frequency domain features; S205, Bottleneck compression coding: The original multimodal frequency domain features are compressed and mapped through a bottleneck convolutional layer to reduce the feature dimension and filter key anti-spoofing information; S206, Local Texture Modeling: Local spectral-temporal texture modeling is performed on the compressed frequency domain features through a depthwise separable convolution module to extract fine-grained forgery traces such as harmonic continuity anomalies, local phase jump anomalies, and instantaneous frequency fluctuations. S207, Dimension Alignment Projection: The features modeled by local texture are mapped to the representation dimension consistent with the hidden layer dimension of the pre-trained speech model through a linear projection layer to obtain the underlying frequency domain feature representation.

[0031] Through steps S201 to S203, we can preserve the energy distribution information in the amplitude spectrum and explicitly retain the microscopic physical clues corresponding to the phase changes, while avoiding the periodic jumps and discontinuities caused by directly modeling the phase angle.

[0032] S3: In the pre-trained speech model branch, the speech signal to be detected is input into the pre-trained speech model. The parameters of the pre-trained model are efficiently adapted by the hybrid spectrum expert module, and a high-level contextual feature representation is obtained by combining multi-layer feature aggregation. Specifically, step S3 involves the following steps: S301. Pre-training weight decomposition: Perform singular value decomposition on the weight matrix of the feedforward network in the pre-trained speech model to obtain the singular vector basis matrix and singular value energy distribution. S302, Spectrum Expert Construction: Divide the multiple singular components after singular value decomposition into multiple spectrum experts. Each spectrum expert includes a frozen singular basis and corresponding trainable energy level parameters. S303, Spectrum Domain Modulation: By adjusting the energy level parameters in each spectrum expert, the mapping intensity of input features in different spectrum subspaces is adaptively adjusted, thereby achieving spectrum domain adaptive modulation of pre-trained weights to enhance the discriminative spectrum components related to forged speech; S304. Dynamic route allocation: Generate activation weights for multiple spectrum experts based on the global context information of the input features, and dynamically select each spectrum expert according to the activation weights. S305, Expert Collaboration and Fusion: The output results of multiple spectrum experts are weighted and summed according to their activation weights to obtain an adaptive incremental representation. The adaptive incremental representation is then fused with the original feedforward network output using residual fusion, thereby enhancing the adaptability of the pre-trained model to the forgery detection task without significantly increasing the parameter scale. S306, Cross-layer sharing and aggregation: Share at least some spectral expert parameters among multiple network layers of a pre-trained speech model, and perform weighted aggregation of the hidden states output by different layers to form a high-level contextual feature representation.

[0033] S4: Since the high-level contextual feature representation output by the pre-trained speech model and the low-level frequency domain feature representation output by the frequency domain feature encoding branch may differ in temporal resolution, the two types of features are first aligned in time dimension before fusion. That is, the low-level frequency domain feature representation and the high-level contextual feature representation are aligned in time dimension and fused with cross-attention to obtain the fused feature representation. Before fusion, the two types of features can also be aligned in time dimension by interpolation, sampling or other time scale transformation methods to improve the matching and fusion stability between features of different granularities.

[0034] Specifically, step S4 involves the following steps: S401, Feature Alignment: Align the low-level frequency domain feature representation and the high-level context feature representation in the time dimension to ensure that the two types of features are consistent in time resolution; S402, Fusion Representation Generation: A cross-attention mechanism is used for feature fusion. High-level contextual features are used as query features, and low-level frequency domain features are used as key and value features. This enables the model to dynamically retrieve and aggregate local physical clues that are highly relevant to the authenticity judgment from the low-level frequency domain information under the guidance of high-level semantics, thereby obtaining the fused multi-granularity feature representation.

[0035] S5: Input the fused multi-granularity feature representation into the classification and discrimination module, and output the detection result of whether the speech to be detected belongs to real speech or deepfake speech; In step S5, the classification and discrimination module outputs a binary classification result based on the fused multi-granularity feature representation, which is used to determine whether the input speech is real speech or deepfake speech.

[0036] This embodiment also provides a speech deep forgery detection device based on hybrid spectrum expert collaboration, used to implement the speech deep forgery detection method based on hybrid spectrum expert collaboration, including an input processing module, a frequency domain feature construction module, a frequency domain coding module, a pre-trained model adaptation module, a feature fusion module, and a classification and discrimination module.

[0037] The input processing module is used to acquire the speech signal to be detected and transmit the speech signal to be detected to the frequency domain feature construction module and the pre-trained model adaptation module, respectively.

[0038] The frequency domain feature construction module is used to perform frequency domain transformation on the input speech signal and extract the logarithmic amplitude spectrum, sine phase component and cosine phase component. The frequency domain feature construction module is also used to extract the amplitude information and phase information of the speech signal.

[0039] The frequency domain coding module is used to jointly encode the amplitude information and phase information to obtain the underlying frequency domain feature representation.

[0040] The pre-trained model adaptation module is used to extract high-level contextual feature representations of the input speech signal based on the hybrid spectrum expert module, and to adapt the pre-trained speech model to the task based on the hybrid spectrum expert module. The hybrid spectrum expert module includes a singular value decomposition unit, a spectrum expert unit, and a dynamic routing unit. The singular value decomposition unit is used to decompose the feedforward network weight matrix in the pre-trained model and freeze the basis matrix. The spectrum expert unit is used to perform spectrum domain modulation based on trainable energy level parameters. The dynamic routing unit is used to generate activation weights for each spectrum expert according to input features and control multiple spectrum experts to collaboratively output high-level context feature representations.

[0041] The feature fusion module is used to perform time-dimensional alignment and cross-attention fusion on the low-level frequency domain feature representation and the high-level context feature representation to generate a fused multi-granularity feature representation.

[0042] The classification and discrimination module is used to receive the fused multi-granularity feature representation and output the authenticity detection result of the speech signal to be detected.

[0043] To verify the impact of each component module in this invention on the performance of speech deepfake detection, a module ablation experiment was conducted on a publicly available speech deepfake detection dataset. The detection results under different module configurations are shown in Table 1.

[0044] Table 1 Comparison of detection results for different module combinations in the embodiments of the present invention

[0045] As shown in Table 1, the system detection performance is relatively low without the pre-trained speech model backbone. Introducing the pre-trained speech model backbone significantly improves the detection performance, indicating that the pre-trained representation can provide effective high-level acoustic priors for deep speech forgery detection. Furthermore, when only the pre-trained speech model backbone is retained and the hybrid spectrum expert module and frequency domain audio encoder are removed, the system performance decreases compared to the complete model, indicating that relying solely on the pre-trained backbone is insufficient to fully uncover fine-grained forgery clues related to authenticity. Simultaneously, when the hybrid spectrum expert module and frequency domain audio encoder are removed respectively, the system detection performance degrades, indicating that the hybrid spectrum expert module enhances the pre-trained speech model's adaptability to forgery detection tasks, and the frequency domain audio encoder supplements the physical spectrum clues related to the underlying amplitude and phase information. The complete model achieves optimal results across all detection metrics, demonstrating that this invention, through a combination of dual-branch feature modeling and efficient parameter adaptation, can achieve good synergistic gains, thereby improving the performance of deep speech forgery detection.

[0046] To further illustrate the performance of the proposed method in terms of detection efficiency and key parameter settings, a comparative analysis was conducted on a publicly available deepfake speech detection dataset. The results are as follows: Figures 2 to 7 As shown.

[0047] Figure 2 The performance comparison results of different fine-tuning strategies in the speech deepfake detection task are shown. Figure 2 As can be seen, compared with full parameter fine-tuning, LoRA, and LoRA+MoE strategies, the MoSE strategy proposed in this invention achieves better detection results on different evaluation sets, indicating that the method can effectively adapt to the speech deep forgery detection task with a low parameter update cost and has good detection performance.

[0048] Figure 3 The comparison results of different fine-tuning strategies in terms of the number of trainable parameters are shown. Since full-parameter fine-tuning requires updating all model parameters, its parameter size is significantly higher than other efficient parameter fine-tuning methods. Figure 3 Displayed using logarithmic coordinates, combined with Figure 2 It can be seen that the MoSE proposed in this invention can still achieve good detection results while maintaining a low parameter update scale, indicating that the method has high parameter utilization efficiency.

[0049] Figure 4 The comparison results of parameter proportions for different fine-tuning strategies are shown. Figure 4It can be seen that LoRA, LoRA+MoE and the MoSE proposed in this invention all only need to update a small proportion of parameters to complete the task adaptation. Among them, the MoSE proposed in this invention still maintains better detection performance under the condition of a low parameter ratio, demonstrating good parameter efficiency and task adaptation capability.

[0050] Figure 5 The results show the impact of different numbers of spectral experts on model detection performance. Under the condition of a fixed cross-layer shared group size, as the number of spectral experts increases, the model detection performance first improves and then slightly declines. This indicates that appropriately introducing a multi-expert mechanism helps enhance the model's ability to represent different spectral artifact patterns, while too many expert divisions may affect the modeling effect of a single expert on continuous acoustic patterns. Therefore, a moderate number of experts is more conducive to balancing model expressive power and structural complexity. Figure 6 The results show the impact of different cross-layer shared group sizes on model detection performance. Figure 6 It can be seen that, under the condition of a fixed number of spectrum experts, the model's detection performance degrades to some extent as the size of the shared group increases. This indicates that, although there is some feature redundancy between adjacent layers of the deep pre-trained encoder, an excessively large cross-layer shared stride may still weaken the ability of the hybrid spectrum expert module to adapt to inter-layer differences in features. Therefore, a smaller shared group size is more conducive to achieving a balance between parameter efficiency and detection performance.

[0051] Figure 7 The results show the impact of different routing temperature settings on the model's detection performance. Figure 7 It can be seen that the model performance fluctuates under fixed temperature conditions, while the model can achieve better results when a learnable temperature mechanism is adopted. This indicates that compared with static temperature parameters that rely on experience, the adaptive temperature mechanism can better balance the smoothness and discriminability of the routing distribution, thereby improving the collaborative modeling effect among multiple spectrum experts.

[0052] As can be seen from the above results, the hybrid spectrum expert collaborative speech deep forgery detection method proposed in this invention shows good comprehensive advantages in terms of detection performance, parameter efficiency, and key structural parameter settings.

[0053] The above embodiments are preferred implementations of the present invention. In addition, the present invention can be implemented in other ways. Any obvious substitutions without departing from the concept of the present technical solution are within the protection scope of the present invention.

Claims

1. A method for detecting deep forgery of speech using hybrid spectrum expert collaboration, characterized in that, Includes the following steps: S1: Acquire the speech signal to be detected; S2: Perform frequency domain transformation on the speech signal to be detected to obtain a complex-valued time-frequency representation; extract the logarithmic amplitude spectrum, sine phase component and cosine phase component based on the complex-valued time-frequency representation, and perform compression encoding, local texture modeling and dimension projection through a frequency domain audio encoder to obtain the underlying frequency domain feature representation; S3: Input the speech signal to be detected into the pre-trained speech model, use the hybrid spectrum expert module to efficiently adapt the parameters of the pre-trained model, and combine multi-layer feature aggregation to obtain a high-level context feature representation; S4: Perform temporal alignment and cross-attention fusion on the lower-level frequency domain feature representation and the higher-level context feature representation to obtain the fused feature representation; S5: Input the fused feature representation into the classification and discrimination module, and output the voice deep forgery detection result.

2. The method for detecting deep forgery of speech using hybrid spectrum expert collaboration as described in claim 1, characterized in that, The specific steps of step S2 are as follows: S201, Frequency Domain Transform Processing: Perform a short-time Fourier transform on the speech signal to be detected to obtain a complex-valued time-frequency representation; S202, Amplitude Information Construction: Perform a logarithmic transformation on the amplitude part of the complex-valued time-frequency representation to obtain the logarithmic amplitude spectrum, which characterizes the energy distribution features of the speech signal; S203. Phase Information Construction: Perform sine mapping and cosine mapping on the phase angle of the complex-valued time-frequency representation to obtain the sine phase component and cosine phase component, so as to avoid the discontinuity caused by phase jump and retain the complete phase information. S204, Multi-component splicing: The logarithmic amplitude spectrum, sine phase component and cosine phase component are spliced ​​along the channel dimension to form the original multimodal frequency domain features; S205, Bottleneck compression coding: The original multimodal frequency domain features are compressed and mapped through a bottleneck convolutional layer to reduce the feature dimension and filter key anti-spoofing information; S206, Local Texture Modeling: Local spectral-temporal texture modeling is performed on the compressed frequency domain features through a depthwise separable convolution module to extract fine-grained forgery traces such as harmonic continuity anomalies, local phase jump anomalies, and instantaneous frequency fluctuations. S207, Dimension Alignment Projection: The features modeled by local texture are mapped to the representation dimension consistent with the hidden layer dimension of the pre-trained speech model through a linear projection layer to obtain the underlying frequency domain feature representation.

3. The method for detecting deep forgery of speech using hybrid spectrum expert collaboration as described in claim 1, characterized in that, The specific steps of step S3 are as follows: S301. Pre-training weight decomposition: Perform singular value decomposition on the weight matrix of the feedforward network in the pre-trained speech model to obtain the singular vector basis matrix and singular value energy distribution. S302, Spectrum Expert Construction: Divide the multiple singular components after singular value decomposition into multiple spectrum experts. Each spectrum expert includes a frozen singular basis and corresponding trainable energy level parameters. S303, Spectrum Domain Modulation: By adjusting the energy level parameters in each spectrum expert, the mapping intensity of input features in different spectrum subspaces is adaptively adjusted to enhance the discriminative spectrum components related to forged speech; S304. Dynamic route allocation: Generate activation weights for multiple spectrum experts based on the global context information of the input features, and dynamically select each spectrum expert according to the activation weights. S305, Expert Collaboration and Fusion: The output results of multiple spectrum experts are weighted and summed according to their activation weights to obtain an adaptive incremental representation. The adaptive incremental representation is then fused with the original feedforward network output using residual fusion, thereby enhancing the adaptability of the pre-trained model to the forgery detection task without significantly increasing the parameter scale. S306, Cross-layer sharing and aggregation: Share at least some spectral expert parameters among multiple network layers of a pre-trained speech model, and perform weighted aggregation of the hidden states output by different layers to form a high-level contextual feature representation.

4. The method for detecting deep forgery of speech using hybrid spectrum expert collaboration as described in claim 1, characterized in that, The specific steps of step S4 are as follows: S401, Feature Alignment: Align the low-level frequency domain feature representation and the high-level context feature representation in the time dimension to ensure that the two types of features are consistent in time resolution; S402, Fusion Representation Generation: A cross-attention mechanism is used for feature fusion. High-level contextual features are used as query features, and low-level frequency domain features are used as key and value features. This enables the model to dynamically retrieve and aggregate local physical clues that are highly relevant to the authenticity judgment from the low-level frequency domain information under the guidance of high-level semantics, thereby obtaining the fused multi-granularity feature representation.

5. The method for detecting deep forgery of speech using hybrid spectrum expert collaboration as described in claim 4, characterized in that, The classification and discrimination module outputs a binary classification result based on the fused multi-granularity feature representation, which is used to determine whether the input speech is real speech or deepfake speech.

6. The method for detecting deep forgery of speech using hybrid spectrum expert collaboration as described in claim 4, characterized in that, Before feature fusion, the two types of features can be aligned in time dimension by interpolation, sampling or other time scale transformations to improve the matching and fusion stability between features of different granularities.

7. A hybrid spectrum expert collaborative speech deep forgery detection device, used to implement the hybrid spectrum expert collaborative speech deep forgery detection method according to any one of claims 1-6, characterized in that, include: The input processing module is used to acquire the speech signal to be detected; The frequency domain feature construction module is used to perform frequency domain transformation on the speech signal to be detected and extract amplitude and phase information; The frequency domain coding module is used to jointly encode the amplitude information, the sinusoidal phase component and the cosine phase component to obtain the underlying frequency domain feature representation; The pre-trained model adaptation module is used to adapt the pre-trained speech model to a task through the hybrid spectrum expert module to obtain high-level contextual feature representations. The feature fusion module is used to fuse the low-level frequency domain feature representation and the high-level context feature representation; The classification and discrimination module is used to output the authenticity detection results of the voice signal to be detected.

8. The voice deepfake detection device with hybrid spectrum expert collaboration according to claim 7, characterized in that, The hybrid spectrum expert module includes: The singular value decomposition unit is used to decompose the weight matrix of the feedforward network in the pre-trained model and freeze the basis matrix. A spectrum expert unit is used to perform spectrum domain modulation based on trainable energy level parameters; The dynamic routing unit is used to generate activation weights for each spectrum expert based on input features and control multiple spectrum experts to collaboratively output high-level context feature representations.

Images