Audio depth forgery detection method fusing multi-source features and cross-scale modeling

Abstract

The invention relates to the field of audio detection, and provides an audio deep counterfeiting detection method and system fusing multi-source features and cross-scale modeling, which effectively improves the recognition capability of a hidden counterfeiting mode in a synthetic sample through a double-branch data enhancement strategy, and improves the recognition capability of the hidden counterfeiting mode while keeping the speech naturalness and auditory acceptability. A representative forgery disturbance space is introduced, so that generalization and robustness under multi-type deep forgery attacks are remarkably improved, multi-level deep audio features and physical acoustic features are obtained respectively and are subjected to feature fusion, the sensitivity to physical layer artifacts is improved, the detection accuracy is further improved, and the detection efficiency is improved. And through multi-scale attention enhancement, weak change characteristics across time and frequency scales in the forged audio are efficiently captured, through structural design of grouped convolution and a multi-branch attention path, local and global perception information is effectively integrated, and the accuracy, robustness and generalization ability of audio deep forging detection are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of audio detection, in particular to an audio deepfake detection method fusing multi-source features and cross-scale modeling. BACKGROUND

[0002] With the development of generative artificial intelligence, audio language models combining speech processing and natural language processing have made significant progress, making voice cloning, voice conversion and text-to-speech technologies increasingly mature. The threshold for creating fake audio has been significantly lowered, and deepfake audio can be highly realistic in terms of semantics and tone, increasing the risk of false dissemination and identity fraud. Therefore, audio deepfake detection has become an essential part.

[0003] In the prior art, traditional detection methods generally rely on high-level speech representations extracted by pre-trained models, lacking explicit attention to underlying acoustic and physical features, making it difficult to capture physical law deviations in the speech generation process. At the same time, fake features often exist in the form of weak artifacts in multiple temporal and spatial scales and abstract levels, making existing traditional detection methods unable to fully cover all fake modes, leading to problems such as insufficient detection accuracy or poor robustness.

[0004] Therefore, how to design a deepfake audio detection method to improve the accurate identification of various types of deepfake audio has become a problem to be solved. SUMMARY

[0005] Therefore, the present application proposes an audio deepfake detection method fusing multi-source features and cross-scale modeling, which effectively improves the recognition ability of hidden fake modes in synthetic samples through a dual-branch data enhancement strategy, while maintaining the naturalness and auditory acceptability of speech, introducing representative fake disturbance spaces, thereby significantly improving the generalization and robustness under multiple types of deepfake attacks. Multi-level deep audio features and physical acoustic features are obtained respectively and fused, combined with channel attention and gating strategies to strengthen the expression ability of fake-related information and improve the sensitivity to physical layer artifacts, further improving detection accuracy. A fake mode recognition framework with multi-scale and spatial context modeling capabilities is constructed through multi-scale attention enhancement, which can efficiently capture weak change features in time and frequency scales in fake audio. Through the structural design of grouped convolution and multi-branch attention path, local and global perception information is effectively integrated to provide strong expressive intermediate representations for the fake detection task. The present application improves the accuracy, robustness and generalization ability of audio deepfake detection.

[0006] The audio deepfake detection method fusing multi-source features and cross-scale modeling proposed by the present application comprises: The original audio data is acquired and data enhancement processing is performed, the data enhancement processing being based on a double-branch data enhancement strategy, the double-branch data enhancement strategy including a semantic artifact injection branch and a perception-invariant perturbation branch; Feature extraction is performed on the data-enhanced original audio data to respectively acquire multi-level deep audio features and physical acoustic features, the multi-level deep audio features being based on a cross-language speech representation learning model; Feature fusion processing is performed according to the multi-level deep audio features and the physical acoustic features to acquire fusion features, the feature fusion processing being based on a channel attention mechanism and a gating mechanism; Multi-scale attention enhancement processing is performed on the fusion features to acquire multi-scale attention enhancement features, the multi-scale attention enhancement processing being based on two parallel convolution branches; Classification is performed according to the multi-scale attention enhancement features to acquire a final audio deepfake detection result.

[0007] In summary, according to the above-mentioned audio deep fake detection method that integrates multi-source features and cross-scale modeling, the dual-branch data enhancement strategy effectively improves the recognition ability of hidden fake patterns in synthetic samples. While maintaining the naturalness and auditory acceptability of the speech, a representative fake perturbation space is introduced, thereby significantly improving the generalization and robustness under multiple types of deep fake attacks. Then, multi-level deep audio features and physical acoustic features are obtained respectively, and feature fusion is performed. The channel attention and gating strategies are combined to enhance the expression ability of fake-related information, improve the sensitivity to physical layer artifacts, and further improve the detection accuracy. A fake pattern recognition framework with multi-scale and spatial context modeling capabilities is constructed through multi-scale attention enhancement, which can efficiently capture the subtle change characteristics across time and frequency scales in fake audio. Through the structural design of grouped convolution and multi-branch attention paths, local and global perception information is effectively integrated to provide a highly expressive intermediate representation for the fake detection task. The present invention improves the accuracy, robustness and generalization ability of audio deep fake detection. Specifically, the original audio data is obtained and data enhancement processing is performed. The data enhancement processing is based on a dual-branch data enhancement strategy. The dual-branch data enhancement strategy includes a semantic artifact injection branch and a perceptually invariant perturbation branch, which effectively improves the ability to recognize hidden forgery patterns in synthetic samples. While maintaining the naturalness and auditory acceptability of the speech, a representative forgery perturbation space is introduced, thereby significantly improving the generalization and robustness under multiple types of deep forgery attacks. Feature extraction is performed on the original audio data after data enhancement to obtain multi-level deep audio features and physical acoustic features respectively. The multi-level deep audio features are based on a cross-language speech representation learning model. Feature fusion processing is performed according to the multi-level deep audio features and physical acoustic features to obtain fusion features. The feature fusion processing is based on a channel attention machine The control and gating mechanism strengthens the expression ability of forgery-related information, improves the sensitivity to physical layer artifacts, and further improves the detection accuracy. The fusion feature is subjected to multi-scale attention enhancement processing to obtain multi-scale attention enhancement features. The multi-scale attention enhancement processing is based on two parallel convolution branches to construct a forgery pattern recognition framework with multi-scale and spatial context modeling capabilities, which can efficiently capture the subtle change characteristics across time and frequency scales in forged audio. Through the structural design of grouped convolution and multi-branch attention paths, local and global perception information are effectively integrated to provide a highly expressive intermediate representation for the forgery detection task. Classification is performed according to the multi-scale attention enhancement features to obtain the final audio deep forgery detection result. The present invention improves the accuracy, robustness and generalization ability of audio deep forgery detection.

[0008] Furthermore, the step of obtaining the original audio data and performing data enhancement processing specifically includes: Obtain the original audio data and input it into the semantic artifact injection branch and the perceptually invariant perturbation branch respectively; The semantic artifact injection branch constructs a forgery-like enhancement sample based on the original audio data, and the perceptually invariant perturbation branch performs hidden generation artifact capture enhancement based on the original audio data, wherein the hidden generation artifact capture enhancement is based on a perturbation generation strategy constrained by perceptual distance; The pseudo-augmented sample construction performs local band masking, band-limited inversion, and time-domain nonlinear stretching on the Mel spectrum to simulate nonlinear distortion of deep synthetic audio. Then, based on the speech style transfer strategy, a pseudo-speech segment is synthesized according to a pre-trained TTS model. The pseudo-speech segment and the original audio data are the same text. The pseudo-speech segment is fused with the original audio data with a low weight to obtain a synthetic timbre artifact. Then, time mismatch artifact simulation is performed based on speech rate modification, local silence segment addition, and background audio superposition to obtain a pseudo-augmented sample. A discrimination boundary optimization is performed according to the forged enhancement samples and the hidden generated artifact capture enhancement samples, and the discrimination boundary optimization is based on a random sampling mechanism.

[0009] Furthermore, the perceptually invariant perturbation branch performs a step of hiding and generating artifact capture enhancement according to the original audio data, specifically comprising: According to the perturbation generation strategy of perceptual distance constraint, a perturbation segment is generated for each audio segment in the original audio data. The perturbation generation strategy of perceptual distance constraint is specifically as follows: , , in, represents the perturbation fragment, Represents the original audio segment in the raw audio data, represents the perturbation coefficient, represents the intermediate representation of the deep feature extraction model, represents the perceived speech quality indicator, Represents the human ear perception threshold control item, Indicates the upper limit of the disturbance amplitude.

[0010] Furthermore, the step of extracting features from the original audio data after data enhancement to obtain multi-level deep audio features and physical acoustic features respectively includes: Multi-level deep audio feature extraction is performed on the original audio data based on the cross-language speech representation learning model. The specific algorithm of the multi-level deep audio feature extraction is as follows: , in, Represents multi-level deep audio features, Represents feature connection, represents the output vector of each transformer layer, Indicates the number of layers; Then, physical acoustic features are obtained based on the original audio data. The physical acoustic features include time domain features, frequency domain features, sub-band energy features and frequency trajectory features. The time domain features include short-time energy and zero-crossing rate. The frequency domain features include spectral centroid and spectral roll-off index. The sub-band energy features divide the frequency bands based on the Mel filter group and extract the energy of each sub-band. The frequency trajectory features are based on the instantaneous main frequency and the first-order difference standard deviation of the instantaneous main frequency.

[0011] Furthermore, the step of performing feature fusion processing based on the multi-level deep audio features and the physical acoustic features to obtain fusion features specifically includes: Align the multi-level deep audio features with the physical acoustic features in the time dimension; The multi-level deep audio features are fused with the physical acoustic features according to an adaptive fusion algorithm to obtain fusion features. The adaptive fusion algorithm includes a channel attention mechanism and a gating mechanism. The adaptive fusion algorithm specifically includes: , in, represents the fusion feature, represents a spatially gated tensor, 、 Represents the channel attention vectors of multi-level deep audio features and physical acoustic features, Represents multi-level deep audio features, Represents physical acoustic features, and the channel attention vector is generated based on global pooling and shared multi-layer perceptron.

[0012] Furthermore, the step of performing multi-scale attention enhancement processing on the fusion feature to obtain a multi-scale attention enhancement feature specifically includes: The fusion feature is subjected to multi-scale attention enhancement processing, and the multi-scale attention enhancement processing is based on two parallel convolution branches, and the two parallel convolution branches include Convolution branches and Convolution branch; The fusion feature is divided into channels to obtain multiple feature sub-tensors, and the feature sub-tensors are input into Convolution branches and Convolution branch; described The convolution branch includes a one-dimensional global average pool along the horizontal dimension and a one-dimensional global average pool along the vertical dimension to obtain horizontal dimension position information and vertical dimension position information respectively. The specific algorithm of the convolution branch is as follows: , , in, Indicates horizontal dimension position information, Indicates vertical dimension position information, 、 Represent the time dimension and frequency dimension respectively, 、 Represents the index of frequency dimension and time dimension, represents the fused feature vector; described The convolution branch is based on the cross-space information aggregation mechanism. The convolution branch includes Convolution branch path and Convolution branch path, the The convolution branch path encodes global spatial information based on two-dimensional global average pooling. The specific algorithm of the two-dimensional global average pooling is as follows: , in, represents the two-dimensional global average pooling output; The The convolution branch and the The output of the convolution branch is subjected to sub-tensor merging processing. The specific algorithm of the sub-tensor merging processing is as follows: , in, represents the multi-scale attention enhancement feature, Indicates tensor merging and reorganization, represents multi-scale attention enhancement processing, represents the feature subtensor, Indicates the number of channel divisions.

[0013] Furthermore, the step of performing classification according to the multi-scale attention enhancement feature to obtain the final audio deep fake detection result specifically includes: Add the multi-scale attention-enhanced features on dimension 1 to reshape the dimension; Then, the reshaped multi-scale attention enhancement features are extracted based on the two-dimensional maximum pooling to obtain the main features; According to the main feature, classification is performed to obtain a final audio deepfake detection result, and a specific algorithm of the classification is as follows: , Among them, indicates the final audio deepfake detection result, 、 indicates the full connection layer weight of the first layer and the third layer, 、 indicates the full connection layer bias of the first layer and the third layer, indicates the main feature, indicates the flattening processing, indicates the two-dimensional maximum pooling processing.

[0014] The application provides an audio deepfake detection system fusing multi-source features and cross-scale modeling, comprising: A data enhancement module is configured to obtain original audio data and perform data enhancement processing, wherein the data enhancement processing is based on a double-branch data enhancement strategy, and the double-branch data enhancement strategy comprises a semantic artifact injection branch and a perception-invariant disturbance branch. A feature extraction module is configured to extract features from the data-enhanced original audio data to obtain multi-level deep audio features and physical acoustic features, wherein the multi-level deep audio features are based on a cross-language speech representation learning model. A feature fusion module is configured to perform feature fusion processing on the multi-level deep audio features and the physical acoustic features to obtain fusion features, wherein the feature fusion processing is based on a channel attention mechanism and a gating mechanism. A multi-scale attention enhancement module is configured to perform multi-scale attention enhancement processing on the fusion features to obtain multi-scale attention enhancement features, wherein the multi-scale attention enhancement processing is based on two parallel convolution branches. A classification module is configured to classify the multi-scale attention enhancement features to obtain a final audio deepfake detection result.

[0015] The application further provides a storage medium storing one or more programs, wherein the programs are executed by a processor to implement the audio deepfake detection method fusing multi-source features and cross-scale modeling as described above.

[0016] The application further provides a computer device comprising a memory and a processor, wherein: The memory is configured to store a computer program; The processor is configured to execute the computer program stored in the memory to implement the audio deepfake detection method fusing multi-source features and cross-scale modeling as described above. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 This is a flowchart of the audio deepfake detection method that integrates multi-source features and cross-scale modeling proposed in the first embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the audio deep fake detection system that integrates multi-source features and cross-scale modeling proposed in the second embodiment of the present invention.

[0018] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0019] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate several embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present invention.

[0020] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0022] See also Figure 1 , shown is a flowchart of an audio deep fake detection method that integrates multi-source features and cross-scale modeling, proposed in a first embodiment of the present invention. The audio deep fake detection method that integrates multi-source features and cross-scale modeling includes steps S01 to S05, wherein: Step S01: Obtain original audio data and perform data enhancement processing; It should be noted that in this embodiment, the data enhancement processing is based on a dual-branch data enhancement strategy, which includes a semantic artifact injection branch and a perceptually invariant perturbation branch. The original audio data is obtained and input into the semantic artifact injection branch and the perceptually invariant perturbation branch respectively. The semantic artifact injection branch performs a fake-like enhancement sample construction according to the original audio data, the perceptually invariant perturbation branch performs a hidden-generated artifact capture enhancement according to the original audio data, and the hidden-generated artifact capture enhancement is based on a perceptual distance constraint-based perturbation generation strategy; The fake-like enhancement sample construction performs local band masking, band-limit inversion, and time-domain nonlinear stretching on a mel-spectrum to simulate deep-synthesized audio nonlinear distortion, and then synthesizes a pseudo-speech segment based on a voice style transfer strategy and a pre-trained TTS model, the pseudo-speech segment and the original audio data are the same text, and the pseudo-speech segment and the original audio data are fused with low weight to obtain a synthesized timbre artifact, and then time mismatch artifact simulation is performed according to speech speed modification processing, local mute segment addition, and background audio superposition to obtain a fake-like enhancement sample. Discrimination boundary optimization is performed according to the fake-like enhancement sample and the hidden-generated artifact capture enhancement sample, and the discrimination boundary optimization is based on a random sampling mechanism.

[0023] According to the perceptual distance constraint-based perturbation generation strategy, a perturbation segment is generated for each audio segment in the original audio data, and the perceptual distance constraint-based perturbation generation strategy is as follows: , , wherein, represents a perturbation segment, represents an original audio segment in the original audio data, represents a perturbation coefficient, represents an intermediate representation of a deep feature extraction model, represents a perceptual speech quality index, represents a human ear perceptual threshold control item, represents a perturbation amplitude upper limit.

[0024] Step S02: Feature extraction is performed on the data-enhanced original audio data to obtain multi-level deep audio features and physical acoustic features, respectively. It should be noted that in the embodiment, the multi-level deep audio features are based on a cross-language speech representation learning model, and the large-scale cross-language speech representation learning model in the embodiment is derived from wav2vec 2.0, which is trained on 128 languages. The original audio signal is first processed by a feature encoder including a plurality of convolutional neural networks (CNN), which extracts a 1024-dimensional vector representation every 20ms through a 25ms receptive field to obtain a latent representation. These encoder features are then fed into a 24-layer transformer network to derive a context representation. The output vectors from all 24 transformer layers are concatenated into a composite representation. In the embodiment, all 24 transformer hidden layer outputs are retained and the hidden layer features of all layers are concatenated. The multi-level deep audio features are extracted from the original audio data based on the cross-language speech representation learning model. The specific algorithm for multi-level deep audio feature extraction is as follows: , Wherein, represents the multi-level deep audio features, represents feature concatenation, represents the output vector of each transformer layer, represents the number of layers. The physical acoustic features are obtained from the original audio data, including time domain features, frequency domain features, sub-band energy features, and frequency trajectory features. The time domain features include short-time energy and zero-crossing rate. The frequency domain features include spectral centroid and spectral roll-off index. The sub-band energy features are based on the division of frequency bands by a Mel filter bank and the extraction of sub-band energy. The frequency trajectory features are based on the instantaneous fundamental frequency and the first-order difference standard deviation of the instantaneous fundamental frequency.

[0025] Step S03: performing feature fusion processing based on the multi-level deep audio features and the physical acoustic features to obtain fusion features; It should be noted that in the embodiment, the feature fusion processing is based on a channel attention mechanism and a gating mechanism to align the time dimension of the multi-level deep audio features and the physical acoustic features. The multi-level deep audio features and the physical acoustic features are fused based on an adaptive fusion algorithm to obtain fusion features. The adaptive fusion algorithm includes a channel attention mechanism and a gating mechanism. The adaptive fusion algorithm specifically includes: , Wherein, represents the fusion features, represents a spatial gating tensor, , represent the channel attention vectors of the multi-level deep audio features and the physical acoustic features, respectively, represents the multi-level deep audio features, Represents physical acoustic features, and the channel attention vector is generated based on global pooling and shared multi-layer perceptron.

[0026] Step S04: performing multi-scale attention enhancement processing on the fused features to obtain multi-scale attention enhancement features; It should be noted that in this embodiment, the multi-scale attention enhancement processing is based on two parallel convolution branches, and the fusion features are subjected to multi-scale attention enhancement processing. The multi-scale attention enhancement processing is based on two parallel convolution branches, and the two parallel convolution branches include Convolution branches and Convolution branch; The fusion feature is divided into channels to obtain multiple feature sub-tensors, and the feature sub-tensors are input into Convolution branches and Convolution branch; described The convolution branch includes a one-dimensional global average pool along the horizontal dimension and a one-dimensional global average pool along the vertical dimension to obtain horizontal dimension position information and vertical dimension position information respectively. The specific algorithm of the convolution branch is as follows: , , in, Indicates horizontal dimension position information, Indicates vertical dimension position information, 、 Represent the time dimension and frequency dimension respectively, 、 Represents the index of frequency dimension and time dimension, represents the fused feature vector; described The convolution branch is based on the cross-space information aggregation mechanism. The convolution branch includes Convolution branch path and Convolution branch path, the The convolution branch path encodes global spatial information based on two-dimensional global average pooling. The specific algorithm of the two-dimensional global average pooling is as follows: , in, represents the two-dimensional global average pooling output; The The convolution branch and the The output of the convolution branch is subjected to sub-tensor merging processing. The specific algorithm of the sub-tensor merging processing is as follows: , wherein, represents a multi-scale attention enhanced feature, represents tensor merging and reorganization, represents a multi-scale attention enhancement process, represents a feature sub-tensor, represents the number of channel divisions.

[0027] Step S05: classifying according to the multi-scale attention enhanced feature to obtain a final audio deepfake detection result; It should be noted that in this embodiment, the multi-scale attention enhanced feature is added in dimension 1 to reshape the dimension; Then, according to the two-dimensional maximum pooling, the multi-scale attention enhanced feature after reshaping the dimension is extracted to obtain the main feature; According to the main feature, classification is performed to obtain a final audio deepfake detection result, and the specific algorithm of the classification is as follows: , wherein, represents the final audio deepfake detection result, , represents the full connection layer weight of the first layer and the third layer, , represents the full connection layer bias of the first layer and the third layer, represents the main feature, represents the flattening process, represents the two-dimensional maximum pooling process.

[0028] In summary, according to the above-mentioned audio deepfake detection method fusing multi-source features and cross-scale modeling, the recognition ability of hidden fake patterns in synthesized samples is effectively improved through a double-branch data enhancement strategy, while maintaining the naturalness and auditory acceptability of the speech, representative fake disturbance spaces are introduced, thereby significantly improving the generalization and robustness under multiple types of deepfake attacks. Multi-level deep audio features and physical acoustic features are obtained respectively, and feature fusion is performed, combined with channel attention and gating strategy, the expression ability of fake related information is strengthened, the sensitivity to physical layer artifacts is improved, and the detection accuracy is further improved. A fake pattern recognition framework with multi-scale and spatial context modeling capabilities is constructed through multi-scale attention enhancement, which can efficiently capture the weak change features of cross-time and frequency scale in fake audio. Through the structural design of grouped convolution and multi-branch attention path, local and global perception information is effectively integrated to provide strong expressive intermediate representation for the fake detection task. The accuracy, robustness and generalization ability of the audio deepfake detection are improved. Specifically, the original audio data is obtained and data enhancement processing is performed, the data enhancement processing is based on a double-branch data enhancement strategy, the double-branch data enhancement strategy includes a semantic artifact injection branch and a perception invariant disturbance branch, which effectively improves the recognition ability of hidden fake patterns in synthesized samples, while maintaining the naturalness and auditory acceptability of the speech, representative fake disturbance spaces are introduced, thereby significantly improving the generalization and robustness under multiple types of deepfake attacks. The original audio data after data enhancement is subjected to feature extraction to obtain multi-level deep audio features and physical acoustic features, the multi-level deep audio features are based on a cross-language speech representation learning model, and feature fusion processing is performed based on the multi-level deep audio features and the physical acoustic features to obtain fusion features. The feature fusion processing is based on a channel attention mechanism and a gating mechanism, which strengthens the expression ability of fake related information and improves the sensitivity to physical layer artifacts, thereby further improving the detection accuracy. The fusion features are subjected to multi-scale attention enhancement processing to obtain multi-scale attention enhancement features, the multi-scale attention enhancement processing is based on two parallel convolution branches, and a fake pattern recognition framework with multi-scale and spatial context modeling capabilities is constructed, which can efficiently capture the weak change features of cross-time and frequency scale in fake audio. Through the structural design of grouped convolution and multi-branch attention path, local and global perception information is effectively integrated to provide strong expressive intermediate representation for the fake detection task. Classification is performed based on the multi-scale attention enhancement features to obtain the final audio deepfake detection result, and the accuracy, robustness and generalization ability of the audio deepfake detection are improved.

[0029] Please refer to Figure 2As shown in the structural schematic diagram of the audio deepfake detection system of the second embodiment of the present application, the system comprises: A data enhancement module 10 is configured to acquire original audio data and perform data enhancement processing, wherein the data enhancement processing is based on a double-branch data enhancement strategy, and the double-branch data enhancement strategy comprises a semantic artifact injection branch and a perception-invariant perturbation branch. A feature extraction module 20 is configured to perform feature extraction on the original audio data after data enhancement, so as to acquire multi-level deep audio features and physical acoustic features, wherein the multi-level deep audio features are based on a cross-lingual speech representation learning model. A feature fusion module 30 is configured to perform feature fusion processing based on the multi-level deep audio features and the physical acoustic features, so as to acquire fusion features, wherein the feature fusion processing is based on a channel attention mechanism and a gating mechanism. A multi-scale attention enhancement module 40 is configured to perform multi-scale attention enhancement processing on the fusion features, so as to acquire multi-scale attention enhancement features, wherein the multi-scale attention enhancement processing is based on two parallel convolution branches. A classification module 50 is configured to perform classification based on the multi-scale attention enhancement features, so as to acquire a final audio deepfake detection result.

[0030] The present application also provides a computer storage medium having one or more programs stored thereon, which programs, when executed by a processor, implement the audio deepfake detection method of the present application.

[0031] The present application also provides a computer device comprising a memory and a processor, wherein the memory is configured to store a computer program, and the processor is configured to execute the computer program stored in the memory, so as to implement the audio deepfake detection method of the present application.

[0032] Those skilled in the art can understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequence list of executable instructions for implementing the logic function, which can be specifically implemented in any computer readable medium for use by or in conjunction with an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor or other system that can fetch and execute instructions from the instruction execution system, device or apparatus. For the present specification, the "computer readable medium" can be any device that can contain a storage, communication, propagation or transmission program for use by or in conjunction with the instruction execution system, device or apparatus.

[0033] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: a portable computer diskette (magnetic device); a random access memory (RAM); a read-only memory (ROM); an erasable programmable read-only memory (EPROM or Flash memory); an optical fiber device; and a portable compact disc read-only memory (CDROM). Additionally, the computer readable medium can be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for example, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0034] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the embodiments described above, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following techniques, which are well known in the art, can be used to implement the application: a hybrid of the techniques mentioned above; a combination of one or more of the techniques mentioned above; or one or more other techniques suitable for use in the computer hardware devices described above.

[0035] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in one or more embodiments or examples.

[0036] The above-described embodiments are merely some embodiments of the present application, and the description is specific and detailed, but should not be understood as limiting the scope of the patent of the present application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. An audio deepfake detection method that integrates multi-source features and cross-scale modeling, characterized by: include: Acquire original audio data and perform data augmentation processing based on a dual-branch data augmentation strategy, comprising a semantic artifact injection branch and a perceptually invariant perturbation branch; Performing feature extraction on the data-augmented raw audio data to obtain multi-level deep audio features and physical acoustic features, respectively, wherein the multi-level deep audio features are based on a cross-language speech representation learning model; Performing feature fusion processing based on the multi-level deep audio features and the physical acoustic features to obtain fusion features, wherein the feature fusion processing is based on a channel attention mechanism and a gating mechanism; Performing a multi-scale attention enhancement process on the fused features to obtain a multi-scale attention enhancement feature, wherein the multi-scale attention enhancement process is based on two parallel convolution branches; Classification is performed based on the multi-scale attention-enhanced features to obtain the final audio deepfake detection result.

2. The audio deepfake detection method integrating multi-source features and cross-scale modeling according to claim 1 is characterized in that: The step of obtaining the original audio data and performing data enhancement processing specifically includes: Obtain the original audio data and input it into the semantic artifact injection branch and the perceptually invariant perturbation branch respectively; The semantic artifact injection branch constructs a forgery-like enhancement sample based on the original audio data, and the perceptually invariant perturbation branch performs hidden generation artifact capture enhancement based on the original audio data, wherein the hidden generation artifact capture enhancement is based on a perturbation generation strategy constrained by perceptual distance; The pseudo-augmented sample construction performs local band masking, band-limited inversion, and time-domain nonlinear stretching on the Mel spectrum to simulate nonlinear distortion of deep synthetic audio. Then, based on the speech style transfer strategy, a pseudo-speech segment is synthesized according to a pre-trained TTS model. The pseudo-speech segment and the original audio data are the same text. The pseudo-speech segment is fused with the original audio data with a low weight to obtain a synthetic timbre artifact. Then, time mismatch artifact simulation is performed based on speech rate modification, local silence segment addition, and background audio superposition to obtain a pseudo-augmented sample. A discrimination boundary optimization is performed according to the forged enhancement samples and the hidden generated artifact capture enhancement samples, and the discrimination boundary optimization is based on a random sampling mechanism.

3. The audio deepfake detection method integrating multi-source features and cross-scale modeling according to claim 2 is characterized in that: The perceptually invariant perturbation branch performs a step of hiding and generating artifact capture enhancement according to the original audio data, specifically comprising: According to the perturbation generation strategy of perceptual distance constraint, a perturbation segment is generated for each audio segment in the original audio data. The perturbation generation strategy of perceptual distance constraint is specifically as follows: ， ， in, represents the perturbation fragment, Represents the original audio segment in the raw audio data, represents the perturbation coefficient, represents the intermediate representation of the deep feature extraction model, represents the perceived speech quality indicator, Represents the human ear perception threshold control item, Indicates the upper limit of the disturbance amplitude.

4. The audio deepfake detection method integrating multi-source features and cross-scale modeling according to claim 1 is characterized in that: The step of extracting features from the raw audio data after data enhancement to obtain multi-level deep audio features and physical acoustic features, specifically includes: Multi-level deep audio feature extraction is performed on the original audio data based on the cross-language speech representation learning model. The specific algorithm of the multi-level deep audio feature extraction is as follows: ， in, Represents multi-level deep audio features, Represents feature connection, represents the output vector of each transformer layer, Indicates the number of layers; Then, physical acoustic features are obtained based on the original audio data. The physical acoustic features include time domain features, frequency domain features, sub-band energy features and frequency trajectory features. The time domain features include short-time energy and zero-crossing rate. The frequency domain features include spectral centroid and spectral roll-off index. The sub-band energy features divide the frequency bands based on the Mel filter group and extract the energy of each sub-band. The frequency trajectory features are based on the instantaneous main frequency and the first-order difference standard deviation of the instantaneous main frequency.

5. The audio deepfake detection method integrating multi-source features and cross-scale modeling according to claim 1 is characterized in that: The step of performing feature fusion processing based on the multi-level deep audio features and the physical acoustic features to obtain fusion features specifically includes: Align the multi-level deep audio features with the physical acoustic features in the time dimension; The multi-level deep audio features are fused with the physical acoustic features according to an adaptive fusion algorithm to obtain fusion features. The adaptive fusion algorithm includes a channel attention mechanism and a gating mechanism. The adaptive fusion algorithm specifically includes: ， in, represents the fusion feature, represents a spatially gated tensor, 、 Represents the channel attention vectors of multi-level deep audio features and physical acoustic features, Represents multi-level deep audio features, Represents physical acoustic features, and the channel attention vector is generated based on global pooling and shared multi-layer perceptron.

6. The audio deepfake detection method integrating multi-source features and cross-scale modeling according to claim 1 is characterized in that: The step of performing multi-scale attention enhancement processing on the fusion feature to obtain a multi-scale attention enhancement feature specifically includes: The fusion feature is subjected to multi-scale attention enhancement processing, and the multi-scale attention enhancement processing is based on two parallel convolution branches, and the two parallel convolution branches include Convolution branches and Convolution branch; The fusion feature is divided into channels to obtain multiple feature sub-tensors, and the feature sub-tensors are input into Convolution branches and Convolution branch; described The convolution branch includes a one-dimensional global average pool along the horizontal dimension and a one-dimensional global average pool along the vertical dimension to obtain horizontal dimension position information and vertical dimension position information respectively. The specific algorithm of the convolution branch is as follows: ， ， in, Indicates horizontal dimension position information, Indicates vertical dimension position information, 、 Represent the time dimension and frequency dimension respectively, 、 Represents the index of frequency dimension and time dimension, represents the fused feature vector; described The convolution branch is based on the cross-space information aggregation mechanism. The convolution branch includes Convolution branch path and Convolution branch path, the The convolution branch path encodes global spatial information based on two-dimensional global average pooling. The specific algorithm of the two-dimensional global average pooling is as follows: ， in, represents the two-dimensional global average pooling output; The The convolution branch and the The output of the convolution branch is subjected to sub-tensor merging processing. The specific algorithm of the sub-tensor merging processing is as follows: ， in, represents the multi-scale attention enhancement feature, Indicates tensor merging and reorganization, represents multi-scale attention enhancement processing, represents the feature subtensor, Indicates the number of channel divisions.

7. The audio deepfake detection method integrating multi-source features and cross-scale modeling according to claim 1 is characterized in that: The step of classifying according to the multi-scale attention enhancement features to obtain the final audio deep fake detection result specifically includes: Add the multi-scale attention-enhanced features on dimension 1 to reshape the dimension; Then, the reshaped multi-scale attention enhancement features are extracted based on the two-dimensional maximum pooling to obtain the main features; Classification is performed based on the main features to obtain the final audio deep fake detection result. The specific algorithm of the classification is as follows: ， in, Represents the final audio deepfake detection result, 、 represents the weights of the first and third fully connected layers, 、 represents the bias of the first and third fully connected layers, Indicates the main features, Indicates flattening. Represents two-dimensional maximum pooling processing.

8. An audio deepfake detection system that integrates multi-source features and cross-scale modeling, characterized by: include: A data enhancement module is used to obtain raw audio data and perform data enhancement processing based on a dual-branch data enhancement strategy, which includes a semantic artifact injection branch and a perceptually invariant perturbation branch; A feature extraction module is used to extract features from the raw audio data after data enhancement to obtain multi-level deep audio features and physical acoustic features, wherein the multi-level deep audio features are based on a cross-language speech representation learning model; A feature fusion module, configured to perform feature fusion processing based on the multi-level deep audio features and physical acoustic features to obtain fused features, wherein the feature fusion processing is based on a channel attention mechanism and a gating mechanism; A multi-scale attention enhancement module, configured to perform multi-scale attention enhancement processing on the fused features to obtain multi-scale attention enhancement features, wherein the multi-scale attention enhancement processing is based on two parallel convolution branches; A classification module is used to perform classification based on the multi-scale attention-enhanced features to obtain a final audio deepfake detection result.

9. A storage medium, characterized in that: The storage medium stores one or more programs, which, when executed by the processor, implement the audio deep fake detection method that integrates multi-source features and cross-scale modeling as described in any one of claims 1-7.

10. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, it implements the audio deep fake detection method that integrates multi-source features and cross-scale modeling as described in any one of claims 1-7.

Images