Multi-language deep synthetic speech adaptive detection method in cross-border communication

Abstract

The invention discloses a multi-language deep synthesis voice adaptive detection method in cross-border communication, and the method comprises the steps: S100, receiving a cross-border voice stream and associated metadata in real time through a communication protocol interface, the metadata comprising equipment information, a network protocol packet header and session time sequence information; s200, constructing a multi-modal feature extraction pipeline, wherein multi-modal features comprise an audio mode, a text mode and a behavior mode; s300, inputting the features into a multi-modal adaptive fusion engine: outputting a language tag and confidence c through a language recognition module, and dynamically adjusting modal weight; feature fusion is realized by adopting a gating multi-mode unit; s400, executing by a detection decision-making layer, and S500, when the detection confidence coefficient is gt; and when 90%, incremental learning is triggered. According to the method, weight dynamic allocation driven by languages is adopted, and cross-language BERT semantic verification is combined, so that the low-resource language detection accuracy is greatly improved, VPN disguise is effectively recognized, the generative attack recognition rate is improved, and the success rate of confronting sample attacks is greatly reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of synthetic speech authentication, in particular to a multi-language deep synthetic speech adaptive detection method in cross-border communication. BACKGROUND

[0002] The current deep synthetic speech (Deepfake speech) detection technology mainly focuses on single language scenarios, and the mainstream methods include: Acoustic feature analysis: CNN / LSTM model based on MFCC, fundamental frequency trajectory and other features; Spectrum detection: use GAN to generate artifact frequency domain features; End-to-end detection: use the fine-tuning scheme of the pre-trained model Wav2Vec 2.0.

[0003] However, the existing technology has many deficiencies in the adaptive detection of multi-language deep synthetic speech in cross-border communication, such as insufficient multi-language adaptability, strong language dependence, and the need for independent model training for each language, resulting in high-resource language (English / Chinese) detection accuracy > 95%, but low-resource language (such as Swahili) drops to < 65%; Unable to handle mixed language conversations, cross-language generalization fails: acoustic features drift between languages, resulting in unaligned phoneme-level features, and unable to capture cross-language synthetic trace commonalities.

[0004] Lack of behavior modal fusion, vulnerability of pure audio detection: ignoring communication behavior characteristics, resulting in an attack success rate of > 80% for adversarial samples, and unable to identify IP spoofing (such as simulating the acoustic environment of the target area after VPN jump); Environmental evidence blank, no device microphone fingerprint library established, difficult to detect, mismatch between synthetic speech and recording device physical characteristics, network transmission delay jitter anomaly (synthetic speech often accompanied by artificial delay injection). SUMMARY

[0005] In this embodiment, a multi-language deep synthetic speech adaptive detection method, system, electronic device and storage medium in cross-border communication are provided to solve the problems of insufficient multi-language adaptability, lack of behavior modal fusion and environmental evidence blank in related technologies.

[0006] In a first aspect, the present application embodiment provides a multi-language deep synthetic speech adaptive detection method in cross-border communication, which comprises the following steps: S100, real-time receive cross-border voice stream and associated metadata through a communication protocol interface, wherein the metadata includes device information, network protocol packet header and session timing information; S200, construct a multi-modal feature extraction pipeline, multi-modal features include audio modal, text modal and behavior modal; Audio modal: adopt a three-level feature extraction structure, the first level extracts time-frequency spectrum through a 256-dimensional mel filter bank, the second level calculates the fundamental frequency trajectory and its first derivative, and the third level extracts phase distortion features through a complex convolution network; Text modal: call a multi-language ASR engine to generate initial translated text, and calculate the semantic consistency score between the source language and the target language through a cross-language BERT model; Behavior modal: analyze the Jitter value of the speech packet transmission interval, and extract the environmental formant feature in the microphone frequency response curve; S300, input the features into a multi-modal adaptive fusion engine: Output language tags and confidence c through a language recognition module, dynamically adjust modal weights, including audio weight , text weight , and behavior weight ; Adopt a gated multi-modal unit to realize feature fusion

[0007] S400, detect the decision layer to execute, input the fusion features into a classifier composed of three residual blocks, and output original and synthetic binary classification probabilities; Through an attack type recognition module, judge cloning, conversion and generation attacks based on spectral residual mode matching; S500, when the detection confidence is > 90%, trigger incremental learning: Cache the sample locally, and start federated learning when 50 samples are accumulated; Adopt Reptile meta-learning algorithm to update feature extractor parameters.

[0008] In an optional embodiment, the audio modal feature extraction specifically includes: Extract frame-level acoustic representation through a pre-trained Wav2Vec 2.0 model; Calculate the fundamental frequency value of each frame , Z-score standardization is performed on the change rate of adjacent 10 frames; Generate a phase spectrum using complex STFT, and detect phase discontinuity points caused by artificial synthesis through a pre-trained DeepPhaseCNN; Perform CQT transformation on the mel spectrum to detect the grid-like artifacts unique to the generative adversarial network in the time-frequency domain.

[0009] In an optional embodiment, the cross-language semantic consistency verification includes: Semantic similarity is calculated using CLS vector of multilingual BERT. Acoustic-text alignment verification, phoneme boundary is calculated using Montreal Forced Aligner; Detect abnormal alignment, when the energy variance of clear consonant segment is greater than threshold or the formant slope of vowel segment is abnormal; back-translation consistency check

[0010] In an optional embodiment, the behavior modal analysis includes: Device fingerprint construction, collect impulse response of device in 0-8kHz frequency band, extract first 6 formant frequencies to form fingerprint vector; Geographical consistency verification, when IP home location and device time zone deviation> 3 hours and acoustic environment is closed space, activate suspicious marker; Transmission anomaly detection, calculate coefficient of variation of speech packet inter-arrival time When >0.25 and there is burst retransmission, mark as abnormal.

[0011] In an optional embodiment, the dynamic weight allocation mechanism satisfies: When LID confidence c<0.6, set to ; When VPN tunnel is detected, set weight to zero and activate audio-text collaborative analysis mode; For resource scarce languages, set =0.3, =0.6, =0.1.

[0012] In an optional embodiment, the incremental learning includes the following steps: Deploy lightweight detection model on edge node, store new samples through secure enclave; Adopt prototype-based few-shot learning: Calculate synthetic speech class center: ; Wherein, is the current number of synthetic samples, is the feature extraction function, is the th synthetic speech sample; Optimization objective: ; Wherein, is the Euclidean distance, is the new sample, is the real speech class center, is a boundary threshold value; upload model gradients through a differential privacy mechanism.

[0013] Compared with the prior art, the multi-language deep-synthesized voice adaptive detection method in cross-border communication has the following beneficial effects: The application significantly improves the detection performance of deep-synthesized voice in cross-border communication through multi-modal dynamic fusion and adaptive detection mechanism, and the specific effects are as follows: Adopting language-driven weight dynamic allocation, combining cross-language BERT semantic verification, the detection accuracy of low-resource language is greatly improved, Facebook MMS model supports 517 languages, solves the acoustic feature drift problem in mixed language dialogue, behavior modal fusion (device fingerprint + transmission jitter analysis) effectively identifies VPN camouflage, geographic consistency verification keeps the detection accuracy at 86.5% in IP spoofing scenarios, the combination of phase distortion features (DeepPhaseCNN) and GAN artifact detection improves the recognition rate of generated attacks, and the success rate of adversarial sample attacks is greatly reduced, the dual-path fusion architecture (GMU main path + auxiliary voting path) realizes 18.7ms ultra-low delay detection, and exhibits real-time and self-evolution ability. The federal incremental learning mechanism (Reptile algorithm + differential privacy) makes the adaptation speed of new attack types increase by 8.3 times, and guarantees data privacy.

[0014] In a second aspect, an embodiment of the application provides a multi-language deep-synthesized voice adaptive detection system in cross-border communication, comprising: A multi-modal acquisition module supports RTP / SRTP / WebRTC protocols, including an audio splitter, a metadata parser, and a data desensitization unit; An extensible feature engine includes: An audio processing chain includes a voiceprint extraction interface of an open-source SpeechBrain framework and a custom phase analysis module; A text processing chain integrates a Facebook MMS model for low-resource language ASR; A behavior analysis chain is based on a high-performance network traffic probe developed by DPDK; An adaptive fusion core includes: A weight allocator receives LID confidence and language resource level signals in real time; A multi-modal attention network uses a multi-head cross-attention mechanism to calculate feature correlation; A defense enhancement module includes an adversarial sample generator, a model integrated voter, and an alarm feedback closed loop.

[0015] In an optional embodiment, the adaptive fusion core adopts a dual-path structure: Main path: input multi-modal features into the cascaded GMU units, and the output formula is: ; ; ; wherein, is the feature transformation result, is the gating signal vector, and are weight matrices, and are bias vectors, is the hyperbolic tangent activation function, is the final output feature; Auxiliary path: input each modality feature into a dedicated detector, and generate an auxiliary decision through weighted voting; The final output is: ; wherein, is the final synthesis probability, is the main model confidence weight, is the main fusion network output probability, is the auxiliary model voting probability; adjusted dynamically by the model confidence.

[0016] In a third aspect, an electronic device is provided, including a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory complete mutual communication through the bus, and the processor can invoke logical instructions in the memory to execute steps of the method provided in the first aspect.

[0017] In a fourth aspect, a non-transitory computer readable storage medium is provided, having a computer program stored thereon, which, when executed by a processor, implements steps of the multi-language deep synthesis voice adaptive detection method in cross-border communication provided in the first aspect.

[0018] Compared with the prior art, the multi-language deep synthesis voice adaptive detection system in cross-border communication, the electronic device and the storage medium of the present application have the same beneficial effects as the multi-language deep synthesis voice adaptive detection method in cross-border communication of the first aspect, and therefore will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required by the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0020] Figure 1 The flow chart of the multilingual deep synthesis voice adaptive detection method in cross-border communication in the embodiment of the present application; Figure 2 The structural block diagram of the multilingual deep synthesis voice adaptive detection system in cross-border communication in the embodiment of the present application; Figure 3 The structural block diagram of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application will be described and explained in the following with reference to the drawings and embodiments.

[0022] Unless otherwise defined, the technical terms or scientific terms involved in the present application shall have the general meanings understood by those skilled in the art. In the present application, "one", "a", "an", "the", "these" and the like similar words do not represent quantitative limitation, and they can be singular or plural. In the present application, the terms "include", "contain", "have" and any variants thereof have the purpose of covering non-exclusive inclusion; for example, the process, method and system, product or device containing a series of steps or modules (units) are not limited to the listed steps or modules (units), but can include steps or modules (units) not listed, or can include other steps or modules (units) inherent to the process, method, product or device. In the present application, the terms "connection", "connection", "coupling" and the like similar words are not limited to physical or mechanical connection, but can include electrical connection, whether direct or indirect. In the present application, "multiple" means two or more. The association between the associated objects is described by "and / or", which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. In general, the character " / " represents the relationship between the front and rear associated objects as "or". In the present application, the terms "first", "second", "third" and the like are only used to distinguish similar objects, and do not represent a specific order for the objects.

[0023] In the embodiment of the present application, a multilingual deep synthesis voice adaptive detection method in cross-border communication is provided, Figure 1is a flowchart of a multilingual deep-synthesis voice adaptive detection method in cross-border communication of the present application, as shown in Figure 1 The flowchart includes the following steps: S100, real-time receiving cross-border voice stream and associated metadata through a communication protocol interface, the metadata including device information, network protocol packet header and session timing information; S200, constructing a multi-modal feature extraction pipeline: Audio modality: adopting a three-level feature extraction structure, the first level extracting time-frequency spectrum through a 256-dimensional Mel filter bank, the second level calculating the fundamental frequency trajectory and its first derivative, and the third level extracting phase distortion features through a complex convolution network; The audio modality feature extraction specifically includes: Extracting frame-level acoustic representation through a pre-trained Wav2Vec 2.0 model; Calculating the fundamental frequency value of each frame , the first derivative of the fundamental frequency value of each frame , and the Z-score standardization of the change rate of the fundamental frequency value of adjacent 10 frames; Generating a phase spectrum using a complex STFT and detecting phase discontinuity points caused by artificial synthesis through a pre-trained DeepPhaseCNN; Performing CQT transformation on the Mel spectrum to detect the grid-like artifacts unique to the generative adversarial network in the time-frequency domain.

[0024] Since the standard deviation of the change rate of the fundamental frequency value of the synthesized voice is 3.2 times that of the real voice, the 768-dimensional frame-level vector output by the Wav2Vec2.0 is output with a step of 10 ms, followed by fundamental frequency analysis, and the DeepPhaseCNN model outputs the coordinates of the phase discontinuity points, thereby greatly improving the detection rate of VC voice cloning through the fundamental frequency feature.

[0025] Text modality: calling a multilingual ASR engine to generate initial translated text and calculating the semantic consistency score between the source language and the target language through a cross-language BERT model; It should be noted that the cross-language semantic consistency verification includes: Using the CLS vector of the multilingual BERT to calculate the semantic similarity: ; Wherein, is the semantic similarity score, is the CLS vector of the multilingual BERT model, is the source language text, is the machine translation text, is the vector inner product.

[0026] In this embodiment, the acoustic-text alignment verification includes:​ Compute phoneme boundary using Montreal Forced Aligner; Detect abnormal alignment: when energy variance of unvoiced segment > threshold or formant slope of vowel segment is abnormal; Back translation consistency check: .

[0027] wherein, Back translation consistency score, is a machine translation evaluation metric, is the i-th back translation result, is the original input text.

[0028] Synthesized speech causes BLEU value to drop by >40% in cross-lingual translation, and the formant changes of real pronunciation are smooth, and synthesized speech has mechanical mutations.

[0029] Therefore, in the present embodiment, semantic similarity calculation is first performed, and then acoustic-text alignment is performed, specifically, phoneme boundaries are labeled using Montreal Forced Aligner, and vowel segment formant slope is detected.

[0030] Specifically, In the present embodiment, when is determined to be abnormal. Through semantic verification, the content-preservation attack miss detection rate is reduced, and the acoustic alignment reduces the detection delay.

[0031] Behavioral modalities: analyze the device MAC address to generate a hardware fingerprint, analyze the Jitter value of the speech packet transmission interval, and extract the environmental formant features in the microphone frequency response curve; Specifically, the behavioral modalities analysis includes: Device fingerprint construction: collect the impulse response of the device in the 0-8 kHz frequency band, and extract the first 6 formant frequencies to form a fingerprint vector; illustratively, a 0.5-8 kHz sweep tone is played, the microphone response is recorded, and the formant is extracted as a fingerprint. The uniqueness of the frequency response curve of the mobile phone microphone is used as the device fingerprint.

[0032] Geographical consistency verification: when the IP attribution and device time zone deviation > 3 hours and the acoustic environment is a closed space, activate the suspicious marker; Illustratively, the IP is located in Nairobi (GMT+3), the device time is GMT+8, and the acoustic environment detection is a conference room, such as a reverberation time <0.3s, then the abnormality is marked. The operator in the synthetic attack does not match the time zone of the IP attribution, which is used as one of the verification methods.

[0033] Transmission anomaly detection: calculate the coefficient of variation of the speech packet arrival interval wherein, is the standard deviation of the inter-arrival interval of speech packets, is the mean of the inter-arrival interval of speech packets, when > 0.25 and there is a burst retransmission, mark the anomaly.

[0034] Exemplarily, is 0.4, which is greater than 0.25, then mark the anomaly. S300, input the features into a multi-modal adaptive fusion engine: Output the language tag and confidence c by a language identification module (LID), and dynamically adjust the modal weight: Audio weight ; Text weight ; wherein, is the intersection over union function, is the text output by speech recognition, is the back-translation text.

[0035] Behavior weight ; wherein, is the spherical distance between IP location and acoustic location; is the maximum effective distance threshold.

[0036] In this embodiment, the dynamic weight distribution mechanism satisfies: When the LID confidence c < 0.6, set to ; When a VPN tunnel is detected, set weight to zero and activate the audio-text collaborative analysis mode; For resource-scarce languages (training data < 10 hours), set = 0.3, = 0.6, = 0.1.

[0037] Firstly, based on the decline of the F1 value of the low-resource language acoustic model, the text features need to be relied on, and the VPN can fake the IP but cannot tamper with the acoustic-text alignment features.

[0038] Exemplarily, it is detected that the language is Zulu, and set = 0.3, = 0.6, = 0.1. When the VPN tunnel is triggered, close the behavior weight = 0, and activate the audio-text collaborative mode, thereby greatly increasing the detection accuracy of rare languages.

[0039] Feature fusion is achieved using gated multimodal units: ; in, To fuse feature vectors, For activation function, and For trainable weight matrix, It is a concatenated vector of audio, text, and behavioral features.

[0040] S400, Detection and decision-making level execution: The fused features are input into a classifier consisting of three residual blocks, and the output is the original / synthetic binary classification probability. The attack type identification module identifies cloning / transformation / generative attacks based on spectral residual pattern matching. S500: Incremental learning is triggered when the detection confidence level is >90%. The sample is encrypted and cached locally. Federated learning will begin once 50 samples have been accumulated. The Reptile meta-learning algorithm is used to update the feature extractor parameters.

[0041] This shortens attack response time and improves the speed of adaptation to new attack types. Raw audio remains local, and differential privacy protects against gradient leakage risks.

[0042] In this embodiment, incremental learning is specifically implemented as follows: Deploy lightweight detection models at edge nodes and store new samples in a secure enclave; Employing prototype-based few-shot learning: Computational Speech Synthesis Center: ; in, This represents the current number of synthesized samples. For feature extraction function, For the first One synthesized speech sample.

[0043] Optimization goal: ; in, For Euclidean distance, For new samples, As a center for real voice data, This is the boundary threshold.

[0044] Upload model gradients via differential privacy mechanism: .

[0045] in, The gradient of the model after adding noise. The gradient of the original model. Gaussian noise distribution, is the noise intensity, is the identity matrix.

[0046] FGSM algorithm is used to generate perturbation samples, the robustness of the model is enhanced, a model ensemble voter is used, and three model votes of audio (ResNet18), text (BiLSTM) and behavior (GBDT) are used; false positive samples are automatically added to an incremental learning queue to realize an alarm feedback closed loop.

[0047] The embodiment of the application also provides a multi-language deep synthesis voice adaptive detection system in cross-border communication, which is used to realize the method embodiment and has been described above. The terms "module", "unit", "sub-unit" and the like used below can be a combination of software and / or hardware that realizes a predetermined function. Although the system described in the following embodiment is preferably realized in software, hardware or a combination of software and hardware is also possible and is conceived.

[0048] As Figure 2 shown, Figure 2 is a structural block diagram of the multi-language deep synthesis voice adaptive detection system in cross-border communication in the application, and the system comprises: A multi-modal acquisition module: supporting RTP / SRTP / WebRTC protocols, including an audio splitter, a metadata parser and a data desensitization unit; An extensible feature engine: An audio processing chain: including a voiceprint extraction interface of an open source SpeechBrain framework and a custom phase analysis module; A text processing chain: integrating a Facebook MMS model for low-resource language ASR; A behavior analysis chain: a high-performance network traffic probe developed based on DPDK; An adaptive fusion core: A weight allocator: real-time receiving of LID confidence and language resource level signals; A multi-modal attention network: using a multi-head cross-attention mechanism to calculate feature correlation; A defense enhancement module: including an adversarial sample generator, a model ensemble voter and an alarm feedback closed loop.

[0049] As Figure 3 shown, the adaptive fusion core adopts a double-path structure: A main path: inputting multi-modal features into a cascaded GMU unit, and the output formula is: ; ; ; wherein, is the feature transformation result, is the gating signal vector, and is the weight matrix, and is the bias vector, is the hyperbolic tangent activation function, is the final output feature.

[0050] Auxiliary path: input each modality feature into a dedicated detector (audio: ResNet-18, text: BiLSTM, behavior: GBDT), and generate an auxiliary decision through weighted voting; The final output is: ; is the final synthesis probability, is the main model confidence weight, is the main fusion network output probability, is the auxiliary model voting probability.

[0051] adjusted dynamically by the model confidence.

[0052] Figure 3 The structural block diagram of the electronic device provided by the embodiment of the application is shown in Figure 3 The electronic device can include a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 complete mutual communication through the communications bus 640. The processor 610 can invoke a logical instruction in the memory 630 to execute the following method: S100, real-time receiving a cross-border voice stream and associated metadata through a communications protocol interface, wherein the metadata includes device information, a network protocol packet header, and session timing information; S200, constructing a multi-modal feature extraction pipeline, wherein the multi-modal features include audio modalities, text modalities, and behavior modalities; Audio modalities: adopting a three-level feature extraction structure, the first level extracts a time-frequency spectrum through a 256-dimensional Mel filter bank, the second level calculates a fundamental frequency trajectory and a first-order derivative thereof, and the third level extracts a phase distortion feature through a complex convolution network; Text modalities: calling a multi-language ASR engine to generate initial translated text, and calculating a semantic consistency score between a source language and a target language through a cross-language BERT model; Behavioral modalities: analyze the device MAC address to generate hardware fingerprints, analyze the Jitter value of the voice packet transmission interval, and extract the environmental resonance peak characteristics in the microphone frequency response curve; S300, input the features into the multi-modal adaptive fusion engine: Output the language label and confidence c through the language recognition module, dynamically adjust the modal weight, including audio weight , text weight And behavior weight ; Adopt a gated multi-modal unit to realize feature fusion; S400, detection decision layer execution, input the fusion features into the classifier composed of 3 residual blocks, and output the original and synthesized binary classification probability; Through the attack type recognition module, based on spectral residual mode matching to judge cloning, conversion and generation type attacks; S500, when the detection confidence is greater than 90%, trigger incremental learning: Cache the sample locally, and start federated learning when 50 samples are accumulated; Update the feature extractor parameters using the Reptile meta-learning algorithm.

[0053] In addition, the logical instructions in the above-mentioned memory 630 can be implemented in the form of a software function unit and sold or used as a separate product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and various program code storage media.

[0054] The embodiments of the present application also provide a non-transitory computer readable storage medium having a computer program stored thereon, which is executed by a processor to implement the method provided by the above-mentioned embodiments.

[0055] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and necessary general hardware platforms through the description of the above embodiments, and of course, the implementation can also be through hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method of each embodiment or some parts of the embodiment.

[0056] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some technical features thereof; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting multi-lingual deep synthesized speech adaptation in cross-border communication, the method comprising: The multilingual deep-synthesis voice adaptive detection method in the cross-border communication comprises: S100, receiving cross-border voice stream and associated metadata in real time through a communication protocol interface, wherein the metadata comprises device information, network protocol packet header and session timing information; S200, constructing a multi-modal feature extraction pipeline, wherein the multi-modal features comprise audio modalities, text modalities and behavior modalities; Audio modalities: adopting a three-level feature extraction structure, the first level extracts time-frequency spectrum through a 256-dimensional Mel filter bank, the second level calculates the fundamental frequency trajectory and its first derivative, and the third level extracts phase distortion features through a complex convolution network; Text modalities: calling a multilingual ASR engine to generate initial translation text, and calculating the semantic consistency score between the source language and the target language through a cross-language BERT model; Behavior modalities: analyzing the Jitter value of the voice packet transmission interval, and extracting the environmental formant features in the microphone frequency response curve; S300, inputting the features into a multi-modal adaptive fusion engine: Output language label and confidence c through language recognition module, dynamically adjust modal weight, including audio weight , text weight and behavior weight ; Adopting a gated multi-modal unit to realize feature fusion; S400, executing a detection decision layer, inputting the fused features into a classifier composed of three residual blocks, and outputting original and synthesis binary classification probabilities; Through an attack type identification module, judging cloning, conversion and generation attacks based on spectral residual pattern matching; S500, triggering incremental learning when the detection confidence is greater than 90%: Encrypting the sample locally, and starting federated learning when 50 samples are accumulated; Using the Reptile meta-learning algorithm to update the feature extractor parameters.

2. The method for detecting multilingual deep synthesized voice adaptation in cross-border communication according to claim 1, wherein, The audio modality feature extraction specifically comprises: Extracting frame-level acoustic representation through a pre-trained Wav2Vec 2.0 model; Compute the fundamental frequency value for each frame Z-score normalization of the rate of change for the adjacent 10 frames Z-score normalization of the rate of change for the adjacent 10 frames Generating a phase spectrum using a complex STFT, and detecting phase discontinuity points caused by artificial synthesis through a pre-trained DeepPhaseCNN; Performing CQT transformation on the Mel spectrum to detect the grid-like artifacts unique to the generative adversarial network in the time-frequency domain.

3. The method for detecting multilingual deep synthesized voice adaptation in cross-border communication according to claim 1, wherein, The cross-language semantic consistency verification comprises: Using the CLS vector of the multilingual BERT to calculate the semantic similarity; Acoustic-text alignment verification, using the Montreal Forced Aligner to calculate the phoneme boundary; Detecting abnormal alignment, when the energy variance of the clear consonant segment is greater than a threshold or the Formant slope of the vowel segment is abnormal; back-translation consistency check.

4. The method for detecting multilingual deep synthesized voice adaptation in cross-border communication according to claim 1, characterized in that The behavior modality analysis comprises: Device fingerprint construction, collecting the impulse response of the device in the 0-8kHz frequency band, and extracting the first 6 formant frequencies to form a fingerprint vector; Geographical consistency verification, when the IP home location and the device time zone deviation are greater than 3 hours and the acoustic environment is a closed space, a suspicious marker is activated; Transmission anomaly detection, compute coefficient of variation of speech packet inter-arrival intervals When > 0.25 and there is a burst retransmission, flag as anomaly.

5. The method for multi-lingual deep synthesized voice adaptation detection in cross-border communication according to claim 1, wherein, The dynamic weight distribution mechanism satisfies: When the LID confidence c < 0.6, then boost to ; When a VPN tunnel is detected, then The weights are set to zero and the audio-text co-analysis mode is activated. For resource-scarce languages, mandatory setting = 0.3, = 0.6, = 0.

1.

6. The method for multi-lingual deep synthesized voice adaptation detection in cross-border communication according to claim 1, wherein, The incremental learning comprises the following steps: Deploying a lightweight detection model on an edge node, and storing new samples through a secure enclave; Using prototype-based few-shot learning: Computing a synthesized speech class center: ; wherein, is the current number of synthesized samples, is a feature extraction function, is the first synthesized speech sample; Optimization objectives: ; wherein, is the Euclidean distance, is the new sample, is the center of the real speech class, is the boundary threshold value; Uploading model gradients through a differential privacy mechanism.

7. A multi-lingual deep synthetic speech adaptive detection system in cross-border communication characterized by, Comprise: A multi-modal acquisition module: supporting RTP / SRTP / WebRTC protocols, including an audio splitter, a metadata parser and a data desensitization unit; An extensible feature engine, comprising: Audio processing chain: contains the voiceprint extraction interface of the open source SpeechBrain framework and the custom phase analysis module; Text processing chain: integrates the Facebook MMS model for low-resource language ASR; Behavior analysis chain: high-performance network traffic probe developed based on DPDK; Adaptive fusion core, including: Weight allocator: real-time receives LID confidence and language resource level signals; Multi-modal attention network: uses a multi-head cross-attention mechanism to calculate feature correlation; Defense enhancement module: contains an adversarial sample generator, a model ensemble voter, and an alarm feedback loop.

8. The multi-lingual deep synthesized voice self-adaptation detection system in cross-border communication according to claim 7, characterized in that, The adaptive fusion core adopts a double-path structure: Main path: inputs multi-modal features into a cascaded GMU unit, and the output formula is: ； ； ； wherein, is the feature transformation result, is a gating signal vector, and is a weight matrix, and is a bias vector, is a hyperbolic tangent activation function, is the final output feature; Auxiliary path: inputs each modal feature into a dedicated detector to generate an auxiliary decision through weighted voting; The final output is: ; In the formula, is the final synthesis probability, is the main model confidence weight, is the main fusion network output probability, is the auxiliary model voting probability; Dynamically adjusted by model confidence.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the multi-language deep synthesis voice adaptive detection method in cross-border communication as claimed in any one of claims 1-6. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the multi-language deep synthesis voice adaptive detection method in cross-border communication as claimed in any one of claims 1-6.

Images