Synthetic speech detection system

Abstract

It is disclosed a synthetic speech detection framework, based on breath-aware temporal discrepancy detection which is able to analyze the interplay between breathing patterns and phonetic production through five functional blocks, thereby significantly enhancing synthetic speech detection accuracy. These functional blocks include the Breathing-Speech Alignment Module (2), Breath-Speech Interaction Embeddings (3), Temporal Consistency Analyzer (4), Breath-Speech Coherence Module (5), and Deepfake Detection Classifier (6). By leveraging the physiological and phonetic cues that are difficult for synthetic speech generation models to replicate, the framework leverages physiological and phonetic cues, offering a robust solution to distinguish between organic and synthetic speech.

Description

TECHINCAL FIELD

[0001] The present disclosure relates to methods and systems for detecting synthetic speech.PRIOR ART

[0002] Synthetic speech, also known as deepfake speech, generated by AI-powered technology, for example through text-to-speech or voice assistant systems, aims to be identical to organic speech produced by a human. While such audio has many well intentioned usage, the potential for dangerous applications has created the need for accurate and automated demarcation of human-spoken from synthetically generated audio.

[0003] The research community has responded with challenges such as ASVspoof, ADD, and SASV. These competitions curate datasets of synthetic and organic speech and invite participants to create detection algorithms to test on these datasets. Subsequently, these datasets are the de facto standard for synthetic speech and give a baseline of comparison for all current and future synthetic speech detectors.

[0004] Most of the currently existing synthetic speech detectors, focus on low-level spectral (e.g., spectrogram, Mel-frequency cepstral coefficients, Linear Frequency Cepstral Coefficients, and constant-Q cepstral coefficients) imperfections, created during the audio generation pipeline and show starkly different classification results vs. human interpreters.

[0005] However, these low-level spectral detection-based methods often fall short in accurately detecting deepfake speech due to the complexities involved in replicating organic speech dynamics, particularly the interaction between breathing patterns and phonetic production, and will be rendered obsolete due to the rapid advancement of the speech generation field.

[0006] In fact, with advances in artificial intelligence and machine learning, the generation of synthetic speech has become increasingly sophisticated, which justifies the need to develop more advanced detection mechanisms that can effectively discern between organic and synthetic speech.

[0007] The present solution intended to innovatively overcome such issues.SUMMARY OF THE DISCLOSURE

[0008] The present disclosure concerns a novel synthetic speech detection framework based on breath-aware temporal discrepancy detection combined with phonetic insights. By focusing on the complex relationship between breathing patterns and phonetic production in natural speech dynamics, the accuracy in the detection of a synthetic speech is enhanced.

[0009] In fact, unlike traditional detection models that emphasize low-level spectral features, the proposed framework leverages the interaction between breath cycles and the phonetic structure of speech, which plays a crucial role in shaping speech rhythm, prosody, and articulation. Consequently, by targeting the physiological and phonetic cues that are difficult for synthetic generation models to replicate, the accuracy in detecting a synthetic speech can be significantly improved.

[0010] The framework consists of five functional blocks: a Breathing-Speech Alignment Module, Breath-Speech Interaction Embeddings, Temporal Consistency Analyzer, Breath-Speech Coherence Module, and Deepfake Detection Classifier. Each functional block plays a crucial role in capturing the intricate details of breath and phonetic interactions, focusing on detecting synthetic speech through temporal analysis, breath-phonetic interaction modeling, and deep feature extraction. By incorporating phonetic insights, these functional blocks are able to capture the intricate details of how breath and phonetic production interact in natural speech, making it possible to identify irregularities in synthetic speech.

[0011] With this context, it is the object of the present disclosure a synthetic speech detection system. In general terms, the system may comprise: A Breathing-Speech Alignment module configured to detect breath events in speech signals using, for example, a combination of a Convolutional Neural Network and a Bidirectional Long Short-Term Memory network, and to align said breath events with natural phonetic units via, in one embodiment, a Transformer-based model; A Breath-Speech Interaction Embeddings module designed to extract detailed multi-dimensional features modeling the interaction between breathing patterns and phonetic production; A Temporal Consistency Analyzer module, which may employ a self-attention mechanism to evaluate the temporal consistency of breath events relative to phonetic production, providing a Phonetic Temporal Consistency Score indicative of natural speech patterns; A Breath-Speech Coherence module that analyzes the coherence between breath events, speech intensity, and phonetic energy patterns using features such as root mean square energy and amplitude envelope, and outputs a Breath-Speech Coherence Score; A Deepfake Detection Classifier module that integrates outputs from the previous modules into a unified feature vector, utilizing, for example, a Multilayer Perceptron or Transformer-based classifier to deliver a binary classification score indicating whether the speech is real or synthetic. DESCRIPTION OF FIGURES

[0012] Figure 1 illustrates a representation of the synthetic speech detection framework of the present disclosure, where the reference signs represent: 1 - input raw speech waveform; 2 - Breathing-Speech Alignment module; a. - Time-stamped breath events aligned with said phonetic and linguistic segments; 3 - Breath-Speech Interaction Embeddings module; b. - Breath-speech embedding record; 4 - Temporal Consistency Analyzer module; c. - Phonetic temporal consistency score; 5 - Breath-Speech Coherence module; d. - Breath-Speech Coherence score; 6 - Synthetic Speech Detection Classifier module; 7 - Classification score. DETAILED DESCRIPTION

[0013] The more general configurations of the present disclosure are described in the Summary of the disclosure. Such configurations are detailed below in accordance with other advantageous and / or preferred embodiments of implementation of the present disclosure.

[0014] The present disclosure is directed towards a sophisticated framework for the detection of synthetic speech utilizing a Breath-Aware Temporal Discrepancy Detection methodology. This framework leverages the complex interplay between breathing patterns and phonetic articulation, which is challenging for synthetic speech generation models to accurately replicate, thereby significantly enhancing the accuracy of the detection.

[0015] This framework is constituted by five functional blocks.

[0016] The first functional block is a Breathing-Speech Alignment (BSA) module (2): this module is responsible for detecting breath events within input speech signals and aligning these events with natural phonetic and linguistic and linguistic segments, ensuring that breath timing matches natural speech structure. In the context of the present disclosure, a phonetic segment is an individual speech sound, that relates to vowels, plosives, and fricatives. As to a linguistic segment, it relates to sentences or clauses and the respective to pauses and breaks. The BSA module (2) may employ a combination of a Convolutional Neural Network (CNN) and a Bidirectional Long Short-Term Memory (BiLSTM) network to accurately detect breath events from raw speech waveforms (1). Following detection, a Transformer-based model may be employed to align these breath events with specific phonetic units including vowels, plosives, and fricatives. Breath is tightly linked to these sounds, as vowels require continuous airflow, while plosives and fricatives depend on precise breath control. The output of this module is a set of time-stamped breath events that are aligned with phonetic and linguistic segments (a.), being crucial for identifying discrepancies in synthetic speech where breath patterns do not naturally follow phonetic articulations.

[0017] The second functional block, is a Breath-Speech Interaction Embeddings (BSIE) module (3): This module is configured to extract detailed, multi-dimensional features that model the interaction between breathing patterns and phonetic production. The feature extraction process may involve several subcomponents: Breath-Event-Aware Acoustic Features unit: to capture acoustic changes around breath events, particularly focusing on the transitions between breath and specific phonetic sounds like vowels and consonants, by track variations in the harmonic-to-noise ratio (HNR) and pitch during these transitions; Temporal Alignment Variability Features unit: to analyze the timing of breaths relative to phonetic unit boundaries (e.g., after plosives or between vowel segments), capturing unnatural pauses or rigid patterns indicative of artificial speech; Speech Intensity-Breath Correlation Features unit: to model how speech intensity naturally varies with phonetic production and breath cycles, focusing on the decay in intensity before breaths and the surge after breaths. This is especially relevant for vowel sounds, which rely on steady airflow, and consonants, which require controlled bursts of air; Breath-Speech Frequency Alignment unit: to capture spectral shifts during breath events and to analyze the influence of breath noise on low-frequency bands, measuring how formant shifts in vowels are affected by breath dynamics, revealing synthetic speech's inability to replicate natural vowel transitions.

[0018] The module (3) focuses on both temporal and acoustic relationships, capturing how breath cycles impact specific phonetic features such as vowel clarity, consonant articulation, and prosodic patterns. The output of this module (3) is a high-dimensional Breath-Speech Interaction Embedding record (b.), encapsulating phonetic-breath dynamics, acoustic features, and temporal variability, providing a detailed representation of natural vs. synthetic speech patterns.

[0019] The third functional block is a Temporal Consistency Analyzer (TCA) module (4): This module evaluates the temporal consistency between detected breath events and phonetic production, ensuring that breath timing follows realistic patterns observed in natural speech. A Transformer-based self-attention network may be employed to model the dependencies between phonetic segments and breath events, focusing on long-range temporal coherence. The module (4) calculates a Phonetic Temporal Consistency (PTC) score (c.) that evaluates how well the timing of breaths aligns with expected patterns for specific phonetic categories (e.g., pauses before vowels, short bursts of breath before plosives). Therefore, the output of this module is the PTC score (c.) that quantifies the degree to which breath events adhere to natural temporal patterns relative to phonetic units. In synthetic speech, misalignments between breath events and phoneme boundaries may lead to an unnatural score.

[0020] The fourth functional block is a Breath-Speech Coherence (BSC) module (5): This module determines if coherence between breath events, speech intensity, and phonetic energy patterns is maintained, which detects whether breath intensity naturally decays or surges in response to phonetic production. By applying features such as root mean square energy and amplitude envelope, the module (5) measures intensity fluctuations, and how phonetic intensity (especially for vowels and consonants) changes before and after breath events. The cross-correlation between breath events and speech intensity may be then computed, focusing on phonetic transitions where breath naturally modulates energy, such as the surge after inhalation when producing vowel sounds or the energy dip before exhalation in fricatives. Therefore, this module (5) outputs a Breath-Speech Coherence (BSC) score (d.) reflecting the natural alignment between breath timing, phonetic intensity, and energy patterns. Synthetic speech may exhibit irregular energy fluctuations around breaths, which this module can detect.

[0021] Finally, the fifth block is a Deepfake Detection Classifier (DDC) module (6): this module integrates the outputs from the previous modules (2, 3, 4, 5) into a final classification decision (7), that determines whether the input speech (1) is natural or synthetic, based on breath-phonetic interaction and temporal patterns. It may involve feature fusion, creating a unified feature vector that captures both temporal and phonetic dependencies. A Multilayer Perceptron or Transformer-based classifier may be employed to analyze and integrate these features, thereby being able to classify speech (1) samples as natural or synthetic, as it is trained to distinguish the subtle differences in breath-phonetic relationships that are difficult for synthetic speech generation models to replicate. Consequently, a binary classification score (7) is generated indicating whether the input speech is natural or a synthetic, based on phonetic and breath-speech interaction features.EMBODIMENTS

[0022] In a preferred embodiment of the system, it is comprised by a Breathing-Speech Alignment (BSA) module (2), a Breath-Speech Interaction Embeddings (BSIE) module (3), a Temporal Consistency Analyzer (TCA) module (4), a Breath-Speech Coherence (BSC) module (5) and a Synthetic Speech Detection Classifier (SSDC) module (6). The modules (2, 3, 4, 5, 6) are provided with processing means and are operatively coupled to each other.

[0023] More particularly, the BSA module (2) is configured to process an input raw speech waveform (1) to detect breath events and to align said breath events with a natural speech structure, defined by phonetic and linguistic segments, thereby generating a set of time-stamped breath events aligned with said phonetic and linguistic segments (a.); the BSIE module (3) is configured to process the set of time-stamped breath events in order to extract multi-dimensional features modeling said time-stamped breath events and phonetic production, thereby generating a breath-speech embedding record (b.), encapsulating phonetic-breath dynamics, acoustic features and temporal variability; the TCA module (4) is configured to evaluate temporal consistency between detected breath events and phonetic segments production, thereby generating a phonetic temporal consistency (PTC) score (c.); the BSC module (5) is configured to measure speech intensity fluctuations and phonetic energy pattern parameters and to cross-correlate said parameters with breath events, thereby generating a Breath-Speech Coherence (BSC) score (d.); the SSDC module (6) is configured to fuse the outputs from the previous modules and to generate a classification score (7) indicating whether the input raw speech waveform (1) is natural or synthetic.

[0024] In one embodiment of the system, the BSA module (2) includes a Breath-detection unit; said unit may be comprised by: a Convolutional Neural Network and a Bidirectional Long Short-Term Memory Network, that are operatively coupled to each other in order to process the raw speech waveform (1) to detect breath events.

[0025] In another embodiment of the system, the BSA module (2) includes an Alignment unit; said unit may be comprised by a transformer-based architecture configured to align the detected breath events with phonetic segments, such as vowels, plosives and fricatives, and with linguistic segments, such as linguistic boundaries defined by pauses and sentence breaks.

[0026] In another embodiment of the system, the BSIE module (3) comprises: a breath-event-aware acoustic features unit, configured to capture acoustic changes around breath events; a temporal alignment variability features unit, configured to analyze time-stamped breath events relative to phonetic segment boundaries, such as after plosives or between vowel segments, to detect unnatural pauses or rigid breath patterns; a speech intensity-breath correlation features unit, configured to model the variability of speech intensity with phonetic production and breath cycles defined by the time-stamped breath events; a breath-speech frequency alignment unit, configured to capture spectral shifts during breath events.

[0027] More particularly, the breath-event-aware acoustic features unit captures acoustic changes around breath events by being configured to: detect transactions between breath events and phonetic segments such as vowels and consonants; and to track variations in the harmonic-to-noise ration and pitch during these transactions.

[0028] Additionally, the breath-speech frequency alignment unit captures spectral shifts during breath events by being configured to measure a relation between formant shifts F1, F2, in vowels, and breath dynamics.

[0029] In another embodiment of the system, the TCA module (4) is operable to employ a transformer-based self-attention network adapted to model dependencies between phonetic segments and detected breath events, in order to generate the PCT score (c.).

[0030] In another embodiment of the system, the BSC module (5) is configured to determine speech intensity and phonetic energy pattern parameters by applying root mean square energy and amplitude envelope features.

[0031] More particularly, the cross-correlation between breath events and speech intensity is computed on phonetic transitions.

[0032] In another embodiment of the system, the SSDC module (6) is configured to fuse: the set of time-stamped breath events aligned with said phonetic and linguistic segments (a.), from the BSA module (2), the breath-speech embedding record (b.), from the BSIE module (3), the PTC score (c.), from the TCA module (4), and the BSC score (d.), from the BSC module (5), into a unified feature vector, adapted to capture both temporal and phonetic dependencies.

[0033] More particularly, the SSDC module (6) is operable to employ a classifier configured to processed the unified feature vector and classify input raw speech waveform (1) samples, thereby generating the classification score (7).

[0034] Even more particularly, the classifier may be a Multilayer Perceptron or a Transformer-based classifier.

[0035] Of course, the preferred embodiments shown above are combinable, in the different possible forms, being herein avoided the repetition all such combinations.

Claims

1. A synthetic speech detection system comprising a Breathing-Speech Alignment (BSA) module (2), a Breath-Speech Interaction Embeddings (BSIE) module (3), a Temporal Consistency Analyzer (TCA) module (4), a Breath-Speech Coherence (BSC) module (5) and a Synthetic Speech Detection Classifier (SSDC) module (6); the modules comprising processing means and being operatively coupled to each other; the BSA module (2) is configured to process an input raw speech waveform (1) to detect breath events and to align said breath events with a natural speech structure, defined by phonetic and linguistic segments, thereby generating a set of time-stamped breath events aligned with said phonetic and linguistic segments (a.); the BSIE module (3) is configured to process the set of time-stamped breath events in order to extract multi-dimensional features modeling said time-stamped breath events and phonetic production, thereby generating a breath-speech embedding record (b.), encapsulating phonetic-breath dynamics, acoustic features and temporal variability; the TCA module (4) is configured to evaluate temporal consistency between detected breath events and phonetic segments production, thereby generating a phonetic temporal consistency (PTC) score (c.); the BSC module (5) is configured to measure speech intensity fluctuations and phonetic energy pattern parameters and to cross-correlate said parameters with breath events, thereby generating a Breath-Speech Coherence (BSC) score (d.); the SSDC module (6) is configured to fuse the outputs from the previous modules and to generate a classification score (7) indicating whether the input raw speech waveform (1) is natural or synthetic.

2. The system according to claim 1, wherein the BSA module (2) includes a Breath-detection unit; said unit being comprised by: - a Convolutional Neural Network and - a Bidirectional Long Short-Term Memory Network, that are operatively coupled to each other in order to process the raw speech waveform (1) to detect breath events.

3. The system according to claims 1 or 2, wherein the BSA module (2) includes an Alignment unit; said unit being comprised by a transformer-based architecture configured to align the detected breath events with phonetic segments, such as vowels, plosives and fricatives, and with linguistic segments, such as linguistic boundaries defined by pauses and sentence breaks.

4. The system according to any of the previous claims wherein the BSIE module (3) comprises: - a breath-event-aware acoustic features unit, configured to capture acoustic changes around breath events; - a temporal alignment variability features unit, configured to analyze time-stamped breath events relative to phonetic segment boundaries, such as after plosives or between vowel segments, to detect unnatural pauses or rigid breath patterns; - a speech intensity-breath correlation features unit, configured to model the variability of speech intensity with phonetic production and breath cycles defined by the time-stamped breath events; - a breath-speech frequency alignment unit, configured to capture spectral shifts during breath events.

5. The system according to claim 4, wherein the breath-event-aware acoustic features unit captures acoustic changes around breath events by being configured to: - detect transactions between breath events and phonetic segments such as vowels and consonants; and to - track variations in the harmonic-to-noise ration and pitch during these transactions.

6. The system according to claim 4 or 5, wherein the breath-speech frequency alignment unit captures spectral shifts during breath events by being configured to measure a relation between formant shifts F1, F2, in vowels, and breath dynamics.

7. The system according to any of the previous claims, wherein the TCA module (4) is operable to employ a transformer-based self-attention network adapted to model dependencies between phonetic segments and detected breath events, in order to generate the PCT score (c.).

8. The system according to any of the previous claims, wherein the BSC module (5) is configured to determine speech intensity and phonetic energy pattern parameters by applying root mean square energy and amplitude envelope features.

9. The system according to claim 8, wherein the cross-correlation between breath events and speech intensity is computed on phonetic transitions.

10. The system according to any of the previous claims, wherein the SSDC module (6) is configured to fuse: - the set of time-stamped breath events aligned with said phonetic and linguistic segments (a.), from the BSA module (2), - the breath-speech embedding record (b.), from the BSIE module (3), - the PTC score (c.), from the TCA module (4), and - the BSC score (d.), from the BSC module (5), into a unified feature vector, adapted to capture both temporal and phonetic dependencies.

11. The system according to claim 10 wherein the SSDC module (6) is operable to employ a classifier configured to processed the unified feature vector and classify input raw speech waveform (1) samples, thereby generating the classification score (7).

12. The system according to claim 11, wherein the classifier is a Multilayer Perceptron or a Transformer-based classifier.

Images