Apparatus and method for detecting synthetic sound

Abstract

A synthesized speech detection device and method are disclosed for detecting synthesized speech by utilizing non-vowel segments that are difficult to naturally reproduce in a speech synthesizer. According to one aspect, the synthesized speech detection device comprises: a time-frequency converter that converts an input speech signal into a frequency domain signal; a non-vowel extraction unit that extracts a non-vowel segment from the frequency domain signal; a synthesized speech-related feature extraction unit that extracts a first speech feature related to synthesized speech from the data of the non-vowel segment and extracts a second speech feature related to synthesized speech from the frequency domain signal; and a classification unit that outputs a detection result regarding whether the input speech signal is synthesized speech based on the first speech feature and the second speech feature.

Description

Technology Field

[0001] The present invention relates to a synthesized sound detection device and method, and more specifically, to a synthesized sound detection device and method capable of detecting whether a voice signal is a real voice or a synthesized sound. Background Technology

[0002] Recently, Text-to-Speech (TTS) technology, which synthesizes text to sound like real human speech, has made significant advancements. In the past, voices synthesized using TTS technology could not perfectly mimic human pronunciation elements, often sounding like a robot speaking and falling far short of real speech. However, with the recent advancement of artificial intelligence (AI) technology, AI-synthesized voices are now very similar to human speech without any awkward elements, leading to their utilization in various fields such as chatbots and counseling services. Nevertheless, along with this technological progress, concerns are emerging that the technology could be exploited for crimes such as voice phishing. Despite these concerns, TTS technology is being actively researched, whereas technology to detect whether a voice is a synthesized sound created using an AI-based speech synthesizer or a real voice is being hardly studied. The problem to be solved

[0003] The purpose of the present invention is to provide a synthesized sound detection device and method that detects synthesized sounds using non-vowel segments that are difficult to naturally reproduce in a speech synthesizer.

[0004] Considering that the linguistic information of synthesized speech is of lower quality compared to actual speech, the present invention has another objective of providing a synthesized speech detection device and method that detects synthesized speech by additionally utilizing the linguistic information of a speech signal. means of solving the problem

[0005] According to one aspect, a synthesized voice detection device comprises: a time-frequency converter that converts an input voice signal into a frequency domain signal; a non-vowel extraction unit that extracts a non-vowel segment from the frequency domain signal; a synthesized voice-related feature extraction unit that extracts a first voice feature related to the synthesized voice from the data of the non-vowel segment and extracts a second voice feature related to the synthesized voice from the frequency domain signal; and a classification unit that outputs a detection result regarding whether the input voice signal is a synthesized voice based on the first voice feature and the second voice feature.

[0006] The above non-vowel extraction unit can extract non-vowel segments by removing vowel segments using phoneme data, or by removing vowel segments using pitch data, or by determining a segment having power greater than or equal to a threshold power on a frame-by-frame basis as a vowel segment and removing the corresponding segment.

[0007] The above-described synthetic sound detection device further includes a speech recognition unit that recognizes the input speech signal and converts it into text data; and a text encoder that extracts linguistic information from the text data; and the classification unit can output a detection result regarding whether the input speech signal is a synthetic sound based on the first and second speech features and the linguistic information.

[0008] The above-mentioned synthesized sound-related feature extraction unit can extract the first speech feature by randomly mixing and augmenting the data of the non-vowel section and extracting the first speech feature from the augmented data.

[0009] According to another aspect, a method for detecting synthesized speech using a synthesized speech detection device comprises: converting an input speech signal into a frequency domain signal; extracting a non-vowel segment from the frequency domain signal; extracting a first speech feature related to synthesized speech from the data of the non-vowel segment; extracting a second speech feature related to synthesized speech from the frequency domain signal; and outputting a detection result regarding whether the input speech signal is synthesized speech based on the first speech feature and the second speech feature.

[0010] The step of extracting the above non-vowel segments can be performed by extracting non-vowel segments by removing vowel segments using phoneme data, or by extracting non-vowel segments by removing vowel segments using pitch data, or by determining a segment having power greater than or equal to a threshold power on a frame-by-frame basis as a vowel segment and removing the corresponding segment.

[0011] The above-described synthetic sound detection method further includes the step of recognizing the input voice signal and converting it into text data; and the step of extracting linguistic information from the text data; and the step of outputting the detection result may output a detection result regarding whether the input voice signal is a synthetic sound based on the first and second voice features and the linguistic information.

[0012] The step of extracting the first voice feature can randomly mix and augment the data of the non-vowel interval and extract the first voice feature from the augmented data. Effects of the invention

[0013] According to the present invention, synthesized speech can be detected more precisely than in the past by extracting and utilizing non-vowel segments that are difficult to naturally reproduce in a speech synthesizer.

[0014] Speech synthesized by a speech synthesizer tends to have a lower Word Error Rate (WER) compared to actual speech data. In other words, when synthesized speech is converted into linguistic information, the completeness of the sentences is lower than that of actual speech. Taking this into account, the present invention enables more precise detection of synthesized speech by additionally utilizing linguistic information for synthesized speech detection. Brief explanation of the drawing

[0015] FIG. 1 is a diagram showing the configuration of a synthetic sound detection device according to one embodiment of the present invention. Figure 2(a) is a Mel Spectrogram obtained by converting a voice signal into a frequency domain signal according to one embodiment of the present invention. Figure 2(b) is the result of analyzing the pitch using Crepe on the Mel spectrogram of Figure 2(a). FIG. 3 is an example of a mel spectrogram of voice according to one embodiment of the present invention. FIG. 4 is a flowchart illustrating a synthetic sound detection method according to one embodiment of the present invention. Specific details for implementing the invention

[0016] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols will be assigned the same reference number, and redundant descriptions thereof will be omitted. The suffix "bu" used for components in the following description is assigned or used interchangeably solely for the ease of drafting the specification and does not have a distinct meaning or role in itself. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0017] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0018] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0019] For convenience of explanation in implementing the present invention, the components may be described in detail; however, these components may be implemented within a single device or module, or a single component may be divided and implemented across multiple devices or modules.

[0020] FIG. 1 is a diagram showing the configuration of a synthetic sound detection device according to an embodiment of the present invention. The synthetic sound detection device (100) according to the present embodiment may include a memory, one or more processors (CPUs), peripheral interfaces, an input / output (I / O) subsystem, a display device, an input device, and a communication circuit.

[0021] Memory can store various information and program instructions, and programs are executed by the processor. Peripheral interfaces connect input / output peripheral devices to the processor and memory. One or more processors perform various functions for the system and process data by executing various software programs and / or instruction sets stored in memory.

[0022] The I / O subsystem provides an interface between input / output peripherals, such as display devices and input devices, and peripheral interfaces. Communication circuits perform communication through external ports or via RF signals. Communication circuits convert electrical signals into RF signals or vice versa, and through these RF signals, they can communicate with communication networks, other mobile gateway devices, and communication devices.

[0023] As illustrated in FIG. 1, the synthetic sound detection device (100) according to the present embodiment includes a time-frequency converter (110), a non-vowel extraction unit (120), a synthetic sound-related feature extraction unit (130), a speech recognition unit (140), a text encoder (150), and a classification unit (160). These components may be implemented as a program, stored in memory, and executed by a processor, or may be implemented as a combination of software and hardware to perform operations.

[0024] The time-frequency conversion unit (110) converts the input time-domain voice signal into a frequency-domain signal and outputs it. Preferably, the time-frequency conversion unit (110) uses a Short Time Fourier Transform to convert the time-domain voice signal into a frequency-domain signal and outputs it. Here, the frequency-domain signal may be, for example, a Mel Spectrogram in which the frequency is converted to a Mel Scale.

[0025] The non-vowel extraction unit (120) receives a frequency domain signal output from the time-frequency conversion unit (110) and extracts non-vowel segments from the received frequency domain signal. Typically, a speech synthesizer generates speech by utilizing speaker characteristic information and pitch information of the utterance. Since vowels have long and stable frequency patterns, it is relatively easy for a speech synthesizer to predict and generate them. On the other hand, consonants require particularly precise temporal resolution and fine sound differences depending on the place of articulation, making it difficult to naturally reproduce the characteristics of consonants. For example, consonants such as ' / s / ', ' / t / ', and ' / k / ' have short pronunciations and their characteristic sounds change rapidly, so sophisticated modeling is required to synthesize them naturally. In addition, when some speech synthesizers attempt to synthesize speech identical to the input utterance data, there may be some noise compared to the original data during the silent segments of the utterance. Therefore, the present invention extracts and utilizes non-vowel segments that are difficult to naturally reproduce in a speech synthesizer.

[0026] In one embodiment, the non-vowel extraction unit (120) can extract non-vowel segments by using a tool such as the Montreal Force Aligner (MFA) to find segments where vowels exist and removing them. By using a properly trained MFA, it is possible to determine which phonemes were uttered at which time, and by utilizing this phoneme data, segments where vowels exist in speech can be found and removed.

[0027] In another embodiment, the non-vowel extraction unit (120) can extract non-vowel segments by analyzing pitch using tools such as Crepe and Praat, and thereby finding and removing segments where vowels exist. Crepe and Praat are tools capable of analyzing the pitch of speech. These tools can obtain pitch data and confidence for specific segments within the speech. Since segments where vowels are uttered have high accuracy regarding pitch, segments where vowels are uttered can be found and removed by appropriately utilizing that accuracy data and pitch data between adjacent frames. FIG. 2(a) is a Mel Spectrogram converted from a speech signal into a frequency domain signal, and FIG. 2(b) is the result of analyzing the pitch of the Mel Spectrogram using Crepe. In FIG. 2(b), the red area is the utterance segment identified as a vowel and to be removed.

[0028] In another embodiment, the non-vowel extraction unit (120) can extract non-vowel sections by determining, on a frame-by-frame basis, a section in the frequency domain signal where the power is above a certain level as a section where a vowel exists and removing the corresponding section. Vowels have higher energy than consonants. Considering this, the non-vowel extraction unit (120) determines, based on a threshold power that can distinguish between consonants and vowels, a section in the frequency domain signal where the power is above the threshold power as a section where a vowel exists. FIG. 3 is an example of a mel spectrogram of speech according to an embodiment of the present invention, where the red area is a section where the power is above the threshold power and is determined to be a vowel and removed.

[0029] The synthesized speech-related feature extraction unit (130) receives non-vowel segment data output from the non-vowel extraction unit (120), extracts and outputs a first speech feature related to the synthesized speech from the received non-vowel segment data, and also receives a frequency domain signal output from the time-frequency conversion unit (110) and extracts and outputs a second speech feature related to the synthesized speech from the received frequency domain signal. Preferably, the synthesized speech-related feature extraction unit (130) may be an artificial intelligence model trained using training data consisting of synthesized speech, real speech, and labels for them.

[0030] Although the synthesized speech-related feature extraction unit (130) may be considered to extract only the first speech feature from the non-vowel segment data, there is a risk of information loss due to the removal of the vowel segment. Therefore, the synthesized speech-related feature extraction unit (130) extracts the first speech feature from the non-vowel segment data and also extracts the second speech feature together from the frequency domain signal for the entire length of the speech signal.

[0031] The synthesized sound-related feature extraction unit (130) can extract and output a first speech feature from non-vowel segment data by randomly mixing and augmenting the non-vowel segment data and extracting the first speech feature from the augmented data. The order of the words constituting the non-vowel segment data is not significant. Therefore, by randomly mixing and augmenting the data, robustness in extracting the first speech feature can be improved.

[0032] The speech recognition unit (140) recognizes the input speech signal in the time domain and converts it into text data. The speech recognition unit (140) may use a general STT (Speech To Text) engine, but is not limited thereto. The text encoder (150) receives the text data converted and output by the speech recognition unit (140) and extracts and outputs linguistic information from the received text data. Linguistic information includes, for example, pronunciation, word order, grammar, semantics, etc. The text encoder (150) is trained in advance to extract speaker-independent linguistic information.

[0033] Non-verbal information, such as tone and intensity of the voice, gender, age, and emotion, is removed from the voice signal by the voice recognition unit (140) and the text encoder (150), and linguistic information, such as word order and stress, can be extracted. Linguistic information in the voice signal serves to determine whether there are parts where the sentence structure of the input utterance is awkward compared to the sentence structure of a general utterance. In addition, voice synthesized by a voice synthesizer tends to have a lower Word Error Rate (WER) compared to actual speech data. That is, when the synthesized voice is converted into linguistic information, the completeness of the sentence is lower compared to the actual voice. Taking these points into account, the present invention utilizes linguistic information for the detection of synthesized voice.

[0034] The classification unit (160) uses the first speech feature and the second speech feature output from the synthesis-related feature extraction unit (130) and the linguistic information output from the text encoder (150) to output a synthesis detection result regarding whether the input speech signal is a synthesis (Fake) or a real speech (Real). The classification unit (160) is a binary classification model that determines whether the input speech signal is a synthesis or a real speech, and the loss function used for learning may be, for example, Cross Entropy. Specific examples of loss functions are as follows.

[0035]

[0036] Here, N is the total number of training data, and is the label of the training data, is the softmax output value of the logit output through the classification unit (160).

[0037] FIG. 4 is a flowchart illustrating a synthetic sound detection method according to one embodiment of the present invention.

[0038] Referring to FIG. 4, in step S501, the synthesized voice detection device (100) receives a voice signal in the time domain (e.g., waveform) and converts the voice signal into a frequency domain signal. Preferably, the synthesized voice detection device (100) converts the voice signal in the time domain into a frequency domain signal using a Short Time Fourier Transform and outputs it. Here, the frequency domain signal may be, for example, a Mel Spectrogram in which the frequency is converted to a Mel Scale.

[0039] In step S502, the synthesized sound detection device (100) extracts non-vowel segments from the frequency domain signal. In one embodiment, the synthesized sound detection device (100) can extract non-vowel segments by using a tool such as MFA (Montreal Force Aligner) to locate and remove segments where vowels exist. In another embodiment, the synthesized sound detection device (100) can extract non-vowel segments by using tools such as Crepe and Praat to analyze pitch and thereby locate and remove segments where vowels exist. In yet another embodiment, the synthesized sound detection device (100) can extract non-vowel segments by determining that segments where the power is above a certain level on a frame-by-frame basis in the frequency domain signal are segments where vowels exist and removing those segments. Vowels have higher energy than consonants. Considering these points, the synthesized sound detection device (100) determines that a section having power greater than or equal to the threshold power in a frequency domain signal on a frame-by-frame basis, based on a threshold power capable of distinguishing between consonants and vowels, is a section where a vowel exists.

[0040] In step S503, the synthesized voice detection device (100) extracts a first voice feature related to the synthesized voice from non-vowel segment data, and in step S504, extracts a second voice feature related to the synthesized voice from the frequency domain signal converted in step S501. Preferably, the synthesized voice detection device (100) can extract the first and second voice features using an artificial intelligence model trained using training data consisting of synthesized voices, real voices, and labels for them. In extracting the first voice feature from the non-vowel segment data, the synthesized voice detection device (100) can randomly mix and augment the non-vowel segment data and extract the first voice feature from the augmented data.

[0041] In step S505, the synthesized speech detection device (100) recognizes the input time-domain speech signal and converts it into text data. The synthesized speech detection device (100) may use a general STT (Speech To Text) engine, but is not limited thereto.

[0042] In step S506, the synthesized speech detection device (100) extracts and outputs linguistic information from the converted text data. The linguistic information is, for example, pronunciation, word order, grammar, semantics, etc. Linguistic information can be extracted through a text encoder (150) that has been pre-trained to extract speaker-independent linguistic information.

[0043] In step S507, the synthesized voice detection device (100) inputs the first voice feature related to the synthesized voice extracted in step S503, the second voice feature related to the synthesized voice extracted in step S504, and the linguistic information extracted in step S506 into the classification unit (160) to output a synthesized voice detection result regarding whether the voice signal is a synthesized voice (Fake) or a real voice (Real). Preferably, the classification unit (160) is a binary classification model that determines whether the input voice signal is a synthesized voice or a real voice, and the loss function used for learning may be, for example, Cross Entropy. Specific examples of loss functions are as described above.

[0044] According to the embodiment described above, it is possible to precisely detect whether an input speech signal is synthesized speech or real speech. As previously explained, speech synthesizers typically generate speech by utilizing speaker characteristic information and pitch information of the utterance. Since vowels have long and stable frequency patterns, it is relatively easy for a speech synthesizer to predict and generate them. On the other hand, consonants require particularly precise temporal resolution and subtle sound differences depending on the place of articulation, making it difficult to naturally reproduce the characteristics of consonants. Furthermore, when some speech synthesizers attempt to synthesize speech identical to the input utterance data, noise may be introduced during the silent segments of the utterance compared to the original data. Therefore, the present invention enables the precise detection of synthesized speech by extracting and utilizing non-vowel segments that are difficult to naturally reproduce by a speech synthesizer. Additionally, speech synthesized by a speech synthesizer tends to have a lower Word Error Rate (WER) compared to actual utterance data. In other words, when synthesized speech is converted into linguistic information, the completeness of the sentence is lower compared to actual speech. Taking these points into consideration, the present invention can detect synthesized sounds more precisely by additionally utilizing linguistic information for synthesized sound detection.

[0045] In the above embodiments, the presence of a synthesized sound was determined using both linguistic information and non-vowel interval data, but this is not limited thereto; the presence of a synthesized sound can be determined using only non-vowel interval data, and in other embodiments, the presence of a synthesized sound can be determined by further considering linguistic information.

[0046] While this specification contains many features, such features should not be interpreted as limiting the scope of the invention or the claims. Additionally, features described in individual embodiments herein may be combined and implemented in a single embodiment. Conversely, various features described in a single embodiment herein may be individually implemented in various embodiments or appropriately combined and implemented.

[0047] Although operations are described in a specific order in the drawings, it should not be understood that such operations are performed in the specific order as illustrated, or in a series of consecutive sequences, or that all described operations are performed to obtain a desired result. Multitasking and parallel processing may be advantageous in certain environments. Furthermore, it should be understood that the distinction between various system components in the above-described embodiments does not require such distinction in all embodiments. The above-described program components and systems may generally be implemented as a package in a single software product or multiple software products.

[0048] The method of the present invention as described above can be implemented as a program and stored in a computer-readable form on a recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.). Since this process can be easily carried out by a person skilled in the art to which the present invention belongs, it will not be described in further detail.

[0049] The present invention described above is not limited by the aforementioned embodiments and attached drawings, as various substitutions, modifications, and changes are possible within the scope of the technical concept of the present invention for those skilled in the art to which the present invention belongs. Explanation of the symbols

[0050] 110 : Time-frequency converter 120 : Non-vowel extraction part 130 : Synthesized sound related feature extraction unit 140 : Voice recognition unit 150 : Text encoder 160 : Classification section

Claims

Claim 1 A synthesized voice detection device comprising: a time-frequency converter that converts an input voice signal into a frequency domain signal; a non-vowel extraction unit that extracts a non-vowel segment from the frequency domain signal; a synthesized voice-related feature extraction unit that extracts a first voice feature related to the synthesized voice from the data of the non-vowel segment and extracts a second voice feature related to the synthesized voice from the frequency domain signal; and a classification unit that outputs a detection result regarding whether the input voice signal is a synthesized voice based on the first voice feature and the second voice feature. Claim 2 In claim 1, the non-vowel extraction unit is a synthetic sound detection device that extracts non-vowel segments by removing vowel segments using phoneme data. Claim 3 In claim 1, the non-vowel extraction unit is a synthetic sound detection device that extracts non-vowel sections by removing vowel sections using pitch data. Claim 4 In claim 1, the non-vowel extraction unit is a synthetic sound detection device that extracts non-vowel sections by determining a section having power greater than or equal to a threshold power on a frame-by-frame basis as a vowel section and removing the corresponding section. Claim 5 A synthesized sound detection device according to claim 1, further comprising: a speech recognition unit that recognizes the input speech signal and converts it into text data; and a text encoder that extracts linguistic information from the text data, wherein the classification unit outputs a detection result regarding whether the input speech signal is a synthesized sound based on the first and second speech features and the linguistic information. Claim 6 In claim 1, the synthetic sound-related feature extraction unit is a synthetic sound detection device that, in extracting the first voice feature, randomly mixes and augments the data of the non-vowel segment and extracts the first voice feature from the augmented data. Claim 7 A method for detecting synthesized sound using a synthesized sound detection device, comprising: a step of converting an input voice signal into a frequency domain signal; a step of extracting a non-vowel segment from the frequency domain signal; a step of extracting a first voice feature related to synthesized sound from the data of the non-vowel segment; a step of extracting a second voice feature related to synthesized sound from the frequency domain signal; and a step of outputting a detection result regarding whether the input voice signal is synthesized sound based on the first voice feature and the second voice feature. Claim 8 In claim 7, the step of extracting the non-vowel segment is a synthetic sound detection method that extracts the non-vowel segment by removing the vowel segment using phoneme data. Claim 9 In claim 7, the step of extracting the non-vowel segment is a synthetic sound detection method that extracts the non-vowel segment by removing the vowel segment using pitch data. Claim 10 In claim 7, the step of extracting non-vowel sections is a synthetic sound detection method that extracts non-vowel sections by determining a section having power greater than or equal to a threshold power on a frame-by-frame basis as a vowel section and removing the corresponding section. Claim 11 A synthetic sound detection method according to claim 7, further comprising: a step of recognizing the input voice signal and converting it into text data; and a step of extracting linguistic information from the text data; wherein the step of outputting the detection result outputs a detection result regarding whether the input voice signal is a synthetic sound based on the first and second voice features and the linguistic information. Claim 12 In claim 7, the step of extracting the first voice feature is a synthetic voice detection method that randomly mixes and augments the data of the non-vowel segment and extracts the first voice feature from the augmented data. Claim 13 A computer program that executes a method according to any one of paragraphs 7 through 12 through a computer system, and is stored on a computer-readable recording medium.

Images