System and method for verifying human voice

Abstract

Exemplary systems and methods include measuring physiological signals emitted from the human body to distinguish humans from automated bots on the internet, as well as integrating biosensors to capture physiological and acoustic signals that indicate natural human speech.

Description

TECHNICAL FIELD

[0001] Cross-Reference to Related Applications This application is a non-provisional patent application claiming the benefit of U.S. Provisional Patent Application No. 63 / 541,773 filed on September 29, 2023, U.S. Provisional Patent Application No. 63 / 541,778 filed on September 29, 2023, and U.S. Provisional Patent Application No. 63 / 645,081 filed on May 9, 2024, all of which are hereby incorporated herein by reference in their entirety.

[0002] The present disclosure relates generally to techniques for distinguishing human users from automated systems or bots, and in particular to a microphone-based system that can prove on a recording device that a recorded acoustic signal originates from a human speaking towards the microphone. BACKGROUND ART

[0003] With the rise of AI and deepfake technologies, we find ourselves in an era where it is increasingly difficult to distinguish real media from synthetic media. 2023 marked a turning point for deepfakes. Consider the following three incidents.

[0004] In June 2023, a deepfake of Russian President Vladimir Putin was broadcast on Russian television and radio. In this video, "Putin" declared martial law and stated that Ukrainian forces had invaded Russia. Authorities later stated that the deepfake was broadcast on national airwaves after the distribution of several broadcast channels was hacked. This incident highlights the potential for deepfakes to spread disinformation, impersonate global leaders, and incite panic or conflict.

[0005] In April 2023, the song "Heart on My Sleeve," which used AI to mimic the music of pop stars Drake and The Weeknd, was released and became a hot topic. The song was so compelling that some even thought it surpassed the talent of the real pop stars. However, the popularity and potential for revenue generation of AI-generated songs has put the music industry's gatekeepers on edge. Universal Music Group, the owner of Drake and The Weeknd's labels, immediately filed a copyright infringement claim to have "Heart on My Sleeve" removed from the platform. Since then, there have been several other examples of AI-processed and manipulated songs. The legal implications of music and copyright law continue to dominate the debate surrounding AI covers in the music industry.

[0006] In April 2023, a mother in Arizona received a fraudulent phone call using AI voice cloning technology to imitate her daughter's voice. The caller claimed to have kidnapped her daughter and demanded a ransom. The mother initially recognized the voice as her daughter's. This incident highlights the dangers of AI deepfake and voice cloning technologies, which can be used to deceive people and commit fraud.

[0007] Deepfakes pose significant social challenges, including the potential to spread misinformation, cause panic or conflict, and undermine media credibility. Deepfakes can impersonate world leaders and potentially lead to geopolitical instability, as seen in the Putin case. In the music industry, AI-generated music could infringe copyright laws, disrupt revenue streams, and challenge traditional administrators. Furthermore, voice cloning technology can be used for malicious purposes such as fraud and deception, as demonstrated by the Arizona phone scam case.

[0008] Unfortunately, these disturbing incidents expose a significant gap in the digital landscape. We lack a reliable way to guarantee the authenticity of human-sounding voices and verify that there was actually a human being behind the microphone, moving their lips to produce the recorded audio. Furthermore, there is an urgent need to make this information accessible to everyone. Currently, whether human-generated, AI-generated, or manipulated / processed media, once the source file is stored and uploaded to a distribution center (e.g., YouTube®, social media, television broadcasting), it is detached from its creation process. In other words, with regard to audio, we don't know whether there was a human voice speaking at the other end of the recording device, or whether the audio was partially or completely generated via AI.

[0009] Similarly, in both packet-switched and circuit-switched communication channels, a voice packet is disconnected from the caller the moment it leaves the caller's device. There is no way to know whether there was a human speaker on the phone or whether the voice was synthesized.

[0010] The pressing problem that this invention aims to solve is how to maintain the authenticity of "human-made" media in this age of advanced deepfake technology.

[0011] A related but different problem, not necessarily unique to media production, is distinguishing between human users and automated bots. CAPTCHA, an acronym for "Fully Automated Public Turing Test to Distinguish Between Computers and Humans," is a technique used to distinguish between human users and automated systems or bots. CAPTCHA was devised to mitigate automated form submissions, unauthorized access, web scraping, spamming, and similar types of malicious activity carried out by computer programs on the web. The technique generally incorporates a challenge-response test in which the user must perform tasks that are, at least conventionally, difficult for a machine to perform. These tasks include identifying distorted text or images, performing simple mathematical calculations, and identifying specific objects within a visual scene or a series of images.

[0012] However, the reliability and effectiveness of existing CAPTCHA methodologies are seriously threatened by recent advances in artificial intelligence (AI) and machine learning (ML). Traditional CAPTCHAs, which often rely on visual-based tasks, are being bypassed by modern AI algorithms that have proven to be proficient in image and text recognition. Optical character recognition (OCR) technology has made remarkable progress, enabling machines to read distorted text, a capability once exclusive to human cognition. Similarly, convolutional neural networks (CNNs), a type of deep learning neural network, have demonstrated exceptional performance in visual recognition tasks, thereby diminishing the effectiveness of image-based CAPTCHAs.

[0013] Furthermore, AI models are learning to more closely mimic human behavior, enhancing their ability to solve puzzles and tasks originally designed for human intelligence. Advances in AI in natural language processing, predictive modeling, and cognitive computing further diminish the effectiveness of traditional CAPTCHA systems. These developments indicate that traditional CAPTCHA systems are becoming increasingly outdated and inadequate to fulfill their primary purpose of distinguishing human users from AI systems. Therefore, it is essential to develop and implement new methodologies that are more resilient to rapidly evolving AI technologies.

[0014] In particular, the various aspects of this disclosure that have been conceived and developed have been based on these observations. [Overview of the Initiative]

[0015] This disclosure provides several examples illustrating methods and systems associated with human voice verification / authentication. In the context of the disclosed methods, devices, techniques, apparatus, systems, etc., the terms “operable to,” “configured to,” and “can be” as used herein are interchangeable.

[0016] In a first set of exemplary examples, the concept of the present invention includes a method for authenticating human speech, comprising the steps of: capturing data associated with a speech event from a plurality of sound sources, wherein the data includes a combination of signals relating to both physiological and acoustic characteristics of the speech event; extracting features from the captured signals representing raw human-generated speech; analyzing the features extracted from the acoustic and physiological signals to authenticate human-generated speech; and verifying the combination of signals relative to each other to further confirm the authenticity of the speech event.

[0017] In some cases, the above method can be translated into machine-readable instructions that can be executed by at least one processor.

[0018] In a second example, the concept of the present invention includes a method for human verification via physiological signals emitted from the human body, comprising: accessing physiological data from a human user passively collected via a radar sensor; processing the physiological data to generate one or more physiological signals from the physiological data; extracting a plurality of features from one or more signals, including spectral features and temporal features, wherein the spectral features define the frequency components of one or more physiological signals and the temporal features represent changes in one or more physiological signals over time; and evaluating the plurality of features by a model to determine whether the plurality of features match a pattern of a human user, wherein the model is pre-trained on a human database having the same features.

[0019] In this example, the method includes the steps of accessing an audio signal captured via a microphone, which is emitted by a human user in response to a prompt, and which includes capturing the audio signal while passively collecting physiological data via a radar sensor, and verifying the human user by a combination of the audio signal and one or more physiological signals.

[0020] In this example, the method may further include the steps of: aligning both one or more physiological signals and speech signals over time to correspond to the correspondence between features extracted from both a radar sensor and a microphone with respect to a common time frame; extracting speech features related to vocal cord vibration and the operation of articulatory organs from both the speech signal and one or more physiological signals, wherein the speech features define the characteristics of the speech production mechanism; and determining whether the speech signal and one or more physiological signals originate from the same sound source by analyzing whether the speech features from both the speech signal and one or more physiological signals represent a common speech production mechanism.

[0021] In some examples, the method may further include filtering acoustic and physiological signals to the same frequency range; aligning the features extracted from the acoustic and physiological signals over time to correspond to the correspondence between features extracted in a common time frame; comparing the feature variations between the acoustic and physiological signals over time for each corresponding set of features extracted from the acoustic and physiological signals; and verifying the authenticity of the human voice if the acoustic and physiological signals exhibit synchronized feature variations over time.

[0022] The aforementioned examples broadly outline the various aspects, features, and technical advantages of the examples provided herein so that the following detailed description may be better understood. It will be further understood that the above-described operations in the context of exemplary methods, devices, and computer-readable media are not required, one or more operations may be omitted, and / or other additional operations described herein may be included. Further features and advantages are described below. The concepts and specific examples illustrated and described herein may readily be used as a basis for modifying or designing other structures to accomplish the same objectives of this disclosure. Such equivalent structures shall not deviate from the spirit and scope of the appended claims. [Brief explanation of the drawing]

[0023] [Figure 1] This is a flowchart illustrating the process of human authentication using physiological signal measurement. [Figure 2] This is a flowchart of the process for measuring physiological signals from radar. [Figure 3] This is a flowchart of the process for detecting human users based on passive radar surveillance. [Figure 4] This is a flowchart of the process for determining whether an active task is required during authentication. [Figure 5] This is a flowchart of the process for verification via active tasks. [Figure 6]This is a diagram showing a source-filter model and an inverse filter in speech processing. [Figure 7] This is a graph diagram of physiological signals obtained by radar. [Figure 8] This is a diagram of radar signals measured during human speech production and related graph diagrams. [Figure 9] FIG. 9A is a time-frequency diagram of radar signals acquired during utterance, and FIG. 9B is a time-frequency diagram from which low-frequency physiological signals have been removed. [Figure 10] FIGS. 10A and 10B are graph diagrams showing feature values calculated over utterance for both a microphone and a radar sensor. [Figure 11] This is a graph diagram showing the normalized distribution of correlation (z-score) when there is a mismatch between an acoustic signal and a radar signal. [Figure 12] It shows radar reflections of a human speaker versus a speaker at three playback levels. [Figure 13] This is a simplified diagram showing a data fusion engine for voice authentication. [Figure 14] This is a simplified diagram showing an exemplary computing system for implementing the aspects of the systems and methods outlined in this specification. [Mode for Carrying Out the Invention]

[0024] Corresponding reference numerals indicate corresponding elements in the figures of the drawings. The headings used in the drawings are not intended to limit the scope of the claims.

[0025] To date, we have relied on the listener's senses to verify the authenticity of media. Until recently, synthetic media sounded and looked synthetic. As the examples highlighted in the previous section show, this is no longer possible because the quality and volume of AI-generated media is already good and is improving rapidly.

[0026] The disclosed technologies to address this have been reactive, using AI to detect AI-generated content. None of these solutions have been commercialized, but pre-trained models capable of detecting AI-generated media are installed in distribution centers (e.g., on YouTube® servers, at broadcasters), and each time a media file is uploaded, it is evaluated through these models to determine whether it was AI-generated. There are several problems with this approach. Firstly, AI for detecting deepfakes is a cat-and-mouse game that, according to those skilled in the art, is unlikely to succeed in the long run.

[0027] Furthermore, even if the AI ​​model at the distribution center is fully functional, the communication channel remains vulnerable to stream hacking. That is, the stream between the distribution center and the user's device could be hacked, allowing new audio samples to be streamed. Under this scenario, similar to the Putin example above, the user would still receive manipulated media. Detecting such attacks using reactive methods would require installing an AI model on each user's device to process all incoming samples. Power-limited edge devices are unlikely to be able to support persistent AI detection of all incoming media.

[0028] Various examples of core, preventative technical solutions to this problem are disclosed herein. These solutions involve a combination of physiological and acoustic signals to verify human speech. In some examples, the concept is based on novel microphone technology that can prove, within the recording device itself, that the recorded acoustic signal originates from a human speaking into the microphone. This proof is linked to a media file and then accessed by the listener to verify that a human produced the speech. The microphone technology is based on a combination of a conventional MEMS (micro-electromechanical systems) microphone (or other types of microphone) that measures acoustic signals and a radar sensor that measures physiological vibrations from the human during speech production. These two signals are verified against each other during recording to prove that the speaker is human. Example 1

[0029] This embodiment discloses a next-generation CAPTCHA system that relies on the direct measurement of physiological signals emitted from the human body to distinguish between humans and automated bots.

[0030] This system uses radar sensors to evaluate physiological signals while the user is passive or, if necessary, performing an active task. Several existing websites have buttons or similar variations that say, "Click here to prove you are human," without any subsequent task. They often rely on more nuanced methods to distinguish between humans and bots. For example, many operate by monitoring mouse movements or touch events. The paths and behaviors of how a human moves a mouse or interacts with a touchscreen provide evidence of human operation and involvement. Therefore, if a button is clicked and there is a corresponding mouse movement or touch event in a way that appears human, the site can reasonably conclude that the user is indeed human. While these methods are less disruptive to the user experience than traditional CAPTCHA systems, they may also be less secure, as they can be easier for sophisticated bots to circumvent. This system combines the less disruptive user experience of this type of CAPTCHA with improved security. This is achieved by directly measuring a person's physiological signals to authenticate that they are human (rather than relying simply on mouse or touch movements).

[0031] Figure 1 shows a flowchart illustrating the proposed process for human authentication by measuring physiological signals. User physiological signals are passively collected via a radar sensor (Algorithm 1) as shown in Figure 2. These signals may include respiration, cardiac activity, lip movements, and / or vocal cord vibrations. These signals are then used to detect whether a human is using the device (Algorithm 2), as shown in Figure 3. If it is ultimately not possible to determine whether a human user is present, an active task is performed as shown in Figure 4 (Algorithm 3).

[0032] The active task includes a speech task in which a prompt is provided to the user. The user is asked to verbally respond to the prompt and generate both an acoustic signal and a radar signal. The acoustic signal is captured by a microphone. The radar signal arises from vocal cord vibration, the operation of articulatory organs, as well as heartbeat and respiration. Next, as shown in Figure 5, both signals are processed to validate the radar signal against the acoustic signal (Algorithm 4). This validation ensures that the task was performed correctly, that both signals originate from the same sound source, and that the signals were actually generated by a human user. The final result of this process is verification that a human generated the signals and confirms the user's authenticity. A description of each algorithm is provided in the following sections.

[0033] This system provides a robust and reliable method for verifying human identity by combining passive physiological signal detection with active task execution as needed. The use of both microphones and radar sensors ensures accurate and comprehensive user verification.

[0034] Generating a range Doppler representation from radar signals suitable for measuring physiological signals from humans involves several preprocessing steps. Some common preprocessing steps include:

[0035] Signal Acquisition: The first step is to acquire the raw radar signal. This involves setting up the radar system and capturing the backscatter signal from a human subject.

[0036] Distance compression: This process involves processing the received radar signal to improve range resolution. This is typically done using matched filters or pulse compression techniques that maximize the signal-to-noise ratio (SNR).

[0037] Doppler Processing: The next step is to perform Doppler processing. This involves applying a Fast Fourier Transform (FFT) to the range-compressed signal to separate the signal based on its Doppler shift. The Doppler shift is the change in frequency due to the relative motion between the radar system and the human subject.

[0038] Clutter Suppression: Clutter refers to undesirable radar reflections that may interfere with the desired signal from a human subject. Clutter can originate from stationary objects in the environment or from unrelated parts of the human body. Clutter suppression techniques, such as high-pass filtering or adaptive filtering, are used to minimize these effects.

[0039] Range Doppler Map Generation: After clutter suppression, the processed signal is used to generate a range Doppler map. This is a 2D representation showing Doppler shift (representing velocity) on one axis and range (representing distance) on the other axis. The intensity at each point in the map represents the radar reflectance at that distance and velocity.

[0040] Normalization: Range Doppler maps are often normalized to a standard scale to facilitate further analysis. This can involve subtracting the mean and dividing by the standard deviation, or scaling the values ​​so that they are between 0 and 1.

[0041] Algorithm 2 is designed to detect whether a human user is interacting with the device, as shown in the flowchart in Figure 3. This algorithm operates on the data representation output by Algorithm 1, which provides a suitable format for extracting physiological signals from the acquired radar signals.

[0042] As shown, the first step in Algorithm 2 is feature extraction. This process involves summarizing the data representation into a more compact form by extracting both spectral and temporal features. Spectral features capture the frequency components of the physiological signal, while temporal features represent how these signals change over time. This dual extraction allows the algorithm to capture a comprehensive profile of the user's physiological signal.

[0043] Once features are extracted, they are evaluated by a pre-trained model. This model is pre-trained on a database of humans with the same features, enabling it to effectively distinguish between human and non-human users. The model evaluates the extracted features and determines whether they match patterns of human users.

[0044] This process is not a one-time operation. Instead, Algorithm 2 is designed to repeat this process periodically while the user is working on the device. This ensures continuous monitoring and provides periodic continuous (or discrete) predictions of whether the user is human.

[0045] Algorithm 3 is designed to determine whether an active task is necessary for human authentication or whether passive-radar-based detection is sufficient. Algorithm 3 operates based on the output of Algorithm 2, which detects whether a human user is interacting with the device. If Algorithm 2 makes a discrete decision that a human has been detected, an active task is not necessary. In this case, the user is authenticated as human based solely on passive-radar-based detection. If Algorithm 2 determines that no human was detected, an active task is initiated.

[0046] If the output of Algorithm 2 is continuous, the decision to initiate an active task is based on a threshold. If the continuous output value falls below the threshold, indicating uncertainty in human detection, the active task is initiated. If the output value exceeds the threshold, indicating firm confidence in human detection, an active task is not required.

[0047] The process shown in Figure 3 ensures that active tasks requiring user interaction are only initiated when necessary, thereby improving the user experience while maintaining robust human authentication.

[0048] Active task activation: This is the initial stage in which an active task is triggered, typically when passive detection from algorithm 3 is uncertain or below a certain threshold.

[0049] Task prompt for verbal output: At this stage, the device prompts the user to perform a task that requires a verbal response. This could be simple spoken words or answering a question.

[0050] Activation of acoustic and radar sensors: When the user speaks in response to a prompt, the acoustic sensor (microphone) captures the acoustic voice signal, and the radar sensor captures the corresponding physiological signal.

[0051] Sensor Temporal Alignment: Signals from acoustic and radar sensors are temporally aligned. This ensures that features extracted from both sensors correspond to the same time frame in the speech generation process. The sensors may be located in the same place or distributed depending on the application.

[0052] Feature extraction: Features related to vocal cord vibration and the operation of articulatory organs are extracted from both acoustic and radar signals. These features capture the essential characteristics of the speech production mechanism.

[0053] Feature Comparison: Features extracted from acoustic signals and radar signals are compared. This comparison aims to ensure that the features represent the same speech generation mechanism and to verify that the microphone signal and radar signal originated from the same sound source.

[0054] Verification of the Same Voice Generation Source: The final stage is verification of the voice generation source. If the comparison of features indicates that both signals are likely to have originated from the same source, the user is verified as human. Otherwise, the process may be repeated, or other measures may be taken. Novel aspects of the present invention

[0055] Novel aspects of this system include the following:

[0056] Radar use for CAPTCHA: The use of radar sensors in CAPTCHA devices is unique. Radar sensors passively evaluate physiological signals.

[0057] Active and Passive Human Detection: The device operates in both passive and active modes for human detection. In passive mode, radar signals are used to evaluate physiological signals such as respiration, heart rate, lip movements, and vocal cord vibrations. If passive detection is uncertain, the device switches to active mode, which involves having the user perform a speech task. The microphone captures the user's speech during the active task. This combination of radar sensor and microphone during the active task enables a more robust and reliable human detection and verification process.

[0058] Use of algorithms for signal processing and human detection: The device includes a processor that preprocesses radar signals, extracts and compares features from radar and acoustic signals, and executes a set of algorithms to determine whether an active task is required. These algorithms executed by the processor enable the device to accurately determine whether the user is human and to verify the source of sound generation.

[0059] Verification of radar signals against acoustic signals: The device verifies radar signal measurements of vocal cord vibration and lip movement against audio signals. This process ensures that the radar data and microphone data have the same sound source, further enhancing the reliability of the human detection process.

[0060] Periodic Human Detection: The device periodically evaluates the user's physiological signals while they are working with the device. This feature enables continuous monitoring and authentication of the user and is particularly useful in preventing automated form submissions, unauthorized access, web scraping, spamming, and similar types of malicious activity carried out by computer programs on the web. Advantages of current technology and impact

[0061] The CAPTCHA devices disclosed herein offer several advantages over current technologies and have the potential to have a significant impact on various fields.

[0062] Enhanced Security: The combination of passive and active human detection mechanisms significantly enhances security. By using physiological signals and voice, the device can more accurately distinguish between humans and automated systems, reducing the risk of unauthorized access or malicious activity.

[0063] Non-intrusive authentication: The device's passive detection mode is non-intrusive because it uses radar to evaluate physiological signals. This can improve the user experience because it does not require active participation unless necessary.

[0064] Sequential Authentication: The device's ability to periodically evaluate the user's physiological signals enables sequential authentication. This offers a significant advantage over traditional one-time authentication methods, as it allows for detection of whether an authenticated user has been replaced by a fraudulent user during a session.

[0065] Versatility: By using both acoustic and radar signals, the device becomes versatile and adaptable to a variety of environments and conditions. This allows it to be used in noisy environments where acoustic signals alone are unreliable, or in situations where physiological signals can provide additional verification.

[0066] Impact on Accessibility: This technology may also have a positive impact on accessibility. For users who may have difficulty with traditional CAPTCHA tasks, such as users with visual impairments, the use of physiological and auditory signals can provide an alternative method of verification.

[0067] Integration Potential: This technology has the potential to be integrated into a wide range of devices and systems, from smartphones and computers to secure entry systems and online platforms, enhancing security across a broad range of applications. Example 2

[0068] This embodiment discloses a core preventative solution to the voice cloning (deepfake) problem. This solution is based on a novel microphone technology that allows the recording device itself to prove that the recorded acoustic signal originates from a human speaking into the microphone. This proof is linked to a given media file, which can then be accessed by the listener to verify that a human produced the voice. The microphone technology is based on a combination of a conventional MEMS (micro-electromechanical systems) microphone (or other type of microphone) that measures acoustic signals and a radar sensor that measures physiological vibrations from the human voice-producing mechanism. These two signals are verified against each other during recording to prove that the speaker is human. The relationship between speech production and acoustic signals, vocal cord vibration, and the operation of articulatory organs.

[0069] Speech production is a complex physiological process involving the finely coordinated interactions of various bodily parts. A simplified physiological model of this process is shown in Figure 6, where speech is produced by vocal cord vibration, further shaped as it passes through the oral and nasal cavities, and ultimately results in speech. This process begins with the production of voiced sounds in the larynx, a process known as phonation. Hereinafter, the vocal folds (also known as the vocal cords), located within the larynx, play a central role. As air from the lungs passes through the tightly closed vocal folds, the air vibrates the vocal folds. The rapid, repetitive opening and closing of the vocal folds modulates the airflow, generating a series of pressure waves.

[0070] These pressure waves form a primary acoustic signal that carries the fundamental frequency of the speaker's voice. This frequency is largely determined by physical characteristics such as the length and tension of the individual's vocal cords, as well as the speed at which they vibrate. This fundamental frequency is typically in the range of approximately 80–240 Hz in adults.

[0071] Next, the primary acoustic signal travels through the vocal tract, namely the throat, oral cavity, and nasal cavity. As it passes through these structures, the signal is shaped and filtered by the resonant properties of the vocal tract, resulting in the rich and complex acoustic signal that is perceived as a human voice.

[0072] Therefore, the speaker's physiological measurements are inherently related to the generation of speech signals. Consequently, the vibration of the vocal cord folds and the operation of the articulatory organs are directly related to the acoustic data captured by the microphone. Audio source filter models and inverse filtering in audio processing

[0073] In the field of speech processing, the sound source-filter model conceptualizes speech generation and serves as a fundamental framework for processing speech signals. According to this model, the generation of spoken speech involves two main components: the sound source and the filter. The sound source component refers to voiced or voiceless sounds generated in the larynx, essentially by vocal cord vibrations for voiced sounds. The filter represents the vocal tract configuration, which resonates these sound source excitations and shapes them into recognizable spoken speech.

[0074] The acoustic signal perceived by the speaker's mouth is a convolution of the sound source excitation and the vocal tract filter response, resulting in a complex and unique speech output recognized as individual utterances. The vocal tract imposes its resonant characteristics on the sound source signal, generating a series of formants or spectral peaks in the acoustic output. Mathematically, this can be modeled as a convolution of the sound source signal (vocal cord vibration) and the vocal tract filter (operation of the articulatory organs) (see the block diagram at the top of Figure 6).

[0075] Inverse filtering is a technique used to deconvolve or separate source and filter components from an observed speech signal. Essentially, it aims to reverse the filtering effect of the vocal tract to obtain the original source excitation, which in the case of voiced sounds is the vocal cord vibration signal (see the block diagram below in Figure 6). This can be achieved, among other methods, by linear predictive coding (LPC) or cepstrum analysis, which estimate the vocal tract filter and then "remove" its effect from the acoustic signal. By successfully applying inverse filtering, the vocal cord vibration signal and the structure of the articulatory organs can be extracted from the acoustic speech signal, thereby providing valuable information about the physiological aspects of speech production, namely, the vibration of the vocal cord folds and the operation of the articulatory organs (lips, jaw, soft palate).

[0076] The estimated source signal via inverse filtering is directly related to the vibration of the vocal cord folds, and the estimated filtered signal is directly related to the movement of the articulatory organs (lips, tongue, jaw, soft palate). In other words, if the vibration of the vocal cord folds or the movement of the articulatory organs is directly measured, the signal closely matches the source signal estimated from the measured acoustic speech signal. Measurement of human physiological signals during speech using radar.

[0077] Radar, an abbreviation for radio detection and ranging, is a technology that uses radio waves to detect and determine the distance, direction, and speed of an object. It works by emitting radio signals that bounce off any object in their path, and the returned signals are intercepted by the radar system. Radar technology accurately measures several important parameters of a detected object. It measures the "range" or "distance" to an object by calculating the time it takes for the radio waves to return after bouncing off the object. It determines the "velocity" or "speed" of an object using the Doppler effect, and the frequency shift of the returned signal indicates movement toward or away from the radar.

[0078] It is possible to acquire a range of physiological signals by using a radar system that targets individuals engaged in speech, measuring vibrations emitted from the individual's body that correspond to various physiological processes. These include vibrations from the heartbeat driven by the rhythmic contraction and relaxation of the heart, respiratory movements from the expansion and contraction of the lungs, articulatory movements including the dynamics of the lips and jaw, and vocal cord vibrations essential for phonation.

[0079] Figure 7 shows each of these physiological processes, the inherent frequency bands they occupy, and the magnitude of their corresponding radar signals. Respiration, typically reflecting the slow, regular expansion and contraction of the lungs, is in the range of 0.2 Hz to 0.5 Hz. Cardiac activity, more specifically the heartbeat, is observed in a slightly broader, higher frequency band of 0.2 to 2 Hz. Articulatory movements of the lips during speech occur in the 3 to 7 Hz band, due to the rapid and complex movements involved in speech production.

[0080] Furthermore, vocal cord vibrations occur in a remarkably high frequency range of 80–240 Hz. The rapid and repetitive opening and closing of the vocal cord folds during phonation causes these vibrations, and these vibrations are essential for the production of voiced sounds.

[0081] While there is some overlap in the frequency bands of respiration and heartbeat, vocal cord vibrations exist in a distinctly separate frequency band. This band does not overlap with any other human-generated physiological signals, making it uniquely identifiable. This characteristic separation simplifies the task of isolating vocal cord vibration signals from comprehensive physiological data collected by radar systems.

[0082] Respiratory movements typically generate the largest radar signals. This is due to large chest wall displacements and considerable internal lung movements during respiration. As a result of the large scale of this action, radar signals are reflected more strongly.

[0083] Cardiac activity is regular, but because the physical displacements involved are relatively small, it produces a weaker radar signal. The heart is a smaller organ, and its movement during contraction is limited compared to the larger-scale respiratory movement.

[0084] Lip and jaw movements during speech have relatively high frequencies but generate relatively small radar signals. The body parts involved in articulation are comparatively small and exhibit limited movement even when articulating complex speech sounds, and therefore the radar signals reflected by the receiver are small.

[0085] Finally, vocal cord vibrations, despite their high frequencies, produce the smallest radar signal. The vocal cord folds are small, their movement during speech is very small in amplitude, and they are located deep within the body. These factors result in a very small radar cross-section, and their movement causes only slight changes in the overall reflected radar signal. However, the uniqueness of the frequency band of vocal cord vibrations allows for their detection and separation for analysis, despite their low signal magnitude.

[0086] This is shown in Figure 8, where a radar system is positioned in front of a human speaker. Here, a small 77 GHz millimeter-wave (mmWave) radar sensor is used to simultaneously measure the human voice and heartbeat while the subject utters a simple word or sentence. The speaker is positioned 0.7 m away from the microphone and radar sensor. The acquired radar signatures in the time and frequency domains are shown in Figure 8. The resulting spectrum shows a clear separation of low-frequency physiological signals (respiration, heart rate, and articulatory organ activity) from high-frequency physiological signals (vocal cord vibration).

[0087] Figure 9A is a time-frequency diagram of the radar signal acquired during speech. This diagram shows vocal cord vibrations associated with higher-frequency speech signals and lower-frequency physiological signals. The low-frequency physiological signals are then removed to obtain a separated diagram of the speech signal captured by the radar sensor, as shown in Figure 9B. This diagram shows the changes in pitch and pitch harmonics over time. This signal can then be converted back to the time domain to generate a time-domain estimate of the generated speech signal. If the speech signal is also captured by a microphone, the radar signal shown in Figure 9B can be compared with the acoustic signal acquired via the microphone. Verification of acoustic signals against radar signals

[0088] Synchronizing and verifying audio and radar signals to ensure they originate from the same individual involves several processing steps. Data collected from both the radar sensor and microphone represent a combination of the individual's physiological and acoustic characteristics. The system can then automatically and algorithmically verify that the measured audio signals were human-generated by linking these data streams together and verifying that they originate from the same sound source. This includes the following steps:

[0089] Data Acquisition and Time Synchronization: Data acquisition is performed simultaneously using radar sensors and microphones. The two sensors are synchronized via a common clock or by timestamping the data from both sensors.

[0090] Audio signal processing: Microphone-recorded audio signals are analyzed using various digital signal processing techniques. This may include inverse filtering to separate the sound source and filter, as well as extracting features such as the fundamental frequency, formant, spectral characteristics, and timing features of the audio signal. This information represents the unique characteristics of an individual's voice.

[0091] Radar signal processing: Radar signals representing physiological movements related to speech generation are processed. This includes the aforementioned processes such as range Doppler processing and signal filtering. Specifically, high-frequency vocal cord vibrations and articulatory movements are important because they are closely coupled with the speech features extracted from the sound source and filter.

[0092] Cross-modal comparison and verification: Next, key features from both the radar signal and the voice signal, such as vocal cord vibration patterns and frequencies, articulatory movement timing, and other speech characteristics, are compared with each other. By correlating these two datasets, it can be verified that they actually originate from the same individual. Advanced signal processing or machine learning algorithms may be used to perform this cross-modal integration. These algorithms may compare and contrast feature sets from both signals, thereby establishing relationships between them. High correlation or agreement may suggest that the radar signal and the voice signal originate from the same person.

[0093] Continuous monitoring and authentication: By continuously monitoring and comparing radar and audio signals, it is possible to prove the persistence of the same speaker. Significant deviations in the comparison metrics may indicate a change in the speaker or a change in the acoustic signal.

[0094] Using the same setup as in Figures 8, 9A, and 9B, a speech utterance from the speaker ("She puts your dark suit in oily laundry water all year round") was collected and recorded using a microphone and radar sensor. The acoustic and radar signals were filtered to the same frequency range (0–500 Hz) and then time-matched by finding the cross-correlation peaks. From each signal, a set of features were extracted to track harmonics, temporal variation, and several spectral parameters of both signals. Figure 10A shows how the normalized harmonic features change synchronously across both modalities when there is a match between the acoustic and radar signals. Figure 10B shows how the same features do not change synchronously when there is no match between the acoustic and radar signals. From Figure 10A, it is clear that features across both modalities can be tracked temporally, enabling cross-modal verification. That is, the high correlation between features indicates that it is possible to verify that the acoustic signal recorded by the microphone originates from the speaker's vocal cord vibrations and articulatory organ activity measured by the radar sensor. The lack of correlation between the two signals in Figure 10B indicates that such cross-modal verification can be performed with high reliability in this case.

[0095] To further evaluate the confidence level at which cross-modal verification can be performed, a distribution of feature correlations with mismatches between acoustic and radar signals was generated. These were randomly selected from an existing corpus of 350 speech samples that were not collected simultaneously with those captured by radar. That is, these speech samples are mismatched with those collected by radar and spoken by different speakers speaking different sentences. For each speech sample, feature values ​​for the acoustic and radar signals were calculated, and the correlation between the two feature sets was calculated for each speech utterance. The normalized distribution of correlations (z-scores) is shown in blue in Figure 11. In the same figure, the true correlation (z-score) when both signals match is shown in red. As the figure shows, even using a single feature, the true correlation value of this feature is approximately 4 standard deviations from the mean, indicating that it is possible to determine the match between the acoustic and radar signals with very high confidence even with a single utterance. Link the certificate to the media.

[0096] After verification, the resulting authentication certificate can be linked to media to verify its origin and ensure that it is not human-generated, altered, or forged. Several methods can be used to link a certificate to media. These include, but are not limited to, the following:

[0097] Digitally signing using existing software: Some media editing or security software may allow you to digitally sign media files directly. Tools tailored for digital rights management (DRM) or secure media processing may include this functionality.

[0098] Hash and Signature: Create a hash of the audio file, which is a unique numerical value representing the file's content. Sign this hash using a private key to create a digital signature. The signature and public key can be stored with the audio file or embedded in the metadata, allowing recipients with the corresponding public key to verify the signature.

[0099] Embedding signatures in file metadata: Audio file metadata can often store additional information such as artist details and album name. Digital signatures can also be embedded in this metadata, but care must be taken to ensure that this does not interfere with file playback.

[0100] Embedding a signature in a watermark: Applying a digital watermark to media. This can be a visible or invisible pattern containing authentication information.

[0101] Blockchain technology: Blockchain is used to create verifiable and immutable records of media. For example, a platform like Ethereum can be used to write smart contracts and encode the authenticity information of media within the blockchain.

[0102] Using standard file signing tools with wrappers: If audio files are contained within a wrapper format (such as a ZIP file) that can be digitally signed using standard file signing tools, the entire package containing the audio content can be signed. The recipient can then verify the package's signature, thereby verifying the authenticity of the audio files inside.

[0103] Other custom solutions: Depending on specific requirements, we can develop custom solutions for digitally signing media files. For example, radar features can be linked to audio files via existing tools (e.g., watermarks, blockchain, etc.). Then, during authentication, the audio features can be extracted and evaluated against the radar features to ensure a match. The complexity and challenges of vocal cord vibration in spoofed radar acquisition signals

[0104] The process of vocal cord vibration is a highly complex physiological phenomenon that directly results in the acoustic signals produced when humans speak. The vocal cord folds, located in the larynx, vibrate at high frequencies in the range of approximately 80–240 Hz in adults during vocalization. However, the vibration is not a simple periodic motion, but rather a complex three-dimensional mucosal wave action involving longitudinal, transverse, and vertical displacements. These vibrations are influenced by numerous factors, including the length, tension, and mass of the vocal cord folds, subglottal pressure, vocal tract configuration, and individual physiological characteristics.

[0105] The radar signals resulting from vocal cord vibrations, along with other physiological features of the anatomical structure of the speaker's vocal organs, carry these complex movement patterns as frequency modulation. When combined with other physiological signals captured by radar sensors, such as respiration and cardiac activity, the combination of these characteristics forms a unique "human print" that is extremely difficult to artificially reproduce or imitate.

[0106] To spoof such radar signals, it is necessary not only to mimic the intrinsic frequency characteristics of a speaker's vocal cord vibrations but also to reproduce those specific physiological features that affect the radar cross-section. Furthermore, the complexity of the three-dimensional high-frequency mucosal wave action of the vocal cord folds further amplifies this challenge. Given the currently available technologies and understanding of human physiology and radar signal processing, successfully spoofing radar signals resulting from vocal cord vibrations presents considerable challenges.

[0107] One simple attempt to impersonate a system could be to place a speaker in front of the radar so that the microphone measures an acoustic signal while the radar measures the vibration of the speaker's diaphragm. To detect attempts to impersonate a system, a machine learning algorithm can be implemented to ensure that a human is in front of the microphone (not a synthetic diaphragm). Through machine learning, the algorithm can be trained with a vast number of radar signals from human speech, allowing it to learn to recognize the subtle patterns and characteristics of radar reflections typical of a living human speaker in speech. This model can then be used to evaluate whether the acquired signal is real. Figure 12 below shows radar reflections from a human speaker and reflections from a vibrating speaker diaphragm (three reproduction levels). This figure shows clear differences between human reflections and speaker reflections. The first obvious difference is the difference in reflections when there is no voiced sound being produced. The background noise level is much higher for speaker reflections. The second obvious difference is that speaker reflections are larger (speaker reflections are redder). A third obvious difference is the difference in reflection for high-frequency harmonics. As the figure shows, radar reflection for high-frequency harmonics is more pronounced, especially at medium and high volume levels. This figure provides evidence that it should be possible to train a statistical model via machine learning to capture radar reflections from human speakers. Novel aspects of the present invention

[0108] A novel aspect of the present invention lies in a unique method for verifying speech authenticity that combines conventional acoustic signal recording with physiological vibration measurement. Some key innovative elements of this method include:

[0109] Dual Sensor Technology: The integration of a conventional MEMS microphone and a radar sensor is a novel approach. While the microphone records acoustic signals, the radar sensor measures physiological vibrations from the human speech-producing mechanism. This dual-sensor configuration provides a more comprehensive and accurate method for verifying the presence of a human speaker. The two sensors can be placed in the same location or dispersed, depending on the application.

[0110] Real-time verification: The ability to verify that the voice generator is human during recording is a significant advancement. This real-time verification can prevent the harmful effects of deepfakes or other synthesized audio from the outset.

[0111] Certificate Linking: The ability to link authenticity certificates to the media files themselves is another novel feature. This feature allows listeners to directly access verification information, providing immediate assurance of the audio's authenticity.

[0112] Physiological Oscillatory Measurement: The use of radar technology to measure physiological oscillations associated with human speech production is a unique approach. This method provides biological confirmation of human speakers, which is difficult to reproduce artificially.

[0113] On-device verification: The fact that the verification process takes place on a computing device while the acoustic signal is being recorded is also innovative. This on-device processing can enhance the safety and efficiency of the verification process. Advantages of current technology and impact

[0114] The proposed voice authentication system offers several advantages over current technologies and could have a significant impact.

[0115] Authenticity Verification: The main advantage of this invention is its ability to verify the authenticity of audio content. This technology can provide a certificate of authenticity that proves the audio was generated by a human and not by AI or deepfake. This feature is becoming increasingly important in an era where deepfakes and synthetic media are becoming more prevalent.

[0116] Prevention of misinformation: By verifying the source of audio signals, the present invention can help prevent the spread of misinformation and false information. It can be an important tool in maintaining the integrity of news broadcasts, political speeches, and other public communications.

[0117] Security and Privacy: This invention can enhance security in communication systems. For example, it can prevent voice cloning fraud by ensuring that the speaker is a real person. It can also be used in secure voice biometric systems by adding an additional layer of security.

[0118] Legal and Forensic Applications: In a legal context, the present invention can help verify the authenticity of audio evidence. It can also be used in forensic investigations to identify speakers in audio recordings.

[0119] Music and Entertainment Industry: In the music and entertainment industry, the present invention can help protect the rights of artists by ensuring that audio recordings are created by human artists and not artificially generated.

[0120] The impact of this invention could be far-reaching, potentially changing how humans trust and engage with audio media. By providing a reliable method for verifying authenticity, which is becoming increasingly important in the digital age, trust in digital media can be restored. Applications of this technology

[0121] Ensuring secure communication: Fraud based on voice cloning is beginning to emerge. Having a human verification checkmark during a conversation ensures that the other party is a human speaker and not an AI bot. In addition to consumer applications, such technology has considerable potential in the defense sector.

[0122] Next-Generation CAPTCHA: With advancements in AI technology, current CAPTCHA techniques will soon become obsolete. It is becoming increasingly easy for AI to detect distorted numbers / text or identify specific objects within a visual scene. Next-generation CAPTCHA will require human verification through physiological measurements. Radar-based systems will enable efficient verification.

[0123] Authenticating music, podcasts, and other forms of content as human-made: As generative AI improves, AI-generated entertainment content may outnumber human-generated content online. Under this scenario, distinguishing between human-made and AI-generated media offers opportunities for new business models (e.g., different monetization strategies for human content versus AI content).

[0124] Protecting and Monetizing the Voices of Celebrities / Influencers: As AI models improve, generating synthetic media from celebrities and influencers will become easier. Synthetic voices are already being used for celebrity hate speech (see, for example, https: / / www.theverge.com / 2023 / 1 / 31 / 23579289 / ai-voice-clone-deepfake-abuse-4chan-elevenlabs) and to generate AI-generated songs in the voices and styles of famous artists (https: / / www.npr.org / 2023 / 04 / 21 / 1171032649 / ai-music-heart-on-my-sleeve-drake-the-weeknd). Verification of human-generated content will allow celebrities to protect their voices and brands. There are also some very interesting future markets. For example, some celebrities are developing AI clones that allow fans to interact with them (see, for example, https: / / www.washingtonpost.com / technology / 2023 / 05 / 13 / caryn-ai-technology-gpt-4 / ). Validation of human-generated content enables tiered pricing, where real-life interactions are more expensive than interactions with AI-generated bots.

[0125] Virtual Assistant Authentication: Human users can be verified by a virtual assistant, such as Alexa or Siri, before any command is executed, providing an additional layer of security when communicating with smart devices.

[0126] Personhood Verification / Proof of Humanity: Personhood verification / proof of human identity is a method of verifying that a user is a unique, genuine human being and not a bot or fake account. There are two main forms of personhood verification: social graph-based and biometric. Social graph-based personhood verification relies on some form of bouching, while biometric personhood verification involves verifying some physical or behavioral trait that distinguishes a human from a bot and distinguishes individual humans from one another. Voice biometrics can play a role in personhood verification by verifying the identity of a registered person. A biometric voice recognition system captures a new voice sample, creates a template from the sample, and compares it to a registered template. A strong match between templates indicates that the same person spoke both samples and thus verifies the person's identity. The proposed technology may be used in the context of such a system to provide additional evidence that the user generating the voice is human. Example 3

[0127] This embodiment outlines a method and system for authenticating human voices and protecting against deepfake voice spoofing. This embodiment integrates various biosensors to capture physiological and acoustic signals that indicate natural human voice production.

[0128] This system includes a wearable device capable of authenticating human voice by detecting a series of biosignals and acoustic signals. Its general form is adaptable and can be embodied as an in-ear device, headphones, or head-mounted device. The device incorporates a series of sensors, including a bone conduction microphone, a photoplethysmography sensor, an accelerometer, a gyroscope, and a temperature sensor. These sensors work together to capture the vibrational, physiological, and thermal characteristics unique to human voice produced by an individual. The objective is to provide a robust method for distinguishing human voice from non-human sound sources in potential applications in security, personal identification, health monitoring, and interactive technologies.

[0129] Deepfake technology necessitates the development of reliable methods for authenticating the authenticity of multimedia content, particularly human speech. The systems and methods outlined herein are a response to the need for robust, real-time methods capable of distinguishing genuine human speech from deepfake-generated speech. These methods leverage recent advances in sensor technology and algorithmic analysis for evaluating bio- and acoustic signals inherent in natural speech, providing a new layer of defense against the escalating threat of deepfakes.

[0130] As in the example above, once a human voice is verified, it can be linked to an associated media file to allow the listener to verify that the voice is human-generated. contribution

[0131] The systems and methods outlined herein make the following contributions:

[0132] Multi-sensor fusion: This method integrates various sensors, such as bone conduction microphones, PPG sensors, accelerometers, gyroscopes, and thermal sensors, in a wearable format to capture a comprehensive dataset of physiological and acoustic signals related to speech generation. This fusion of multiple data types for raw speech verification is unique.

[0133] Real-time deepfake detection: By analyzing physiological signals that are difficult to synthesize, such as subtle blood flow changes or unique patterns of jaw movement associated with speech, the system can detect deepfakes in real time. This is a significant improvement over existing technologies that typically only analyze speech and lack real-time processing capabilities.

[0134] Wearable form factor: This method is designed to be integrated into wearable devices (e.g., hearables, head-mounted microphones, combinations, etc.) and provides a discreet and convenient form factor for the user.

[0135] Physiological Signal Authentication: Existing technologies primarily focus on the acoustic analysis of speech for deepfake detection. In contrast, the systems outlined herein authenticate speech through underlying physiological processes, providing a novel approach to inherently more secure verification against current and future deepfake technologies.

[0136] Enhanced Security: By utilizing physiological data alongside acoustic signals, the systems outlined herein offer an inherently higher level of security against impersonation attempts, including sophisticated deepfake voice spoofing. Current deepfake detection techniques, which rely on acoustic output, are inconsistent in performance and lag behind the rapid advancements in deepfake generation.

[0137] Real-time authentication: Unlike many current technologies that operate on pre-recorded samples and lack the ability to perform on-the-fly verification, this method enables real-time processing and authentication of raw speech, which is crucial for dynamic security systems and interactive applications.

[0138] Versatility of application: The wearable nature of the device enables a wide range of applications, from mobile security to healthcare monitoring, without requiring the special environments or settings that some current technologies require.

[0139] Non-invasive and user-friendly: The systems outlined herein are designed to be non-invasive and user-friendly, seamlessly integrating into everyday devices such as headphones or glasses, providing users with a frictionless experience.

[0140] Robustness to environmental fluctuations: Due to its ability to capture and analyze physiological signals, the systems outlined herein are less susceptible to environmental noise and fluctuations that can affect only acoustically-based technologies. Core components and methods Sensor array

[0141] The system uses multiple sensors that can be configured in various wearable forms, including but not limited to in-ear devices, neckbands, headsets, or glasses. These sensors include, but are not limited to, the following:

[0142] Bone conduction microphone: Captures vocal cord vibrations transmitted through the skull.

[0143] Photoplethysmography (PPG) sensor: Corresponds to human skin and detects changes in blood volume in ear or skin tissue that can respond to speech effort.

[0144] Accelerometer and gyroscope: Monitor vibrations during speech generation and minute movements associated with jaw and head movements during speech.

[0145] Thermal sensor: Measures the temperature near the skin surface, which may fluctuate in conjunction with speech due to changes in blood flow.

[0146] High-fidelity microphone: Uses ambient noise reduction technology to capture voice signals inside the ear canal or near the mouth. Signal processing

[0147] The system uses advanced signal processing algorithms to analyze data collected from multiple sensing modalities and prepares it for a data fusion engine. This includes extracting features from different sensing modalities.

[0148] Temporal features: Features that measure the timing and sequence of speech-related physiological changes.

[0149] Spectral features: Features that are useful for decomposing an audio signal into its constituent frequencies and identifying the acoustic, vibrational, and other physiological cues associated with speech generation. Data Fusion Engine

[0150] Figure 13 shows a data fusion engine 1300 that can be implemented using a computing device (e.g., computing device 1400 in Figure 14) that is mounted on or communicates with various wearable devices outlined herein. The data fusion engine ensures the human authenticity of the speech. This can be done in two ways. First, the collected signals are fused to create a multidimensional profile of the speech event using features extracted in the previous step. This profile is compared to known profiles of authentic speech to determine the likelihood that the speech is from a living human. Second, several sensing modalities are evaluated against each other to ensure the presence of speech from a living human. For example, vibration signatures (measured from a bone conduction microphone), articulatory organ movements (measured from an accelerometer), and acoustic output (measured from a microphone) can be validated against each other to ensure they have the same sound source (human speaker). Output and Integration

[0151] The system provides an output that demonstrates the authenticity of the human voice. This output can be integrated into security systems requiring voice authentication, communication devices for verifying speaker identity, media platforms for screening deepfake content, or content authentication platforms for verifying genuine human voices. User Interface

[0152] The system includes a user interface for calibration, monitoring, and alerting the user regarding authentication status. It can be a standalone application or integrated into an existing software ecosystem. Applicability and Embodiments

[0153] Device form factors can be adapted to suit discreet use, comfort, and user-specific applications, from consumer electronics to high-security communication devices.

[0154] The sensor arrays and processing algorithms can be customized to suit various security levels, from basic authentication to highly secure environments where voice spoofing poses a significant risk.

[0155] This method is adaptable to different groups and languages ​​and accommodates variations in speech patterns and physiological responses.

[0156] The system is designed for both real-time, raw voice authentication and post-recorded voice analysis, allowing for retrospective verification of its authenticity. Related features for signal processing components

[0157] Possible features that can be extracted from each perceptual modality to generate a human profile during speech include (this is not an exhaustive list): Bone conduction microphone:

[0158] Vibrational energy: This measures the energy of vocal cord vibrations transmitted through the skull, demonstrating raw speech production.

[0159] Fundamental frequency (F0): Tracks the pitch of the speaker's voice, which naturally fluctuates in the speech of a living human being. Photoplethysmography (PPG) sensor:

[0160] Blood volume pulse wave (BVP): Detects heart rate variability that may correlate with speech effort and emotional state.

[0161] Oxygen saturation: Monitor changes in blood oxygen levels, which may vary depending on the breathing pattern associated with the voice. Accelerometer and gyroscope:

[0162] Jaw movement patterns: These are unique to each individual and difficult to reproduce with synthesized speech; they capture the dynamics of jaw movements during speech.

[0163] Head movements: Analyzing head movements associated with natural speech provides clues to the authenticity of speech acts. Thermal sensor:

[0164] Temperature fluctuations: Measure fluctuations in skin temperature near the mouth or throat, which can change due to speech caused by changes in blood flow.

[0165] Respiratory temperature: Detects the temperature of exhaled breath during speech, which can demonstrate the production of natural speech. High-fidelity microphone:

[0166] Spectral features: Extracts features such as Mel-frequency cepstrum coefficients (MFCCs) that capture the unique timbre of the speaker's voice.

[0167] Temporal Dynamics: Analyzes the timing and duration of phonemes, syllables, and words, which are characteristic of natural speech rhythm. Data Fusion Engine

[0168] The data fusion engine integrates and analyzes data from multiple sensors to authenticate the human origin of voice signals. The process includes the following: Creating multidimensional profiles:

[0169] The engine first aggregates features extracted from multiple sensors, each providing a unique perspective on the speech event. These include vibration data from bone conduction microphones, physiological signals from PPG sensors, motion data from accelerometers and gyroscopes, temperature fluctuations from thermal sensors, and acoustic signals from high-fidelity microphones. These features may also include energy, blood volume pulses, jaw movement patterns, temperature variability, and spectral features. These features represent the natural human speech production mechanism.

[0170] These extracted features are fused to create a multidimensional profile of the speech event. This profile encompasses the complex interactions of physiological and acoustic signals that characterize speech from a living human being. Comparison with known profiles:

[0171] The multidimensional profile is compared to a database of profiles associated with authentic human speech. This database is built from a wide range of samples of raw speech events, capturing the wide range of natural variations in speech production across different individuals and contexts.

[0172] The comparison process involves machine learning algorithms that capture the subtle nuances that characterize human speech. Exemplary machine learning models could include Gaussian mixture models that model the distribution of multidimensional profiles. Cross-modal verification:

[0173] In addition to comparing multidimensional profiles with known profiles, the engine also performs cross-modal validation, which involves evaluating consistency and correlation between different sensing modalities.

[0174] For example, vibration signatures captured by bone conduction microphones are compared with motion data (movement of articulatory organs) from accelerometers and acoustic output from high-fidelity microphones. The engine evaluates whether these different data streams corroborate each other and indicates that they originate from the same raw audio event.

[0175] This process is crucial for detecting deepfake attempts that can convincingly reproduce one aspect of speech generation (e.g., acoustic output) but fail to accurately mimic the overall interconnected nature of human speech. Output generation:

[0176] Based on the results of profile comparison and cross-modal verification, the engine generates output indicating the likelihood that the audio event is from a real person. This output may be a binary determination (authenticity or authenticity), a confidence score, or a detailed report highlighting the analysis's findings.

[0177] This output can then be used by various downstream systems that require voice authentication, such as security systems, communication devices, and content verification platforms, to make decisions based on information regarding the authenticity of the voice.

[0178] The verification signal can also be embedded within the audio signal as a watermark, added as metadata to the recording, or used to cryptographically sign the audio as authentic human voice. Consideration

[0179] This disclosure outlines a system and method for authenticating the voice of a living human being by analyzing physiological and acoustic signals using multiple sensors embedded in a wearable device. This method addresses the urgent need for reliable verification in the face of advanced deepfake technologies, which pose increasing threats to personal security, information integrity, and privacy. The commercial potential of this invention lies in its applications across a wide range of fields, including security, personal electronics, health monitoring, and communications. It offers a novel solution to companies seeking to improve the authenticity of digital interaction and voice control systems and promises broad licensing opportunities in areas where human presence verification is critical. Computerized Systems

[0180] Figure 14 is a schematic block diagram of an exemplary computing device 1400 that may be used in conjunction with one or more embodiments described herein as a component of a system that implements an aspect of the method described herein, including, for example, a wearable device outlined herein.

[0181] The computing device 1400 comprises one or more network interfaces 1410 (e.g., wired, wireless, PLC, etc.) interconnected by a system bus 1450, at least one processor 1420, memory 1440, and a power supply 1460 (e.g., battery, plug-in, etc.).

[0182] The computing device 1400 includes a display device 1430 that transmits the output of a user interface and / or the methods outlined herein to the user, or can otherwise communicate with the display device 1430.

[0183] The network interface(s) 1410 include mechanical, electrical, and signaling circuits for transmitting data over a communication link coupled to a communication network. The network interface(s) 1410 are configured to transmit and / or receive data using various different communication protocols. As shown in the figure, the box representing the network interface(s) 1410 is shown for simplification, and it is understood that such interfaces may represent different types of network connections, such as wireless and wired (physical) connections. Although the network interface(s) 1410 are shown separately from the power supply(s) 1460, it is understood that interfaces supporting the PLC protocol may communicate via the power supply(s) 1460 and / or may be an integrated component coupled to the power supply(s) 1460.

[0184] Memory 1440 includes a plurality of storage locations addressable by the processor 1420 and the network interface 1410 for storing software programs and data structures associated with the embodiments described herein. In some embodiments, the computing device 1400 may have limited memory or no memory at all (e.g., no memory for storage other than programs / processes running on the device and associated cache). Memory 1440 may include instructions that, when executed by the processor 1420, cause the processor 1420 to implement embodiments of the systems and methods outlined herein.

[0185] The processor 1420 comprises hardware elements or logic adapted to execute software programs (e.g., instructions) and manipulate data structures 1445. A portion of these typically resides in memory 1440, and the operating system 1442, executed by the processor, functionally organizes the computing device 1400 by, among other things, invoking operations that support software processes and / or services running on the device. These software processes and / or services may include a voice authentication process / service 1490, which may include various modular methods and / or implementations described herein. While the voice authentication process / service 1490 is shown in centralized memory 1440, it should be noted that alternative embodiments may provide a process operating within a network interface 1410, such as a component of the MAC layer, and / or as part of a distributed computing network environment.

[0186] It will be apparent to those skilled in the art that program instructions relating to the techniques described herein may be stored and executed using other processors and memory types, including various computer-readable media. Furthermore, while the description illustrates various processes, it is explicitly intended that these processes may be embodied as modules or engines configured to operate in accordance with the techniques described herein (e.g., according to the functionality of similar processes). In this regard, the terms module and engine may be used interchangeably. Generally, the terms module or engine refer to a model or organization of interrelated software components / functions. Additionally, while the voice authentication process / service 1490 is presented as a standalone process, those skilled in the art will understand that this process may be executed as a routine or module within other processes.

[0187] As described above, specific embodiments have been illustrated and explained, but it should be understood that, as will be obvious to those skilled in the art, various modifications can be made without departing from the spirit and scope of the invention. Such changes and modifications fall within the scope and teachings of the invention as defined in the appended claims.

Claims

1. A method for authenticating human voices, The capture of data associated with audio events from multiple sound sources, wherein the data includes a combination of signals related to both the physiological and acoustic characteristics of the audio events. Extracting features from the captured signal that represents the voice generated by a real human being, The characteristics extracted from the aforementioned acoustic signal and physiological signal are analyzed to authenticate human-generated speech. A method comprising verifying the combination of signals to each other in order to further confirm the authenticity of the aforementioned audio event.

2. Passively capturing the audio signal of the aforementioned signal combination via a microphone, The method further includes simultaneously capturing the physiological signals of the signal combination via at least one sensor. The method according to claim 1.

3. Extracting speech features related to vocal cord vibration and the operation of articulatory organs from both speech signals and one or more physiological signals from a combination of said signals, further comprising extracting said features, wherein the speech features define the characteristics of the speech production mechanism. The method according to claim 1.

4. Extracting features, which includes extracting spectral features and temporal features from one or more signals related to speech generation, wherein the spectral features define frequency components of the one or more physiological signals, and the temporal features represent changes in the one or more physiological signals over time. The method according to claim 1.

5. Accessing physiological signals from the combination of signals captured via radar sensors, The method of generating a range Doppler map from the physiological signal, wherein the range Doppler map includes a two-dimensional representation that defines a Doppler shift representing velocity on one axis and a range representing distance on another axis, and the intensity at each point in the range Doppler map represents the reflectivity of the sensor at a velocity useful for sound generation, within that range. The method according to claim 1.

6. The fusion of the features extracted to create a multidimensional profile of the aforementioned speech event, wherein the multidimensional profile defines the interaction of physiological and acoustic signals that characterize speech from a living human being. The further includes comparing the multidimensional profile with known profiles of authentic speech to assess the likelihood that the speech event is from a real person, The method according to claim 1.

7. To generate a feature representation of the radar signature and acoustic signal accessed from the data captured by the radar sensor, The process further includes identifying the relationship between the feature values ​​calculated from the data relating to the vocalization of the sound from the radar signature and the acoustic signal, and verifying the acoustic signal and the radar signature as a biosignal from a human as originating from a human. The method according to claim 5.

8. Further comprising generating a media file associated with the audio event, which includes a certificate of verification of the signal combination as being of human origin, The method according to claim 1.

9. The further includes generating a separated representation of the audio event from the data by removing low-frequency physiological signals from the aforementioned combination of signals. The method according to claim 1.

10. The aforementioned combination of signals A vibration signal transmitted via bone conduction or accelerometer during the aforementioned audio event, Other physiological signals to verify the presence of a real human speaker, Including an acoustic signal associated with the aforementioned audio event measured from a microphone, The method according to claim 1.

11. Filtering the aforementioned acoustic signal and the aforementioned physiological signal to the same frequency range, Aligning the extracted features from the aforementioned acoustic signal and physiological signal over time, Further includes analyzing the combination of signals, which involves comparing the characteristic changes between the acoustic signal and the physiological signal over time with respect to the extracted features, The method according to claim 1.

12. The following further includes verifying the combination of signals, including confirming the authenticity of the audio event when the acoustic signal and the physiological signal exhibit synchronous characteristic changes over time. The method according to claim 11.

Images