Audio signal authenticator and method of authenticating an audio signal

Abstract

An audio signal authenticator (200) is provided which comprises a block bounds detector and block divider (205) configured to receive an audio signal (171) and to output a sequence of a plurality of audio blocks (201) with embedded authentication information or with embedded authentication access information (212, 162) relating to where the authentication information is retrievable. The authentication information (141) comprises audio features (151) belonging to the current audio block (201). The audio signal authenticator (200) comprises a perceptual similarity analyzer (240) configured to compare the audio blocks (201) of the received audio signal (171) and the audio features (151) of the extracted or retrieved authentication information (141) or to compare audio features extracted from the audio blocks (201) of the received audio signal (171) and the audio features (151) of the extracted or retrieved authentication information (141) to each other providing a similarity result (241). The audio features are human auditory perception audio features relating to a human auditory perception.

Description

[0001] Audio signal authenticator and method of authenticating an audio signal

[0002] The present invention relates to an audio signal authenticator and a method of authenticating an audio signal.

[0003] In view of modern capabilities of creating deepfake video and deepfake audio, the need has risen to be able to verify or authenticate an audio signal like a speech of a politician etc.

[0004] Until now it has been very difficult to authenticate an audio signal of a person. Typically, a lot of manual research must be done in order to verify or authenticate such an audio signal and to verify the source of the audio signal.

[0005] One way for the authentication of an audio signal it to analyze its characteristics and search for irregularities which make it suspicious to having been manipulated. Such approaches, however, tend to fail due to the increasing capabilities of Al to generate manipulated signals.

[0006] It is therefore an object of the invention to provide an audio signal authenticator and a method of authenticating an audio signal which enable an easy and robust way to authenticate a variety of different audio signals.

[0007] This object is solved by an audio signal authenticator according to claim 1 and a method of authenticating an audio signal according to claim 10.

[0008] Hence, audio signal authenticator comprising a block bounds detector and block divider configured to receive an audio signal and to output a sequence of a plurality of audio blocks with embedded authentication information or with embedded authentication access information relating to where the authentication information is retrievable, is provided. The authentication information comprises at least audio features belonging to the current audio block. The authenticator also comprises a perceptual similarity analyzer configured to compare the audio blocks of the received audio signal and the audio features of the extracted or retrieved authentication information or to compare audio features extracted from the audio blocks of the received audio signal and the audio features of the extracted or retrieved authentication information to each other providing a similarity result

[0009] The audio features are human auditory perception audio features relating to a human auditory perception.

[0010] In the similarity analyzer the audio blocks can be compared to the audio features of the original audio signal. The audio features can be a true copy of the original audio signal. This will however require large storage space to store the original audio signal. If the audio features relate to other characteristics of the original audio signal, the similarity analyzer should compare the received audio features with the audio features of the audio signal to be authenticated. Hence, the required audio features can be extracted from the audio signal to be authenticated. This can be performed e.g. in the similarity analyzer. In other words, the similarity analyzer should compare the audio features (as received via the authentication access information, i.e. the audio features of the original audio signal) and corresponding audio features of the audio signal to be authenticated.

[0011] When authenticating an audio signal, the audio blocks are analyzed to extract authentication information directly or the extract authentication access information from the audio block. The authentication access information comprises information on where or how the authentication information can be retrieved. Audio features relate to an audio representation trying to capture the relevant aspects for human auditory perception.

[0012] According to an example the human auditory perception audio features are computed by a model mimicking the human auditory perception, in particular a perception model. In addition or alternatively the human auditory perception audio features are a Mel-frequency power spectrum of the audio blocks.

[0013] According to an example the perceptual similarity analyzer is configured to perform soft authentication decisions providing as result a probability for a successful authentication.

[0014] According to an example, the audio signal comprise authentication access information on where or how the authentication information is retrievable. An audio signal is received and a sequence of a plurality of audio blocks with embedded authentication access information is outputted. The authentication access information is extracted from a current audio block and access information on how and / or where signed authentication information belonging to the current block is retrievable from the memory is extracted. Signed authentication information is retrieved from the memory. The signed authentication information comprises a signature and audio features belonging to the current block. The audio block and the audio features are compared to each other to provide a similarity result. The audio features are human auditory perception audio features relating to a human auditory perception.

[0015] Every audio block has specific audio features. Fora robust authentication of an audio signal audio features are required that are insensitive towards typical audio processing like resampling or MP3 or AAC conversions, if a sufficiently high bit rate is employed to enable perceptually transparent coding. Preferably such audio features will not be effected by such audio processing.

[0016] An acoustic stimulus can be transferred into an audio internal representations of a user. In other words, an audio signal received by a user will lead to certain reactions in the auditory system of the user. The audio features of an audio block can relate to such internal representations. Internal representations relate to auditory system specific features of the audio signal. In other words, the internal representation relates to audio features concerning human auditory perception. The human auditory perception audio features can relate to an audio representation trying to capture the relevant aspects for human auditory perception.

[0017] During the generation of an authenticable audio signal, the audio blocks of the audio signal can be analyzed to determine the respective audio features like the respective internal representation, namely human auditory perception audio features.

[0018] Audio features relate to an audio representation of an audio signal trying to capture the relevant aspects for human auditory perception. Another example for audio features could be the output of any model mimicking the human auditory perception.

[0019] One example of the human auditory perception audio feature is a Mel-frequency power spectrum of an audio signal.

[0020] Further examples of the human auditory perception audio features are described by Torsten Dau, Dirk Pueschel, and Armin Kohlrausch. A quantitative model of the "effective" signal processing in the auditory system. I. model structure. The Journal of the Acoustical Society of America, 99(6):3615-3622, 1996, in the following referred to as the perception model (PEMO). This paper describes a quantitative model aimed at describing how the auditory system processes acoustic signals. The focus is on developing a model that mimics the functional processing of auditory stimuli as perceived by humans, particularly in the context of complex sounds like speech or music.

[0021] Key components of the model can include peripheral processing, envelope extraction, modulation filtering and decision stage. The peripheral processing part of the model captures the initial stages of auditory processing, including the transformation of acoustic signals into neural representations through mechanisms such as the outer and middle ear filtering, as well as nonlinearities in the cochlea. The envelope extraction part relates to the extraction of the temporal envelope of sound, which is crucial for understanding amplitude modulation and speech processing. The modulation filtering part relates to a system that filters amplitude modulations, mimicking how the auditory system is sensitive to different modulation frequencies. At the decision stage the processed auditory signal is used to make decisions about the nature of the sound, representing higher-level auditory perception.

[0022] The model is designed to be consistent with experimental psychoacoustic data, particularly in its ability to simulate the auditory system's response to amplitude-modulated sounds. This approach provides a framework for understanding the effective signal processing in human auditory perception, especially for tasks involving the detection and discrimination of complex auditory patterns.

[0023] According to an aspect the authentication access information interpreter is configured to determine identity information of the user or entity who or which has originally signed the audio signal, wherein a public key and an identity of the user can be retrieved from an external database based on the determined identity information.

[0024] According to an aspect a signature verifier is provided which is configured to verify a combination of the signature and the authentication information using the public key and to prove a signature verification result.

[0025] According to an aspect an audio signal comprises a combiner configured to combine the perceptual similarity statement and the verification result to a final authentication result.

[0026] Carrying out the comparison of the received audio blocks and the extracted or retrieved authentication information in a transformed domain (namely regarding the human auditory perception audio features) ensures that the identified differences are perceptually meaningful. In principle any similarity measure, e.g., the empirical cross-correlation coefficient or the relative error with respect to the reference, may be used to map the result of the comparison onto a numerical value.

[0027] When applying further transformations to the Mel-frequency power spectrum of an audio signal, Mel Frequency Cepstral Coefficients (MFCCs) are. MFCCs use the Mel scale, which is a perceptual scale of pitches which serves to approximate the way humans perceive sound. The Mel scale is nonlinear, such that it spaces out lower frequencies more finely and higher frequencies more coarsely, aligning more closely with human auditory perception.

[0028] Mel Frequency Cepstral Coefficients use the cepstrum as a transformation that converts a signal from the frequency domain back into a domain where the signal's rate of change can be analyzed. The idea is to capture the spectral properties of a signal in a way that emphasizes the information-carrying components.

[0029] The process to compute MFCCs from an audio signal can involve several steps: Pre-em- phasis: Boosting the high-frequency components of the signal to balance the spectrum, usually by applying a high-pass filter, Framing: The audio signal is divided into short overlapping frames, typically 20-40 milliseconds long. This is because speech signals are non- stationary, but they can be considered quasi-stationary within these short frames, and Windowing: Each frame is multiplied by a window function, such as a Hamming window, to reduce edge effects and smooth the signal. Fourier Transform: A Fast Fourier Transform (FFT) is applied to each windowed frame to convert the time-domain signal into the frequency domain. Mel Filter Bank: The resulting spectrum is passed through a series of triangular bandpass filters spaced according to the Mel scale. This step approximates how humans perceive sound frequencies. Logarithm: The logarithm of the power of each Mel- filtered spectrum is computed. This simulates the human ear's response to loudness, Discrete Cosine Transform (DCT): Finally, the logarithmic Mel spectrum is transformed using the DCT. The result is a set of coefficients called MFCCs. Typically, only the first 12-13 coefficients are kept, as these contain the most important information.

[0030] MFCCs can capture the broad spectral shape of the audio signal in a compact form, making them highly useful for machine learning algorithms in speech and audio processing. They help in distinguishing different phonemes, speaker identities, and other audio characteristics by providing a representation that is robust to variations in pitch, volume, and other factors.

[0031] The human auditory perception audio feature can also comprise:

[0032] - Linear Predictive Coding (LPC) which is a method that represents the spectral envelope of a digital signal of speech in compressed form, using the information from a linear predictive model. It estimates the parameters of a filter that can be used to recreate the signal. LPCs are highly effective for modeling the formants (resonant frequencies) of speech sounds.

[0033] - Perceptual Linear Predictive (PLP) Coefficients which is similar to Linear Predictive Coding but incorporates aspects of human auditory perception, such as the critical-band spectral resolution, equal-loudness curve, and intensity-loudness power law. Perceptual Linear Predictive (PLP) Coefficients aims to mimic the non-linear perception of loudness and frequency by the human ear.

[0034] - Gammatone filterbank features relate to filterbanks which simulate the filtering that occurs in the human cochlea. They are similarto Mel filterbanks but use gammatone filters instead of triangular filters. The gammatone filterbank features are useful for capturing the detailed frequency structure of audio, particularly in tasks involving environmental sound classification or hearing aid design.

[0035] - Chroma Features, which represent the 12 different pitch classes (C, C#, D, etc.) of a musical octave. They capture harmonic and melodic characteristics of music. These are particularly useful in music information retrieval, key detection, and chord recognition tasks.

[0036] - Mel Spectrogram: While similar to MFCCs, a Mel spectrogram is the result of directly applying the Mel filterbank to a power spectrogram without further transformation (like DCT). The Mel spectrogram retains more detailed frequency information and is often used as input to deep learning models and is used in conjunction with Convolutional Neural Networks (CNNs) in tasks like audio event detection, speech recognition, and music genre classification.

[0037] - Constant-Q Transform (CQT), which is a time-frequency representation that provides a logarithmic frequency scale, similarto the Mel scale, but with a variable time resolution that matches the frequency resolution. It is particularly useful for musical applications because it provides a better representation of musical pitch than the linear FFT or even the Mel scale.

[0038] - Deep Learning-Based Features: Learned features from deep learning models, such as embeddings from a trained neural network, can also be used as an alternative to traditional hand-crafted features like MFCCs. These features are often more robust and can capture complex patterns that are not easily modeled by traditional methods.

[0039] - Spectral Subband Centroids (SSC) which captures the centroid of energy distribution in different subbands of the signal, which can be interpreted as the "center of mass" of the spectrum within each subband. It provides information about the distribution of energy across frequency bands and is sometimes used as a complement to MFCCs.

[0040] - RelAtive SpecTrAI (RASTA) Features which involves filtering the log energy of the speech signal to emphasize modulation frequencies important for speech recognition. RASTA- PLP is a combination of RASTA filtering and PLP analysis, providing robust features against noise and channel variations.

[0041] Audio features relate to an audio representation of an audio signal trying to capture relevant aspects for human auditory perception of the audio signal. The audio features should be selected to reduce the required data rate (for the transmission of the audio features) and hence storage space required on a server. Although the audio features could possibly be the original audio signal itself, preferably the audio features can be a transparently coded version of the original signal requiring much less storage space. Another example for audio features could be the output of any model mimicking the human auditory perception. The audio features can be a Mel-frequency power spectrum of an audio signal with Mel Frequency Cepstral Coefficients (MFCCs).

[0042] In particular the audio features can be acoustical characteristics, spectral characteristics, and statistical characteristics of the audio signal.

[0043] Acoustical features can include pitch (Fundamental Frequency); prosody (Rhythm, Stress, Intonation) and / or duration and silence patterns (inconsistencies in pauses, speech timing, or breathing sounds can be used as audio features Spectral Features can include formant frequencies (like resonant frequencies in speech); Mel-Frequency Cepstral Coefficients (MFCCs) (the short-term power spectrum of sound and can reveal artifacts introduced during synthesis or manipulation); spectrogram analysis (a manipulation may exhibit unusual or smeared energy distribution in the time-frequency representation); and / or phase information.

[0044] Acoustical features can include statistical and signal processing features, like noise and residuals (like subtle background noises or artifacts introduced during synthesis may not match natural recordings); high-frequency content; and / or phase distortions (some manipulations may introduce phase anomalies, which can be detected with signal processing techniques)

[0045] Acoustical features can include temporal features, like jitter and shimmer (like frequency and amplitude variations); temporal coherence (like sudden transitions or changes in speech characteristics, like unnatural breaks in the signal)

[0046] Acoustical features can include behavioral or semantic Inconsistencies, like content coherence (logical inconsistencies or unnatural flow in the spoken content might be indicative of manipulation); emotion and naturalness (The emotional tone may not align with the content, or the voice may lack the natural nuances of human emotion).

[0047] The audio features can be any one of the above mentioned audio characteristics of the audio signal or a combination of them.

[0048] By combining these features with machine learning or signal processing techniques, models can be trained to detect artifacts and inconsistencies indicative of deepfake audio.

[0049] According to an aspect a method of authenticating an audio signal by an authenticator is provided which comprises the steps of receiving an audio signal, outputting a plurality of audio blocks with embedded authentication access information, extracting the authentication access information from the audio blocks indicating where the authentication information belonging to the current audio block is retrievable, retrieving from a memory authentication information comprising a signature, audio features representing the audio block and / or further information and perceptually comparing the audio blocks and the audio features to each other providing a similarity result. The authentication information can be signed with a private key of the microphone or the device performing the capture or post processing of the audio signals to generate a dedicated signature indicative of the source of the authentication information. The private key can be associated with a user of the signing module, e.g. a microphone. At the decoders side a public key can be used to decode or validate the authentication information. In other words, the signature is used to verify the source of the authentication information.

[0050] Hence, by means of the signature it can be verified that the authentication information originates from a certain source that is owner of the private key. By further comparing the audio signal with the authentication information it can be verified that the audio signal has not been manipulated beyond an allowed level. Only a signature is generated based on the private key e.g. of the microphone.

[0051] Storing the authentication information like the metadata not in the audio block but externally is advantageous as the bit rate and the bandwidth required is not increased. Merely, the authentication access information is transmitted together with the audio blocks. If an audio signal is to be authenticated, the authentication access information is extracted and the associated authentication information e.g. the metadata is retrieved.

[0052] The authentication information is thus provided independently of the audio blocks of the captured audio signal. If no authentication is to be performed, the authentication information is not required, and a significant amount of bandwidth and bitrate can be saved compared to the case if the authentication information was attached to the audio signal in an appropriate container format. Furthermore, when an authentication is to be performed, the access information can be extracted from the audio blocks and the authentication information can be retrieved. As the authentication process is not latency critical it is sufficient to extract the authentication information from an external source before performing the authentication.

[0053] The authentication access information can be embedded into the audio block as watermark or in a dedicated container as part of the audio blocks, or, in a file-based context as part of the audio file.

[0054] The authentication information and the authentication access information can be provided for at least one audio block. Hence, each audio block can be individually authenticated. This allows a fine combed authentication. Accordingly, even a slight manipulation of an audio signal (for which the authentication information is available) can be detected. This significantly increases the security of captured audio signals. Providing the authentication information of each audio block is advantageous as it even allows to verify an audio signal comprising different audio blocks from different audio sources. As long as the necessary authentication information is available any audio signal comprising different audio blocks can be validated.

[0055] In the following an example to improve the security of a transmission or storage of the audio signal is described. Audio features of the digital audio signal blocks can be extracted and signatures based on the audio features can be generated using a private key of a person or entity. The captured audio signal and the signed authentication information can be outputted.

[0056] It becomes thus possible to determine whether a transmitted audio signal was captured and signed by a specific microphone or a specific software entity with a specific private key by verifying the combination of the received authentication information and signature using the public key and by subsequently comparing the audio signal with the audio features included in the authentication information.

[0057] According to an aspect of the invention, the microphone also comprises a watermark generator to generate a watermark based on an authentication access information.

[0058] The signing and verification of a signature of the audio blocks can be performed with a private key and public key pair.

[0059] According to an aspect of the invention, in addition to the audio features, the authentication information can comprise a microphone identification, position / location information of the microphone, time and data of the recording and / or a microphone model, sequence number etc.

[0060] Authentication information relates to information that can be used to authenticate a received or selected audio signal. The authentication information enables an authenticator to perform the authentication of an audio signal. The authentication information can comprise information relating to the audio signal or the properties of the audio signal, namely audio features. The authentication information can also comprise metadata relating to information which are independent of the actual audio signal. This information can be an ID of the user, an ID of the microphone used to capture the audio signal, a microphone model, the date, the time, a position or location (GPS position), and / or a sequence number of the audio blocks.

[0061] The authenticity check of the received audio signal can be performed by a decoder which can, for example, be implemented at a cloud service, a computer, a tablet or a smart device.

[0062] The block divider can be used to divide the audio signals into a plurality of audio blocks. The length of the audio blocks can for example be between 0.5s to 20s. It is in particular also possible that the block length varies over time, which might for example be indicated by the watermark. In each audio block, a digital watermark can be embedded. It is also possible to embed the watermark in each nthaudio block.

[0063] The sequence of the audio blocks of the audio signals can be used to associate a sequence number to each audio block. This is advantageous as later on during the authentication of the audio signal it can be determined whether an audio block has been removed from the chain of audio blocks by examining the sequence numbers of the audio blocks. This can also detect a changed order of the original blocks.

[0064] Optionally, metadata of the audio block can be the user of the microphone, the date, the time, a GPS position, etc. Thus, the probability is improved that the result of the authentication is reliable.

[0065] The authentication information can comprise audio features representing the audio signal or audio blocks of the audio signal based on human perception or the original audio signal in a transparently coded version to reduce the storage space required by a server.

[0066] The authentication of the audio signal can be performed by detecting the block boundaries from special information in the watermark or audio container, and extracting the authentication access information (e.g. a link (e.g., consisting of UUID and user based unique block number,) to the block specific authentication information either from the audio watermark or directly from the audio container. The public cryptographic key of the user (e.g. identified by the UUID from link) can be used to verify the authenticity of the authentication information. The audio signal block is compared against the audio features contained in the authentication information to verify its authenticity.

[0067] With the above-described method of authenticating an audio signal it becomes possible to detect malicious modifications of the audio signal by comparing an audio block with the authentication information associated to the audio block. It becomes also possible to detect a modification of the audio signal to be authenticated and / or a modification of the authentication information externally stored by checking the signature of the authentication information. If an audio block of an audio signal is modified and the authentication access information is changed as well, this will be noted by the authentication process as the signature of the authentication information is that of another person or device.

[0068] According to an aspect, the authentication information stored externally can be used to manually authenticate the audio signal orthe audio block if the authenticating process does not give a clear indication of the authenticity of the audio signal or the audio blocks.

[0069] According to an example, the actual audio features extracted from the audio blocks and optional additional metadata are stored in a signed way, e.g. on a server. A unique (unambiguous) link to the metadata stored e.g. on the server is embedded into watermark or into a suitable audio container. The link can comprise a unique user identifier (UUID) in combination with a user based unique block number of the respective audio block. Audio features relate to an audio representation of an audio signal trying to capture the relevant aspects for human auditory perception and at the same time preferably reduces the required data rate and hence storage space required on the server. Although it could possibly be the original audio signal itself, preferably it can be a transparently coded version of the original signal requiring much less storage space. Another example for audio features could be the output of any model mimicking the human auditory perception.

[0070] In order to authenticate the audio signal of an audio block, the block boundaries can be detected from special information in the watermark or audio container. Then the link (e.g., consisting of UUID and user based unique block number,) to the block specific signed authentication information, consisting of the audio features, optional metadata and the signature, is extracted either from the audio watermark or directly from the audio container. By means of a public cryptographic key of the user (e.g., identified by the UUID from link,) the combination of the authentication information and the signature is verified. Accordingly, it is verified whether the authentication information has been signed by the claimed source of the audio signal. The audio signal block is compared against the audio features to verify perceptual similarity, and thus the authenticity of the audio signal block. For the perceptual comparison, audio features can be determined from the received audio signal block, which can be directly compared to the audio features from the signed authentication information stored e.g. on a server. The proposed authentication method is able to prevent any malicious modification of the actual audio signal comparing the audio signal block against the signed authentication information. The authentication might fail in case the comparison allows for some limited amount of modification and the allowed modification threshold is exceeded. Moreover, If together with a malicious modification of the audio signal block the authentication information on the server is altered accordingly, this can be detected by a mismatch of the authentication information and the signature associated to the authentication information. Furthermore, if together with a malicious modification of the audio signal block the link (within the watermark or audio container) is changed to refer to different (but suitable) signed authentication information on the server, the verification of the signature and the subsequent comparison may be successful indeed, but the signature will be that of a different user, assuming that the private key of the original user for the cryptographic signing is inaccessible to the attacker.

[0071] Assuming that the (automated) comparison between the audio signal to be authenticated and the authentication information is carried out in a way to allow some minor modifications (e.g. by perceptual coding) and the result of this comparison is not clear (i.e., no unambiguous statement about the authenticity of the audio signal can be made), the link to the authentication information (in case the contained audio features would correspond to the original audio file or a perceptually coded version of it) could be used for a “manual” perceptual comparison, in order to increase the trust in the method.

[0072] By providing an audio block specific link (e.g., by the UUID and user based unique block number,) allows the advantage (e.g., in the watermarking scenario) that the audio file can be cut at any time instant without the cutting tool being aware of the authentication process and the embedded links. Alternatively, in case an audio container is used for the joint transport of the audio with the link, a file-based link might be once embedded into the contained header to reduce the data rate. It might be composed of the UUID and a position within the meta data (e.g., a sample index of the original file) to indicate how the audio file in question is aligned with the reference on the server.

[0073] These and other aspects of the invention are described in more detail with reference to the following figures.

[0074] Fig. 1 shows a block diagram of an audio signal signing module,

[0075] Fig. 2 shows a diagram of an overall audio signing and authentication workflow, Fig. 3. shows a block diagram of an audio signal authenticator,

[0076] Fig. 4 shows a block diagram of an audio signal signing module,

[0077] Fig. 5 shows a block diagram of an audio signal authenticator,

[0078] Fig. 6 shows a block diagram of an audio signal signing module,

[0079] Fig. 7 shows a block diagram of an audio signal authenticator, and

[0080] Fig. 8 shows an illustration of how to visualize to authentication result.

[0081] According to an example, an authentication is performed for an audio signal. The audio signal is authenticated based on information embedded in the audio blocks of the audio signal. The information embedded in the audio blocks can be authentication information directly embedded into the audio block e.g. be means of watermarks or alternatively authentication access information. The authentication access information contains information on where and how stored authentication information relating to an audio block can be retrieved from a memory, like a server. In this case the authentication information is not directly embedded in the audio blocks.

[0082] In order to better understand the authentication of an audio signal first of all the generation of the authenticable audio signal is described.

[0083] Fig. 1 shows a diagram of an audio signal signing module. The audio signal signing module 500 receives an input audio signal 121 via an audio input 502 from a microphone 100 or another audio source such as an audio recorder and outputs an output audio signal 501 via an audio output 190. The output audio signal 501 can be based on the input audio signal 121. The microphone 100 can comprise at least one microphone capsule 110 and an ana- log-to-digital AD converter 120. The at least one microphone capsule 1 10 can capture an audio signal and can output a captured analogue audio signal 1 11. The captured analogue audio signal 111 can be forwarded to the analog-to-digital AD converter 120 which can digitize the audio signal 11 1 and can output a digital audio signal 121. The audio signal signing module 500 can be implemented as a device separate from the microphone 100 or can be comprised in the microphone 100.

[0084] The audio signal signing module 500 can receive the audio signal 121 from the microphone 100 or from another source, which is indicated in Fig. 1 by the selector 101 . The digital audio signal 121 is inputted to a block divider 130 which divides the audio signal 121 into audio blocks 131 and outputs a sequence of a plurality of audio blocks 131. Optionally, if the audio signal 121 is already divided into audio blocks 131 , the block divider 130 can be omitted. The sequence of audio blocks 131 are forwarded to an audio feature extractor 150 which generates audio features 151 for each audio block 131 in the sequence of audio blocks. A private key signature unit 140 receives authentication information comprising audio features 151 of each audio block 131 togetherwith optional further information 1 13 (e.g. metadata) like location or time and signs the authentication information with the private key 142, providing signed authentication information 141 , which can be stored on a memory 600 e.g. located on an external server. The signed authentication information 141 can comprise the authentication information 152 (see Fig. 2) comprising audio features 151 and optional further information 1 13 and a signature 144 (Fig. 2) generated by the private key signature unit 140. Alternatively, the authentication information 141 can be stored in an internal memory or any other memory, as long as it can be accessed by an authenticator used to authenticate a received audio signal.

[0085] The signed authentication information 141 itself comprises the authentication information, namely (a) audio features 151 , optionally (b) the further information 113 and (c) the signature 144. The individual audio blocks 131 are e.g. in parallel fed to an information embedder 160 for embedding authentication access information 162 e.g. into each audio block 131. The authentication access information 162 comprises information on where and / or how the authentication information 152 (or more precisely the signed authentication information 141), for that particular audio block 131 , can be accessed at the memory 600 e.g. on a server. The embedding can for instance be accomplished by means of watermarking.

[0086] Hence, the information embedder 160 can be implemented as a watermark generator which generates a watermark 163. Another option is to embed the authentication access information 162 into an appropriate audio container next to the actual audio blocks 131. Accordingly, the information embedder 160 outputs audio blocks 161 with embedded authentication access information 162. The output of the embedder 160 corresponds to the output audio signal 501 of the module 500 at the audio output terminal 190. The output audio signal 501 can be broadcasted or stored e.g. via a network 300 (Fig. 2). In particular, it can be even further modified to a little extent, including modification types like sample rate conversion, perceptual compression and cutting.

[0087] In the case that the information embedder 160 is a watermark generator its output 161 could alternatively be input to the audio feature extractor 150 instead of the original audio blocks 131 , which is indicated in Fig. 1 by the selector 102. This alternative offers the advantage that the audio features are extracted based on the final signal 161 that will be output by the audio signal signing module 500 rather than an intermediate signal 131 that is not outputted. As the watermark generator is designed to perceptually alter an audio signal as little as possible, both input options (131 or 161) for the audio feature extractor 150 are reasonable.

[0088] Audio features relate to an audio representation trying to capture the relevant aspects for human auditory perception and at the same time preferably reduces the required data rate and hence storage space required on the server or memory 600. Although it could possibly be the original audio signal itself, preferably it can be a transparently coded version of the original signal requiring much less storage space. Another example for audio features could be the output of any model mimicking the human auditory perception.

[0089] The audio signal 501 (with the embedded watermark or authentication access information) at the output of the audio signal module 500 can be stored or transmitted. As a watermark can be embedded into e. g. substantially each audio block 131 , each audio block 131 can be individually authenticated. The watermark 103 can also be embedded only in some of the audio blocks. The watermark 163 can comprise apart from the authentication access information further information like the block boundaries. This information can be used when authenticating the audio signal to determine a block length.

[0090] An audio watermark 163 can be a distinct identification which is embedded in an audio signal and which is for example previously used for identifying a copyright information of the audio signal. Preferably, the watermark is embedded into the audio signal such that it becomes very difficult to remove or destroy the watermark. Preferably, if the audio signal with the embedded watermark is copied, stored or transmitted, the embedded watermark will not change. The same reasoning applies to authentication access information included in an audio container.

[0091] Fig. 2 shows a diagram of an overall audio signing and authentication workflow. An audio signal is captured e.g. by a microphone 100 outputting a digital audio signal 121 . From this audio signal 121 audio features 151 are extracted and signed with the private key 142 of e.g., audio signal signing module 500. The audio features 151 together with the digital signature 144 generated by the signature unit 140 form the signed authentication information 141. The signed authentication information 141 can be forwarded to a server or memory 600, where the signed authentication information 141 is stored for later retrieval. The information on where and / or how the signed authentication information 141 can be retrieved, namely the authentication access information 162 is provided for the audio signal signing module 500. The audio signal signing module 500 uses this authentication access information 162 and embeds this information into the audio blocks 131 of the audio signal. Hence, the output signal 501 of the audio signal signing module 500 comprises the audio blocks 131 with the embedded authentication access information. The audio blocks 131 of the audio signal with the embedded authentication access information 162 are outputted by the audio signal signing module 500 and can be stored or distributed e.g. through a network 300.

[0092] This distribution can involve a modification or manipulation of either the authentication information 152 and / or the audio signal 121 . The signed authentication information 141 is validated using the public key 143. This validation indicates a potential modification of the audio features. By extracting audio features from the received signal and comparing them with the features contained in the signed authentication information 141 the integrity of the audio signal can be verified. A final authentication result 261 is determined from both checks.

[0093] An audio signal signing module 500 e.g. implemented as part of a microphone 100 is used to capture an audio signal. Alternatively, an audio signal signing module 500 can receive the audio signal. The audio signal can also be an audio signal captured beforehand by other means. As described with reference to Fig. 1 , the captured audio signal is divided into a plurality of audio blocks and the authentication information of at least one audio block is signed based on the private key 142 of the audio signal signing module 500. The signed authentication information 141 is outputted by the audio signal signing module 500 and can be stored on a server or memory 600. Authentication access information 162, regarding information where the authentication information can be retrieved from the server or memory 600 can be provided to the audio signal signing module 500. The authentication access information 162 is embedded into the audio block received or captured by the audio signal signing module 500. Thus, the audio signal signing module 500 outputs a) an output audio signal 501 , which comprises audio blocks with embedded authentication access information to a network 300 for storing or forwarding and b) signed authentication information 141 to the server or memory 600 where this information ca be stored.

[0094] Optionally, the authentication access information can be embedded into the audio blocks of the audio signal by means of a watermark. The watermark can relate to authentication access information relating to a location of a memory 600 where authentication information is stored. This authentication information can be generated by an audio feature extractor 150 as described above with reference to Fig. 1 . The audio blocks of the audio signal with the embedded authentication access information can be transmitted via a network 300 or stored. An authenticator 200 can receive the audio signal 501 with the embedded watermark and extract the authentication access information. The authentication information is retrieved from the server 600 based on the authentication access information. The signature of the authentication information as stored on the server or memory 600 can be validated based on a public key 143 associated to the audio signal signing module 500 which can be stored, for example, on a server 400 or a distributed ledger.

[0095] Fig. 3 shows a block diagram of an audio signal authenticator. The signal authenticator of Fig. 3 can operate with an audio signal signing module of Fig. 1 . The audio signal authenticator 200 receives an audio signal 171 with embedded authentication access information. This audio signal 171 can be the output audio signal 501 from the audio signal signing module 500 of Fig. 1. This audio signal 171 is input into a block bounds detector and block divider 205, which determines block boundaries and outputs a plurality of audio blocks 201 with embedded authentication access information 212. If the authentication access information 212 has been embedded by means of watermarking, it is assumed that the block boundaries have been previously coded into the watermark, from which they can be detected at this stage. In case the audio signal is transported by an audio container that contains the authentication access information 212 as metadata, it is assumed that the authentication access information is available per block with the block boundaries being part of the metadata.

[0096] The audio blocks 201 with embedded authentication access information are input into the access information extractor 210, which extracts the authentication access information 212 (corresponding to the authentication access information 162 from the signing modules if no manipulation has taken place) from the audio blocks 201 . The authentication access information interpreter 230 determines access information from the authentication access information 212 namely, where or how to find the signed authentication information 141 belonging to the current audio block 201 on a server or memory 600. The authentication information 141 retrieved from the server or memory 600 comprise firstly audio features 151 representing the audio block 201 , secondly the signature 221 and thirdly optional further information 113 like location and / or time where and / or when the signing has taken place. The signature 221 corresponds to the signature 144 generated by the signature unit 140 if no manipulation has taken place.

[0097] In a perceptual similarity analyzer 240 the audio blocks 201 and the audio features 151 are perceptually compared to each other, providing a similarity statement 241 . In the similarity analyzer 240 the audio blocks 201 can be compared to the audio features 151 of the original audio signal. The audio features can be a true copy of the original audio signal. This will however require large storage space to store the original audio signal. If the audio features relate to other characteristics of the original audio signal, the similarity analyzer should compare the received audio features with the audio features of the audio signal to be authenticated. Hence, the required audio features can be extracted from the audio signal to be authenticated. This can be performed e.g. in the similarity analyzer 240. In other words, the similarity analyzer 240 should compare the audio features (as received via the authentication access information, i.e. the audio features of the original audio signal) and corresponding audio features of the audio signal to be authenticated.

[0098] In the simplest case this can be a binary decision about whether the compared quantities are perceptually sufficiently equal or not. Alternatively, the similarity statement 241 can be more detailed, proving for instance a similarity score. An authentication access information interpreter 230 can further determine from the authentication access information 212 which owner of the private key 142 has originally signed the audio signal 171 , e.g. represented by an ID. The owner of the private key 142 can be a natural person, an organization, a registered microphone to name only a few possibilities. Using this ID, a public key 143 and the identity 270 of the owner of the private key can be retrieved from an external database. The public key 143 is subsequently employed to verify the combination of the signature 221 and the authentication information 152 (the audio features 151 and the further information 113) within a signature verifier 250, proving a signature verification result 251 . The perceptual similarity statement 241 and the verification result 251 are finally combined in a combiner 260 to a final authentication result 261 .

[0099] Note that the processing that is here proposed for audio content can be similarly applied to any time dependent data. In particular, it can be also applied to video content, if the following modifications are applied, i.e. the microphone 100 is replaced by a video camera, the audio feature extractor 150 is replaced by a video feature extractor, the authentication access information embedding unit 160 uses a visual watermark instead an audio watermark, or uses a video container instead of an audio container and the auditory perceptual comparison 240 is replaced by a visual perceptual comparison.

[0100] Fig. 4 shows a schematic block diagram of an audio signal signing module. The audio signal signing module 500 receives a digital audio signal 121 at its audio input 502, which is inputted to the block divider 130 which outputs a sequence of plurality of audio blocks 131 . The audio blocks 131 are forwarded to an audio feature extractor 150 which generates audio features 151 for each audio block 131 . In a private key signature unit 140, authentication information comprising audio features 151 of each audio block 131 together with optional further information 113 like location or time is signed with the private key 142 of a user or entity, providing signed authentication information 141. The signed authentication information 141 itself consists of the audio features 151 , the generated signature 144 and the optional further information 113. Both components, the individual audio blocks 131 and the signed authentication information 141 are required during the verification of the authentication method shown in Fig. 5. They can be in principle embedded into the same suitable audio container, but can also be distributed over different physical channels. Optionally, a user or entity identifier 145 can be added to both components, which can for instance enable the retrieval of the suitable public key for an automated verification.

[0101] Fig. 5 shows a schematic block diagram of an authenticator 200. The authenticator 200 receives an audio signal e.g. in form of audio blocks 201 which are to be verified as well as signed authentication information 141 . The signed authentication information 141 comprises audio features 151 , further information 113 and the signature 221 . The audio features 151 contained in the authentication information 141 are perceptually compared to the audio blocks 201 within the perceptual similarity analyzer 240, providing a similarity statement 241 . In the simplest case this can be a binary decision about whether the compared quantities are perceptually sufficiently equal or not. Alternatively, the similarity statement 241 can be more detailed, providing for instance a similarity score. Further, with knowledge about the user or entity that created the signature 221 , the corresponding public key 143 and optionally the user identity 270 are obtained. Optionally, this knowledge can be derived from a user or entity identifier 145 provided together with the authentication information 141 and the audio blocks 201 . The public key 143 is subsequently employed to verify the combination of the signature 221 and the authentication information 141 within the signature verification block 250, proving a signature verification result 251 . The perceptual similarity statement 241 and the verification result 251 are finally combined in combiner 260 to a final authentication result 261 .

[0102] In the following Fig. 6 generation of an authenticable audio signal based on embedding authentication information into the audio blocks as well an authentication of the audio signal based on the embedded authentication information is described. This authentication can be used in the authentication of an audio signal as described in the following.

[0103] Fig. 6 shows a diagram of an audio signal signing module. The audio signing module of

[0104] Fig. 6 is merely an example of an audio signing module. The audio signal signing module 500 receives an input audio signal 121 via an audio input 502 from a microphone 100 or another audio source such as an audio recorder and outputs an output audio signal 501 via an audio output 190. The output audio signal 501 can be based on the input audio signal 121. The microphone 100 can comprise at least one microphone capsule 110 and an ana- log-to-digital AD converter 120. The at least one microphone capsule 1 10 can capture an audio signal and can output a captured analogue audio signal 1 11. The captured analogue audio signal 111 can be forwarded to the analog-to-digital AD converter 120 which can digitize the audio signal 111 and can output a digital audio signal 121 .The audio signal signing module 500 can be implemented as a device separate from the microphone 100 or can be comprised in the microphone 100.

[0105] The audio signal signing module 500 can receive the audio signal 121 from the microphone 100 or from another source, which is indicated in Fig. 6 by the selector 101 . The digital audio signal 121 is inputted to a block divider 130 which divides the audio signal 121 into audio blocks 131 each having a block length and outputs a sequence of a plurality of audio blocks 131 . The block divider 130 can divide the audio signal 121 into a plurality of audio blocks. Optionally, the audio blocks each receive a block number, which can be a subsequent block number. Optionally, if the audio signal 121 is already divided into audio blocks 131 , the block divider 130 can be omitted. The sequence of audio blocks 131 are forwarded to an audio feature extractor 150 which generates audio features 151 for each audio block 131 in the sequence of audio blocks 131 . A private key signature unit 140 receives authentication information comprising audio features 151 of each audio block 131 together with optional further information 113 (e.g. metadata) like location or time and signs the authentication information with the private key 142, providing signed authentication information 141. The signed authentication information 141 can comprise the authentication information 152 comprising audio features 151 and optional further information 113 and a signature 144 generated by the private key signature unit 140.

[0106] The signed authentication information 141 itself comprises the authentication information, namely (a) audio features 151 , optionally (b) the further information 113 and (c) the signature 144. The individual audio blocks 131 are e.g. in parallel fed to an information embedder 160 for embedding the signed authentication information 141 e.g. into each audio block 131. The embedding can for instance be accomplished by means of watermarking.

[0107] Hence, the information embedder 160 can be implemented as a watermark generator which generates a watermark 163. Another option is to embed the signed authentication information 141 into an appropriate audio container next to the actual audio blocks 131. Accordingly, the information embedder 160 outputs audio blocks 161 with embedded signed authentication information 141 . The output of the embedder 160 corresponds to the output audio signal 501 of the module 500 at the audio output terminal 190. The output audio signal 501 can be broadcasted or stored e.g. via a network 300 (Fig. 7). In particular, it can be even further modified to a little extent, including modification types like sample rate conversion, perceptual compression and cutting.

[0108] In the case that the information embedder 160 is a watermark generator its output 161 could alternatively be input to the audio feature extractor 150 instead of the original audio blocks 131 by the selector 102. This alternative offers the advantage that the audio features are extracted based on the final signal 161 that will be output by the audio signal signing module 500 rather than an intermediate signal 131 that is not outputted. As the watermark generator is designed to perceptually alter an audio signal as little as possible, both input options (131 or 161) for the audio feature extractor 150 are reasonable.

[0109] The audio signal 501 (with the embedded watermark or authentication access information included in an audio container) at the output of the audio signal module 500 can be stored or transmitted. As a watermark can be embedded into e. g. substantially each audio block 131 , each audio block 131 can be individually authenticated. The watermark 103 can also be embedded only in some of the audio blocks. The watermark 163 can comprise apart from the authentication access information further information like the block boundaries. This information can be used when authenticating the audio signal to determine a block length.

[0110] An audio watermark 163 can be a distinct identification which is embedded in an audio signal and which is for example previously used for identifying a copyright information of the audio signal. Preferably, the watermark is embedded into the audio signal such that it becomes very difficult to remove or destroy the watermark. Preferably, if the audio signal with the embedded watermark is copied, stored or transmitted, the embedded watermark will not change. The same reasoning applies to authentication access information included in an audio container.

[0111] Fig. 7 shows a block diagram of an audio signal authenticator corresponding to the audio signal authenticator of Fig. 6. The audio signal authenticator 200 receives an audio signal 171 with embedded authentication access information. This audio signal 171 can be the output audio signal 501 from the audio signal signing module 500 of Fig. 6. This audio signal 171 is input into a block bounds detector and block divider 205, which determines block boundaries and outputs a plurality of audio blocks 201 with embedded authentication information.

[0112] The block bounds detector and block divider 205 detects the block boundaries e.g. by means of information in the block headers.

[0113] The audio blocks 201 are inputted to an authentication information extractor 206 which extracts signed authentication information 141 embedded in the audio blocks, which can comprise audio features 151 , a signature 144 and thirdly optional further information 1 13 like location and / or time where and / or when the signing has taken place. The signature 221 corresponds to the signature 144 generated by the signature unit 140 if no manipulation has taken place.

[0114] In a perceptual similarity analyzer 240 the audio blocks 201 and the audio features 151 are perceptually compared to each other, providing a similarity statement 241. In the simplest case this can be a binary decision about whether the compared quantities are perceptually sufficiently equal or not. Alternatively, the similarity statement 241 can be more detailed, proving for instance a similarity score.

[0115] An authentication information extractor 206 further determines from the authentication information 141 which owner of the private key 142 has originally signed the audio signal 171 , e.g. represented by an ID. The owner of the private key 142 can be a natural person, an organization, a registered microphone to name only a few possibilities. Using this ID, a public key 143 and the identity 270 of the owner of the private key can be retrieved from an external database. The public key 143 is subsequently employed to verify the combination of the signature 144 and the authentication information 152 (the audio features 151 and the further information 113) within a signature verifier 250, proving a signature verification result 251 . The perceptual similarity statement 241 and the verification result 251 are finally combined in a combiner 260 to a final authentication result 261 .

[0116] Note that the processing that is here proposed for audio content can be similarly applied to any time dependent data. In particular, it can be also applied to video content, if the following modifications are applied, i.e. the microphone 100 is replaced by a video camera, the audio feature extractor 150 is replaced by a video feature extractor, the authentication access information embedding unit 160 uses a visual watermark instead an audio watermark, or uses a video container instead of an audio container and the auditory perceptual comparison 240 is replaced by a visual perceptual comparison. The comparison between the audio blocks 201 and the audio features 151 in Figs. 3, 5 and 7 can be accomplished for instance by extracting the same kind of features from the audio blocks 201 and performing the comparison on the features. Alternatively, a different type of features could be selected for the comparison. In this case, this type of features has to be extracted from the audio blocks 201 and the audio features from the authentication information have to be transformed into this kind of features, before the actual comparison is carried out. The idea is that the type of features available within the authentication information does not necessarily have to correspond to the type of features for the perceptual comparison. In particular, it could be favorable to choose the type of features for the comparison depending on the importance of particular perceptual aspects captured by different kind of features.

[0117] Fig. 8 shows an exemplary illustration of a visualization of an authentication result. In particular, an exemplary original speech signal that is sub-divided into frames on which an individual authentication is foreseen is depicted in the upper part. An exemplary modified signal is displayed the lower part, where the particular modification area is marked by an arrow, where a consonant has been replaced compared to the original speech signal. Below the wave form individual bars are displayed, indicating by color the authentication result, where the color green denotes a positive and the color red a negative authentication result.

[0118] In the perceptual similarity analyzer the received audio blocks are compared to the extracted or retrieved audio features. The audio features are human auditory perception audio features relating to a human auditory perception. Hence, the received audio signal is analyzed to determine its human auditory perception audio features. These audio features are compared to the received or extracted audio features.

[0119] As an example the human auditory perception audio features can be a Mel-frequency power spectrum of the audio blocks.

[0120] Audio features relate to an audio representation of an audio signal trying to capture the relevant aspects for human auditory perception. Another example for audio features could be the output of any model mimicking the human auditory perception.

[0121] Carrying out the comparison of the received audio blocks and the extracted or retrieved authentication information in a transformed domain (namely regarding the human auditory perception audio features) ensures that the identified differences are perceptually meaningful. In principle any similarity measure, e.g., the empirical cross-correlation coefficient or the relative error with respect to the reference, may be used to map the result of the comparison onto a numerical value.

[0122] According to an aspect the comparison between the received audio signal to be authenticated and the audio features from the original audio signal can be based on a soft decision. By mapping the interval of possible similarity values to the interval of possible probabilities (i.e. the interval [0, 1]), such a probability can be directly derived from the similarity values. Such a mapping should be monotonously growing in general, since higher similarity values should imply a higher authentication probability.

[0123] To generate such a mapping:

[0124] 1 . A set of L test signals are collected that are considered to be representative for the target use case.

[0125] 2. A set of representative signal manipulation methods, being composed of a number G of good-natured and a number M of malicious methods are defined.

[0126] 3. Each of the G + M manipulation methods are applied to each of the L test signals and the similarity between the original and modified test signal is computed according to a similarity measure. Hence, a total number of (G + M) x [_ similarity values, (assuming the similarity is computed based on the total length of the individual signals rather than on signal sub- intervals) are provided.

[0127] 4. The whole interval of possible similarity values is discretized into a number N of individual sub-intervals In, n = 1 , . . . , N.

[0128] 5. For each sub-interval In, the number Gn of good-natured and the number Mn of malicious modification executions are counted that deliver similarity values within this sub-interval.

[0129] 6. Using these numbers, the relative frequencies and the desired authentication probabilities Pk for the individual similarity value intervals Ik , k=1 , N by are computed.

[0130] Pk is monotonously growing, as desired, since Hn > 0 and that PN = 1 .

[0131] The proposed soft authentication decision can also be applied to individual audio blocks of the audio signal instead of considering a complete audio signal. Analogously, the example of how to compute the mapping from a similarity value to an authentication probability could be modified in step 3 by computing similarity values for all consecutive audio blocks of the audio signals instead of only for the complete signals. Hence, by correspondingly increasing the number N of sub-intervals to discretize the possible similarity value interval, the resolution of the resulting mapping can be increased.

[0132] An audio block-wise authentication check is advantageous as it allows an exact determination of any manipulated audio blocks. Furthermore, it allows an easy method of visualizing where the manipulation has occurred.

[0133] The modified audio signal (i.e. the audio signals not corresponding to the original audio signal) can be displayed by marking the modified area or the modified audio blocks e.g. by an arrow. Furthermore, the wave form of the audio block can be displayed. To further improve the display of the manipulated audio blocks individual bars can be provided indicating by color the authentication result. The color green can denote a positive and the color red a negative authentication result.

[0134] As an alternative to displaying the authentication decision by two colors (like the colors green and red), the probability of a successful authentication may be displayed using for instance a color scale, where different probabilities are displayed by different colors. A further option is to take into consideration for the visualization that in case of a malicious signal manipulation several consecutive blocks will be affected more likelythan just a single one. Hence, it is reasonable to specially mark the occurrence of several blocks with a negative authentication result. This might happen, e.g., by using a darker shade of the red color every time the length of the presumably modified excerpt exceeds a certain length, e.g. 5 sec. Another option would be to increase the darkness of the red tone with the length of the presumably modified excerpt.

[0135] Additionally, if the metadata contains numbers of the authenticated blocks, any occurrence of non-contiguous block numbers could be specially indicated with the authentication results by e.g. a flash or a broken chain link, thereby hinting to a cut in the original signal.

[0136] An examples of the human auditory perception audio features is described by Torsten Dau, Dirk Pueschel, and Armin Kohlrausch., in “A quantitative model of the "effective" signal processing in the auditory system. I. model structure”, in The Journal of the Acoustical Society of America, 99(6):3615-3622, 1996, in the following referred to as the perception model PEMO. This paper describes a quantitative model aimed at describing how the auditory system processes acoustic signals. The focus is on developing a model that mimics the functional processing of auditory stimuli as perceived by humans, particularly in the context of complex sounds like speech or music.

[0137] Key components of the model can include peripheral processing, envelope extraction, modulation filtering and decision stage. The peripheral processing part of the model captures the initial stages of auditory processing, including the transformation of acoustic signals into neural representations through mechanisms such as the outer and middle ear filtering, as well as nonlinearities in the cochlea. The envelope extraction part relates to the extraction of the temporal envelope of sound, which is crucial for understanding amplitude modulation and speech processing. The modulation filtering part relates to a system that filters amplitude modulations, mimicking how the auditory system is sensitive to different modulation frequencies. At the decision stage the processed auditory signal is used to make decisions about the nature of the sound, representing higher-level auditory perception.

[0138] The model was designed to be consistent with experimental psychoacoustic data, particularly in its ability to simulate the auditory system's response to amplitude-modulated sounds. This approach provides a framework for understanding the effective signal processing in human auditory perception, especially for tasks involving the detection and discrimination of complex auditory patterns.

[0139] This model is applicable to masking conditions with stochastic as well as with deterministic maskers, independent of the temporal relation between masker and signal. As part of the peripheral processing part, the model contains a stage that simulates aspects of temporal adaptation. This part is responsible forthe ability of the model to predict the decay of thresholds in forward masking conditions. The decision stage of the model is realized as an optimal detection process allowing the validity of the preprocessing stages to be assessed. One main property of the model is that it deals with actual time signals and that threshold values are derived using an adaptive tracking procedure in a simulated three interval forced-choice procedure. Therefore, the same stimuli as in psychoacoustic experiments can be used in the simulations, and the results of the optimal processor model can be directly compared with the thresholds obtained by human observers.

[0140] Because detection performance of the model depends on the preprocessing of the signals, the different processing stages in the auditory system and their realization in the model are described more precisely. Sinusoids of different frequencies produce peak responses at different places along the basilar membrane. This frequency-place transformation implies that each place has a bandpass-filter characteristic, which was simulated by a linear basilar-membrane model. This model can be realized as a wave-digital filter and computes the filtered signals in e.g. 120 output segments. As long as broadband noise maskers are used, the use of off-frequency information is not advantageous for the subjects. The signal at the output of the specific basilar-membrane segment was half-wave rectified and low-pass filtered at 1 kHz. This stage roughly simulates the transformation of the mechanical oscillations of the basilar membrane into receptor potentials in the inner hair cells. The low-pass filtering essentially preserves the envelope of the signal for high carrier frequencies. Effects of adaptation were simulated by feedback loops. This stage compresses stationary signals almost logarithmically whereas fast fluctuations of the input are transformed more linearly. These feedback loops were investigated to explain temporal masking effects and they will be described more precisely in the following subsection. In the stage following the feedback loops, the signal was low-pass filtered at 8 Hz, corresponding to a time constant of 20 ms. Since the nonlinear adaptation model combined with the optimal detector process has an important influence on the results. The model incorporates an adaptive properties of the auditory periphery. The adaptation stage in the present model consists of a chain of five feedback loops in series with different time constants. Within each single element, the low- pass filtered output is fed back to form the denominator of the dividing element. The divisor is the momentary charging state of the low-pass filter, determining the attenuation applied to the input. The time constants can range between 5 and 500 ms. This model has the following properties: After the onset of stationary signals the low-pass filters are charged according to their time constants. Because the charging state of the low-pass filter enters as divisor, a greater charging causes a greater attenuation of the input signal. For stationary signals an input value I produces a value of O I at the output of the first feedback loop derived from the stationary condition I / O. For a chain of n loops we obtain an output of O 2n I. For n 5 this approaches a logarithmic transform. The range of derived output values for stationary inputs with levels between 0 and 100 dB was linearly mapped to the range of 0 to 100 model units MU. For a perfect log transformation, a change of 1 MU would correspond to a 1-dB change at the input, irrespective of the input level. With five feedback loops we obtain an input-output characteristic, i.e., a transformation of the input signal level in dB SPL into model units MU. There is a small deviation from a straight line which would represent a logarithm transform on this scale. In the case of an input signal at a low level a certain increment of the input level leads to a somewhat smaller increment at the output than in the case of an input signal at a high level. Since we assume a constant sensitivity in terms of output values MU, this slight deviation from a logarithmic transform predicts a smaller just noticeable change in level at high input levels compared with low input levels. Input variations that are rapid compared with the time constants of the low-pass filters are transformed linearly. If these changes are slow enough to be followed by the charging state of the capacitor, the attenuation gain is also changed. Thus each element combines a static compressive nonlinearity with a higher sensitivity for fast temporal variations. When a signal is turned off the charge of the capacitor, and therefore the attenuation applied to the input, decreases in each stage according to its time constant. Thus the time constants determine how fast the system returns to its state of rest. The charge of each low-pass element decays exponentially, which implies a linear decay on a dB scale. By combining several linear decays with different slopes, a piecewise approximation a forward masking curve can be achieved. In this way a given signal can reduce the response to a signal presented subsequently. This makes it possible to simulate the nonlinear dependence of forward masked threshold functions on masker level and masker duration. With a short masker the stages with the long time constants cannot be fully charged and therefore only the decay due to the fast stages influences the forward-masked thresholds. To establish an absolute threshold for an auditory signal, the minimal value at the input of the adaptation stage is limited by a constant value. This can be interpreted as simulating “physiological noise” that is added before the logarithmic transformation. If the current output of the bas- ilar-membrane-filtered signal is smaller than the threshold it is replaced by the threshold value. In simultaneous masking situations this scaling is of little importance. When simulating mainly at absolute thresholds a stochastic noise instead of our constant threshold value should be used at this point in the model. The calibration of the model is based on the 1- dB criterion in intensity discrimination tasks. In the first step of adjusting the model parameters, this value of a just-noticeable.

[0141] Optionally, the audio signal could be part of a video file.

[0142] Optionally, the signed authentication information can be included in an audio file (e.g., similar to ADM file format) instead of including it into a watermark. In this case the audio signal may not be modified.

[0143] Optionally, the process of generating a signed audio signal could be performed in a software solution based on a pre-existing audio recording or an audio stream. The private key can be entered to the software e.g., by dongle, text input, and authorized e.g., by biometric signal like fingerprint or face detection.

[0144] The authentication process can calculate a similarity score between the audio features of the signed audio signal and the analyzed signal. This could allow to determine a likelihood that the signal is still authentic even if minor signal processing like e.g., gain has happened.

[0145] For authentication the audio signal as well as the signed authentication information is needed. The signed authentication information might be provided via a side channel. In case watermarking is used the authentication access information must be read out from the audio signal and the authentication information must be retrieved accordingly. In case the watermarking readout needs the block boundaries one approach to achieve this is to try out different offsets of block boundaries until a successful watermark readout is possible. Another approach could be to embed some synchronization signal with a watermark into the audio signal.

[0146] To enhance the security of the system a process is provided that allows to withdraw keys e.g., in case they are stolen.

[0147] The public keys for authentication can be provided in various ways. One option is to store all public keys in one centralized database to make it easy to find the needed key. To tackle potential trust issues in this central instance the database with all public keys could be provided on a distributed ledger technology. Another option is that the organization / person that uses the technology provides the public key on their own website.

[0148] In an example, a watermark may comprise a chronological sequence of a pilot sequence (indicating a beginning of a watermark, a user ID and a block ID). If the watermark is detected and can only be partially decoded (e.g. only pilot sequence, but no meaningful user ID or block ID), it is nevertheless reasonable to notify the user accordingly. An advance user (for example a news agency) may deduce from the information that the content was signed at a certain point in time but seems to have been altered afterwards. The partial presence of the watermark could further be used to refute the claims that the file has never been changed.

[0149] In a further example, if authenticating information (providence information) is not bound to the content by a watermark but e.g. via a time-dependent metadata format, there is a possibility that the digital audio workstation references all relevant providence information of all source files with accurate time stamps. In this case, the above described authenticating procedure is not necessary since all information is already available.

[0150] According to an example, each audio signal which is captured by the microphone 100 and which is outputted by the microphone can comprise a watermark by means of which it is possible to determine the actual microphone which has captured the audio signal. If the microphone is used by several users, several user IDs can be associated to the microphone. If the microphone is registered, then it is possible to identify the respective microphone. The user IDs can also be registered. The user IDs can be dedicated IDs or sub IDs of the microphone ID. The microphone ID and / or the user IDs can be part of the watermark. By means of the sequence numbers of the audio blocks, it can be determined when part of the audio signal has been removed.

[0151] Optionally, the audio signal could be part of a video file.

[0152] Optionally, the signed authentication information can be included in an audio file (e.g., similar to ADM file format) instead of including it into a watermark. In this case the audio signal may not be modified.

[0153] Optionally, the process of generating a signed audio signal could be performed in a software solution based on a pre-existing audio recording or an audio stream. The private key can be entered to the software e.g., by dongle, text input, and authorized e.g., by biometric signal like fingerprint or face detection.

[0154] The authentication process can calculate a similarity score between the audio features of the signed audio signal and the analysed signal. This could allow to determine a likelihood that the signal is still authentic even if minor signal processing like e.g., gain has happened.

[0155] For authentication the audio signal as well as the signed authentication information is needed. The signed authentication information might be provided via a side channel. In case watermarking is used the authentication access information must be read out from the audio signal and the authentication information must be retrieved accordingly. In case the watermarking readout needs the block boundaries one approach to achieve this is to try out different offsets of block boundaries until a successful watermark readout is possible. Another approach could be to embed some synchronization signal with a watermark into the audio signal.

[0156] To enhance the security of the system a process is provided that allows to withdraw keys e.g., in case they are stolen.

[0157] The public keys for authentication can be provided in various ways. One option is to store all public keys in one centralized database to make it easy to find the needed key. To tackle potential trust issues in this central instance the database with all public keys could be provided on a distributed ledger technology. Another option is that the organization / person that uses the technology provides the public key on their own website.

[0158] In the following further examples are discussed:

[0159] Example 1 : Audio signal authenticator (200), comprising: a block bounds detector and block divider (205) configured to receive an audio signal (171) and to output a sequence of a plurality of audio blocks (201) with embedded authentication information orwith embedded authentication access information (212, 162) relating to where the authentication information is retrievable, wherein the authentication information (141) comprises at least audio features (151) belonging to the current audio block (201), and a perceptual similarity analyzer (240) configured to compare the audio blocks (201) and the audio features (151) to each other providing a similarity result (241), wherein the audio features are computed by a model mimicking the human auditory perception.

[0160] Example 2: Audio signal authenticator (200) according to example 1 , wherein the model mimicking the human auditory perceptual is the perception model (PEMO).

[0161] Example 3. Audio signal authenticator (200) according to example 1 or 2, wherein the perceptual similarity analyzer (240) is configured to perform soft authentication decisions providing as result a probability for a successful authentication.

[0162] Example 4. Audio signal authenticator (200) according to any one of the example 1 to 3, further comprising: an access information extractor (210) configured to extract the authentication access information (212, 162) from a current audio block (201), an authentication access information interpreter (230) configured to extract access information on how and / or where the signed authentication information (141) belonging to the current audio block (201) is retrievable from a memory (600), wherein the signed authentication information (141) retrieved from the memory (600) comprises a signature (144, 221) and audio features (151) belonging to the current audio block (201).

[0163] Example 5. Audio signal authenticator (200) according to example 4, further comprising: a signature verifier (250) configured to verify a combination of the signature (144, 221) and the human auditory perception audio features (151) using a public key (143) and to provide a signature verification result (251), wherein the public key (143) is associated to a private key (142) that can be used by an audio signal signing module (500) for generating an authenticable audio signal (501).

[0164] Example 6. Audio signal authenticator (200) according to xample 4 or 5, wherein the authentication access information interpreter (230) is configured to determine an identifier (145) of the user or entity who or which has originally signed the audio signal, wherein the public key (143) and an identity (270) of the user or entity can be retrieved from an external database (400) based on the determined identifier (145).

[0165] Example 7. Audio signal authenticator (200) according to example 5, further comprising a combiner (260) configured to combine the similarity result (241) and the signature verification result (251) to a final authentication result (261).

[0166] Example s. Method of authenticating an audio signal (171) by an authenticator (200), comprising the steps of: receiving an audio signal (171), outputting a sequence of a plurality of audio blocks (201) with embedded authentication information or with embedded authentication access information (162) on where the authentication information is retrievable, wherein the authentication information (141) comprising at least audio features (151) belonging to the current audio block (201), and comparing the received audio blocks (201) and extracted or retrieved the audio features (151) to each other providing a similarity result (241), wherein the audio features are computed by a model mimicking the human auditory perception.

[0167] Example 9. Computer program product for authenticating an audio signal, wherein the computer program product comprises program code means causing the audio signal authenticator according to claim 1 to carry out the method of claim 8.

[0168] Example 10. Audio signal authenticator (200) according to any one of the example 1 to 7, wherein the perceptual similarity analyzer (240) is further configured to compute a selected kind of audio features from the audio blocks (201) and wherein the comparison is based on the selected kind of audio features.

[0169] Example 11. Audio signal authenticator (200) according to example 11 , wherein the perceptual similarity analyzer (240) is further configured to transform the audio features within authentication information (151) into the selected kind of audio features.

[0170] Example 12. Audio signal signing module (500), comprising an input configured to receive a digital audio signal (121), a block divider (130) configured to divide the digital audio signal (121) into a sequence of audio blocks (131), wherein the sequence consists of at least one audio block, an audio feature extractor (150) configured to extract audio features (151) from a current audio block (131), a signature unit (140) configured to generate a signature (144) belonging to the current audio block (131) by applying a private key (142) to the audio features (151) and for providing signed authentication information (141), which comprises the audio features (151) and the signature (144), wherein the audio features are computed by a model mimicking the human auditory perception, and wherein the audio features (151) are represented by a format with a lower data rate than the audio blocks (131), and wherein the audio signal signing module (500) outputs the signed authentication information (141) and the audio blocks (131).

[0171] List of reference signs

[0172] 100 microphone

[0173] 101 selector

[0174] 102 selector 110 microphone capsule

[0175] 111 microphone analog audio signal

[0176] 113 metadata

[0177] 120 AD converter

[0178] 121 digital audio signal 130 block divider

[0179] 131 audio blocks

[0180] 140 private key signature unit

[0181] 141 signed authentication information

[0182] 142 private key 143 public key

[0183] 144 signature

[0184] 145 identifier

[0185] 150 audio feature extractor

[0186] 151 audio features 152 authentication information

[0187] 160 information embedder / watermark generator

[0188] 161 audio blocks with embedded information

[0189] 162 authentication access information 163 watermark

[0190] 165 size information

[0191] 171 audio signal

[0192] 190 audio output

[0193] 200 audio signal authenticator 205 block bounds detector and block divider

[0194] 206 authentication information extractor

[0195] 210 access information extractor

[0196] 212 authentication access information

[0197] 221 signature 230 authentication access information interpreter

[0198] 240 perceptual similarity analyzer

[0199] 241 similarity result

[0200] 250 signature verifier

[0201] 251 signature verification result 260 combiner

[0202] 261 authentication result

[0203] 270 identity

[0204] 300 network 400 server

[0205] 500 audio signal signing module

[0206] 501 authenticable audio signal

[0207] 502 audio input

[0208] 600 server or memory

Claims

Claims1 . Audio signal authenticator (200), comprising: a block bounds detector and block divider (205) configured to receive an audio signal (171) and to output a sequence of a plurality of audio blocks (201) with embedded authentication information orwith embedded authentication access information (212, 162) relating to where the authentication information is retrievable, wherein the authentication information (141) comprises audio features (151) belonging to the current audio block (201), and a perceptual similarity analyzer (240) configured to compare the audio blocks (201) of the received audio signal (171) and the audio features (151) of the extracted or retrieved authentication information (141) or to compare audio features extracted from the audio blocks (201) ofthe received audio signal (171) and the audio features (151) ofthe extracted or retrieved authentication information (141) to each other providing a similarity result (241), wherein the audio features are human auditory perception audio features relating to a human auditory perception.

2. Audio signal authenticator (200) according to claim 1 , wherein the human auditory perception audio features are computed by a model mimicking the human auditory perception, in particular a perception model (PEMO).

3. Audio signal authenticator (200) according to clam 1 or 2, wherein the perceptual similarity analyzer (240) is configured to perform soft authentication decisions providing as result a probability for a successful authentication.

4. Audio signal authenticator (200) according to any one of the clams 1 to 3, further comprising: an access information extractor (210) configured to extract the authentication access information (212, 162) from a current audio block (201), an authentication access information interpreter (230) configured to extract access information on how and / or where the signed authentication information (141) belonging to the current audio block (201) is retrievable from a memory (600), wherein the signed authentication information (141) retrieved from the memory (600) comprises a signature (144, 221) and audio features (151) belonging to the current audio block (201).

5. Audio signal authenticator (200) according to claim 4, further comprising:a signature verifier (250) configured to verify a combination of the signature (144, 221) and the human auditory perception audio features (151) using a public key (143) and to provide a signature verification result (251), wherein the public key (143) is associated to a private key (142) that can be used by an audio signal signing module (500) for generating an authenticable audio signal (501).

6. Audio signal authenticator (200) according to claim 4 or 5, wherein the authentication access information interpreter (230) is configured to determine an identifier (145) of the user or entity who or which has originally signed the audio signal, wherein the public key (143) and an identity (270) of the user or entity can be retrieved from an external database (400) based on the determined identifier (145).

7. Audio signal authenticator (200) according to claim 5 or 6, further comprising a combiner (260) configured to combine the similarity result (241) and the signature verification result (251) to a final authentication result (261).

8. Audio signal authenticator (200) according to any one of the clams 1 to 7, wherein the perceptual similarity analyzer (240) is further configured to compute audio features from the audio blocks (201) of the received audio signal (171) and wherein the comparison is based on the selected kind of audio features.

9. Audio signal authenticator (200) according to claim 8, wherein the perceptual similarity analyzer (240) is further configured to transform the audio features within authentication information (151) into the selected kind of audio features.

10. Method of authenticating an audio signal (171) by an authenticator (200), comprising the steps of: receiving an audio signal (171), outputting a sequence of a plurality of audio blocks (201) with embedded authentication information or with embedded authentication access information (162) on where the authentication information is retrievable, wherein the authentication information (141) comprising at least audio features (151) belonging to the current audio block (201), and comparing the audio blocks (201) of the received audio signal (171) and extracted or retrieved the audio features (151) to each other providing a similarity result (241), orcomparing audio features extracted from the audio blocks (201) of the received audio signal (171) and the retrieved audio features (151) to each other providing a similarity result (241), wherein the audio features are human auditory perception audio features relating to a human auditory perception.11 . Audio signal signing module (500), comprising an input configured to receive a digital audio signal (121), a block divider (130) configured to divide the digital audio signal (121) into a sequence of audio blocks (131), wherein the sequence consists of at least one audio block, an audio feature extractor (150) configured to extract audio features (151) from a current audio block (131), a signature unit (140) configured to generate a signature (144) belonging to the current audio block (131) by applying a private key (142) to the audio features (151) and for providing signed authentication information (141), which comprises the audio features (151) and the signature (144), wherein the audio features are computed by a model mimicking a human auditory perception, and wherein the audio features (151) are represented by a format with a lower data rate than the audio blocks (131), and wherein the audio signal signing module (500) outputs the signed authentication information.

12. Computer program product for authenticating an audio signal, wherein the computer program product comprises program code means causing the audio signal authenticator according to claim 1 to carry out the method of claim 10.

Images