Audio signal signing module and a method of generating an authenticable audio signal

Abstract

An audio signal signing module (500) is provided, which comprises an input (502) configured to receive a digital audio signal (121) and a block divider (130) configured to divide the received digital audio signal (121) into a sequence of audio blocks (131) having a block length. A signature unit (140) generates a signature (144) associated to the current audio block (131) by applying a private key (142) to extracted audio features (151). The combined information is provided as signed authentication information (141) and stored in a memory (600) at a predetermined memory location. The predetermined memory location is made available as authentication access information (162) and embedded into the current audio block (131) by an information embedder. An audio output (190) provides an authenticable audio signal (501) comprising a sequence of the audio blocks with embedded authentication access information (162). The block divider (130) adapts the block length of the audio block depending on the size the authenticating access information be embedded into the audio block.

Description

[0001] Audio signal signing module and a method of generating an authenticable audio signal

[0002] The present invention relates to an audio signal signing module, a microphone and a method of generating an authenticable audio signal, an audio signal authenticator and a method of authenticating an audio signal.

[0003] In view of modern capabilities of creating deepfake video and deepfake audio, the need has risen to be able to verify or authenticate an audio signal like a speech of a politician etc.

[0004] Until now it has been very difficult to authenticate an audio signal of a person. Typically, a lot of manual research must be done in order to verify or authenticate such an audio signal and to verify the source of the audio signal.

[0005] It is therefore an object of the invention to provide means to enable a generation of an authenticable audio signal and means to authenticate such an audio signal which enable an easy and robust way to authenticate an audio signal.

[0006] This object is solved by an audio signal signing module according to claim 1 , a microphone according to claim 5 and a method of generating an authenticable audio signal according to claim 6 as well as an audio signal authenticator according to claim 8, a method of authenticating an audio signal according to claim 12, and computer program products according to claims 14 and 15.

[0007] The invention relates to a generation of an authenticable audio signal by means of an audio signal signing module or a method of generating an authenticable audio signal and to an authentication of an audio signal by means of an authenticator or a method of authenticating an audio signal. An audio signal must be processed by the audio signal signing module to include information which can enable a later authentication. At the authentication stage, a received or stored audio signal is authenticated by an authenticator based on the information associated to the audio signal.

[0008] Hence, an audio signal signing module is provided, which comprises an input for receiving a digital signal, a block divider for dividing the received digital audio signal into a sequence of a plurality of audio blocks each having a block length having a block length and a block specific payload capacity, an audio feature extractor for extracting audio features from a current audio block and a signature unit for generating a signature associated to a current audio block by applying a private key to the audio features. The signature unit also provides signed authentication information which comprises the extracted audio features and the signature. The audio signal signing module also comprises an information embedder for generating an audio block with embedded information by embedding authentication access information into the current audio block. Furthermore, an audio output is provided for out- putting an authenticable audio signal comprising a sequence of audio blocks with embedded authentication access information. The block divider adapts the block length of the audio block depending on the size the authenticating access information to be embedded into the audio block to be able to accommodate the authenticating access information into the audio block without creating (significant) audible artefacts in the audio signal. Thus, the block lengths of the audio blocks can be varied according to the requirements of the information to be embedded. This also allows an easy adaptation of the block length in view of a trade-off between short block lengths for a temporally more fine-grained verification and long block lengths to embed authentication access information.

[0009] According to an aspect the information embedder is implemented as a watermark generator to introduce the authentication access information as a watermark into the current audio block such that the authenticable audio signal comprises a sequence of audio blocks with embedded watermarks.

[0010] According to an aspect the at least one watermark has a specific size, and the block generator adapts the block length of the audio block according to the specific size of the at least one watermark.

[0011] According to an aspect the block divider analyses the audio characteristics of audio blocks to determine for each audio block a block specific payload capacity and adapting the length of the audio block such that the block specific payload capacity surpasses the size of the watermark to be embedded.

[0012] The amount of information that can be embedded into an audio block by means of watermarking (without creating significant acoustical artefacts) is here referred to as the block specific (watermarking) payload capacity, which is dependent on the signal characteristics of the audio block. In order to be able to embed watermarks of a given size into audio blocks, the block length can be adapted. For example, if the block specific payload capacity of an audio block is lower than the size of the watermark to be embedded, the block length can be increased appropriately, thereby causing the required increase of the block specific payload capacity.

[0013] It should be further noted that the block-specific payload capacity depends on the specified robustness of the watermark against manipulation, i.e. processing steps to which the signal is intended to be subjected after watermarking, e.g. perceptual coding at a certain bit-rate. In general, the demand for greater robustness goes hand in hand with a decrease of the block specific payload capacity.

[0014] To make watermarking a feasible option for authenticating an audio signal, the data rate of the information to be embedded should be - at least partly - below the watermarking payload capacity rate, which is the average quantity of information per time that can be inserted into the audio signal while maintaining audible imperceptibility. Watermarking typically exploits effects in the human perception such as temporal and frequency masking to maintain imperceptibility. In particular, the actual amount of information that can be embedded per time highly depends on the short-term signal characteristics, and thus is usually highly time varying.

[0015] Assuming a uniform block length overtime for the above-mentioned authentication method causes the problem that for individual blocks the data size for the authentication access information may be too large compared to what may be embedded into the watermark, rendering the watermarking concept infeasible. Hence to remedy this, the block length can be adapted according to the watermarking requirements. The block length of an individual audio block or the block length of a group of audio blocks can be adapted according to the watermarking requirements. The block divider can analyse the audio characteristics to determine how much watermarking payload can be introduced and adapt the block length accordingly.

[0016] There is a trade-off between the idea to use short blocks for the authenticating procedure, which are in the order of a low single digit number of seconds and the need to include watermarks into the audio blocks.

[0017] The adaptation of the block length can be performed by selecting the smallest pre-defined block length before analysing the signal properties of the audio block to determine if an embedding of the required information, e.g., the link to the block-specific authentication information is possible. If the condition is not satisfied, the block length of the considered audio block is increased and the test for the feasibility of embedding is carried out once again. These two steps are repeated until an embedding is feasible. In this case, the actual embedding is accomplished and the next audio block is processed. The information about the eventually used block length has to be prepended (possibly in a coded way) to the embedded data, in order to enable correct watermark decoding and thereby correct reconstruction of the block length for authentication.

[0018] According to an aspect, the signed authentication information further comprises metadata belonging to the current block. The metadata can comprise at least one of an identifier of the user or entity who or which has signed the audio signal, a location and a time in particular of the signing of the audio signal or the generation of the audio signal.

[0019] The invention also relates to a microphone comprising a microphone capsule adapted to capture an audio signal, and an audio signal signing module as described above.

[0020] The invention also relates to a method of generating an authenticable audio signal. A digital audio signal is received and the digital audio signal is divided into a sequence of a plurality of audio blocks each having a block length by a block divider. Audio features are extracted from a current audio block and a signature associated to the current audio block is generated by applying a private key to the extracted audio features. Signed authentication information is provided comprising the audio features and the signature. An audio block with embedded information is generated by embedding authentication access information into the current block. The authenticable audio signal is outputted. The authenticable audio signal comprises a sequence of a plurality of audio blocks with embedded authentication access information. The block divider adapts the block length of the audio block depending on the size the authenticating access information to be embedded into the audio block.

[0021] According to an advantageous embodiment the method comprises analyzing the payload capacity of an audio block to determine whether the payload capacity is sufficient to embed the authentication access information and increasing the block length only if the payload capacity is not sufficient to embed the authentication access information. In many cases it turns out that a preselected block length is sufficient and no further steps need to be implemented.

[0022] The invention also relates to an audio signal authenticator which comprises a block boundary detector and block divider for receiving an audio signal, detecting block boundaries or a block length of the audio blocks and to output a sequence of a plurality of audio blocks with embedded authentication access information. An authentication access information extractor is provided configured to extract authentication access information from the audio blocks. The extracted authentication access information comprises a signature and audio features belonging to the current audio block. Furthermore, perceptional similarity analyzer is provided for comparing the audio blocks and the audio features to each other and providing a similarity result.

[0023] According to an aspect, a signature verifier is provided to verify a combination of the signature and the audio features using a public key and to provide a signature verification result. The public key is associated to a private key that can be used by an audio signal signing module for generating an authenticable audio signal.

[0024] According to an aspect, a combiner is provided to combine the similarity result and the signature verification result to a final authentication result.

[0025] In the similarity analyzer the audio blocks can be compared to the audio features of the original audio signal. The audio features can be a true copy of the original audio signal. This will however require large storage space to store the original audio signal. If the audio features relate to other characteristics of the original audio signal, the similarity analyzer should compare the received audio features with the audio features of the audio signal to be authenticated. Hence, the required audio features can be extracted from the audio signal to be authenticated. This can be performed e.g. in the similarity analyzer. In other words, the similarity analyzer should compare the audio features (as received via the authentication access information, i.e. the audio features of the original audio signal) and corresponding audio features of the audio signal to be authenticated.

[0026] Audio features relate to an audio representation trying to capture the relevant aspects for human auditory perception and at the same time preferably reduces the required data rate and hence storage space required on a server. Although it could possibly be the original audio signal itself, preferably it can be a transparently coded version of the original signal requiring much less storage space. Another example for audio features could be the output of any model mimicking the human auditory perception.

[0027] According to an aspect a method of authenticating an audio signal by an authenticator is provided which comprises the steps of receiving an audio signal, outputting a plurality of audio blocks with embedded authentication access information, extracting the authentica- tion access information from the audio blocks indicating where the authentication information belonging to the current audio block is retrievable, retrieving from a memory authentication information comprising a signature, audio features representing the audio block and / or further information and perceptually comparing the audio blocks and the audio features to each other providing a similarity result.

[0028] The authentication information is signed with a private key of the microphone or the device performing the capture or post processing of the audio signals to generate a dedicated signature indicative of the source of the authentication information. The private key can be associated with a user of the signing module, e.g. a microphone. At the decoder side a public key can be used to decode or validate the authentication information. In otherwords, the signature is used to verify the source of the authentication information.

[0029] Hence, by means of the signature it can be verified that the authentication information originates from a certain source that is owner of the private key. By further comparing the audio signal with the authentication information it can be verified that the audio signal has not been manipulated beyond an allowed level. Only a signature is generated based on the private key e.g. of the microphone.

[0030] Storing the authentication information like the audio features not in the audio block but externally is advantageous as the bit rate and the bandwidth required is not increased. Merely, the authentication access information is transmitted together with the audio blocks. If an audio signal is to be authenticated, the authentication access information is extracted and the associated authentication information e.g. the audio features are retrieved.

[0031] The authentication information is thus provided independently of the audio blocks of the captured audio signal. If no authentication is to be performed, the authentication information is not required, and a significant amount of bandwidth and bitrate can be saved compared to the case if the authentication information was attached to the audio signal in an appropriate container format. Furthermore, when an authentication is to be performed, the access information can be extracted from the audio blocks and the authentication information can be retrieved. As the authentication process is not latency critical it is sufficient to extract the authentication information from an external source before performing the authentication. The authentication access information can be embedded into the audio block as watermark or in a dedicated container as part of the audio blocks, or, in a file-based context as part of the audio file.

[0032] The authentication information and the authentication access information can be provided for at least one audio block. Hence, each audio block can be individually authenticated. This allows a fine combed authentication. Accordingly, even a slight manipulation of an audio signal (for which the authentication information is available) can be detected. This significantly increases the security of captured audio signals. Providing the authentication information of each audio block is advantageous as it even allows to verify an audio signal comprising different audio blocks from different audio sources. As long as the necessary authentication information is available any audio signal comprising different audio blocks can be validated.

[0033] Audio features of the digital audio signal blocks can be extracted and signatures based on the audio features can be generated using a private key of a person or entity. The captured audio signal and the signed authentication information can be outputted.

[0034] It becomes thus possible to determine whether a transmitted audio signal was captured and signed by a specific microphone or a specific software entity with a specific private key by verifying the combination of the received authentication information and signature using the public key and by subsequently comparing the audio signal with the audio features included in the authentication information.

[0035] According to example, a microphone is provided that comprises a microphone capsule and an audio signal signing module as described above.

[0036] The signing and verification of a signature of the audio blocks can be performed with a private key and public key pair.

[0037] According to an aspect of the invention, in addition to the audio features, the authentication information can comprise a microphone identification, position / location information of the microphone, time and data of the recording and / or a microphone model, sequence number etc.

[0038] Authentication information relates to information that can be used to authenticate a received or selected audio signal. The authentication information enables an authenticator to perform the authentication of an audio signal. The authentication information can comprise information relating to the audio signal or the properties of the audio signal, namely audio features. The authentication information can also comprise metadata relating to information which are independent of the actual audio signal. This information can be an ID of the user, an ID of the microphone used to capture the audio signal, a microphone model, the date, the time, a position or location (GPS position), and / or a sequence number of the audio blocks.

[0039] The authenticity check of the received audio signal can be performed by a decoder which can, for example, be implemented at a cloud service, a computer, a tablet or a smart device.

[0040] The block divider can be used to divide the audio signals into a plurality of audio blocks. The length of the audio blocks can for example be between 0.5s to 20s. It is in particular also possible that the block length varies over time, which might for example be indicated by the watermark. In each audio block, a digital watermark can be embedded. It is also possible to embed the watermark in each nthaudio block.

[0041] The sequence of the audio blocks of the audio signals can be used to associate a sequence number to each audio block. This is advantageous as later on during the authentication of the audio signal it can be determined whether an audio block has been removed from the chain of audio blocks by examining the sequence numbers of the audio blocks. This can also detect a changed order of the original blocks.

[0042] Optionally, metadata of the audio block can be the user of the microphone, the date, the time, a GPS position, etc. Thus, the probability is improved that the result of the authentication is reliable.

[0043] The authentication information can comprise audio features representing the audio signal or audio blocks of the audio signal based on human perception or the original audio signal in a transparently coded version to reduce the storage space required by a server.

[0044] The authentication of the audio signal can be performed by detecting the block boundaries from special information in the watermark or audio container and extracting the authentication access information (e.g. a link (e.g., consisting of UUID and user based unique block number,) to the block specific authentication information either from the audio watermark or directly from the audio container. The public cryptographic key of the user (e.g. identified by the UUID from link) can be used to verify the authenticity of the authentication information. The audio signal block is compared against the audio features contained in the authentication information to verify its authenticity.

[0045] With the above-described method of authenticating an audio signal it becomes possible to detect malicious modifications of the audio signal by comparing an audio block with the authentication information associated to the audio block. It becomes also possible to detect a modification of the audio signal to be authenticated and / or a modification of the authentication information externally stored by checking the signature of the authentication information. If an audio block of an audio signal is modified and the authentication access information is changed as well, this will be noted by the authentication process as the signature of the authentication information is that of another person or device.

[0046] According to an aspect, the authentication information stored externally can be used to manually authenticate the audio signal orthe audio block if the authenticating process does not give a clear indication of the authenticity of the audio signal or the audio blocks.

[0047] According to an example, the actual audio features extracted from the audio blocks and optional additional metadata are stored in a signed way, e.g. on a server. A unique (unambiguous) link to the metadata stored e.g. on the server is embedded into watermark or into a suitable audio container. The link can comprise a unique user identifier (UUID) in combination with a user based unique block number of the respective audio block. Audio features relate to an audio representation of an audio signal trying to capture the relevant aspects for human auditory perception and at the same time preferably reduces the required data rate and hence storage space required on the server. Although it could possibly be the original audio signal itself, preferably it can be a transparently coded version of the original signal requiring much less storage space. Another example for audio features could be the output of any model mimicking the human auditory perception.

[0048] In order to authenticate the audio signal of an audio block, the block boundaries can be detected from special information in the watermark or audio container. Then the link (e.g., consisting of UUID and user based unique block number,) to the block specific signed authentication information, consisting of the audio features, optional metadata and the signature, is extracted either from the audio watermark or directly from the audio container. By means of a public cryptographic key of the user (e.g., identified by the UUID from link,) the combination of the authentication information and the signature is verified. Accordingly, it is verified whether the authentication information has been signed by the claimed source of the audio signal. The audio signal block is compared against the audio features to verify perceptual similarity, and thus the authenticity of the audio signal block. For the perceptual comparison, audio features can be determined from the received audio signal block, which can be directly compared to the audio features from the signed authentication information stored e.g. on a server.

[0049] The proposed authentication method is able to prevent any malicious modification of the actual audio signal comparing the audio signal block against the signed authentication information. The authentication might fail in case the comparison allows for some limited amount of modification and the allowed modification threshold is exceeded. Moreover, If together with a malicious modification of the audio signal block the authentication information on the server is altered accordingly, this can be detected by a mismatch of the authentication information and the signature associated to the authentication information. Furthermore, if together with a malicious modification of the audio signal block the link (within the watermark or audio container) is changed to refer to different (but suitable) signed authentication information on the server, the verification of the signature and the subsequent comparison may be successful indeed, but the signature will be that of a different user, assuming that the private key of the original user for the cryptographic signing is inaccessible to the attacker.

[0050] Assuming that the (automated) comparison between the audio signal to be authenticated and the authentication information is carried out in a way to allow some minor modifications (e.g. by perceptual coding) and the result of this comparison is not clear (i.e., no unambiguous statement about the authenticity of the audio signal can be made), the link to the authentication information (in case the contained audio features would correspond to the original audio file or a perceptually coded version of it) could be used for a “manual” perceptual comparison, in order to increase the trust in the method.

[0051] By providing an audio block specific link (e.g., by the UUID and user based unique block number,) allows the advantage (e.g., in the watermarking scenario) that the audio file can be cut at any time instant without the cutting tool being aware of the authentication process and the embedded links. Alternatively, in case an audio container is used for the joint transport of the audio with the link, a file-based link might be once embedded into the contained header to reduce the data rate. It might be composed of the UUID and a position within the meta data (e.g., a sample index of the original file) to indicate how the audio file in question is aligned with the reference on the server. The invention also relates to a computer program product for generating an authenticable audio signal. The computer program product comprises program code, which, when executed by a processor, causes an above described audio signal signing module to carry out the above described method for generating an authenticable audio signal.

[0052] The invention also relates to a computer program product for authenticating an audio signal. The computer program product comprises program code which, when executed by a processor, causes the above described audio signal authenticator to carry out the method of authenticating an audio signal.

[0053] These and other aspects of the invention are described in more detail with reference to the following figures.

[0054] Fig. 1 shows a block diagram of an audio signal signing module,

[0055] Fig. 2 shows a flow chart of a procedure to adapt the block length of the audio blocks, and

[0056] Fig. 3 shows a block diagram of an audio signal authenticator.

[0057] Fig. 1 shows a diagram of an audio signal signing module 500 receiving a digital audio signal 121 from a microphone 100 or another audio source such as an audio recorder (not shown). A selector switch 101 can selectively forward the audio signal 121 either from the microphone 100 or from the other audio source to an input 502 of the audio signal signing module 500. The audio signal signing module 500 outputs an output audio signal 501 via an audio output 190. The output audio signal 501 is based on the input audio signal 121 .

[0058] The microphone 100 comprises at least one microphone capsule 110 and an analog-to- digital AD converter 120. The at least one microphone capsule 110 is configured to capture an audio signal and outputs an analogue output audio signal 11 1 . The analogue output audio signal 11 1 is transmitted to an analog-to-digital AD converter 120 converting the analogue audio output signal 11 1 into a digital audio signal 121 .

[0059] In one embodiment the audio signal signing module 500 is integrated in the microphone 100. In an alternative embodiment the audio signal signing module 500 is implemented as a device separate from the microphone 100.

[0060] The digital audio signal 121 is inputted to a block divider 130 which divides the audio signal 121 into audio blocks 131 and outputs a sequence of audio blocks 131. Each audio block 131 has a specific block length. The sequence of audio blocks 131 are forwarded to an audio feature extractor 150 which generates audio features 151 for each audio block 131 in the sequence of audio blocks 131 . A private key signature unit 140 receives authentication information comprising audio features 151 of each audio block 131 together with optional further information 113 (e.g. metadata), like location or time, and signs the authentication information with a private key 142 to generate signed authentication information 141 , which can be stored in a memory 600 implemented as a storage device e.g. as an external server. Alternatively, the signed authentication information 141 can be stored in an internal memory or any other memory, provided it can be accessed by an authenticator used to authenticate a received audio signal.

[0061] In all cases, a coded representation of the memory location where the signed authentication information 141 of the currently analyzed audio block 131 will be stored within the memory 600 can be already made available before the corresponding audio block 131 is processed in the audio feature extractor 150. Such a coded representation may consist of a unique user identifier and a user dependent block number which facilitates an unambiguous identification of the memory location. The actual mapping between the coded and the actual memory location can be accomplished by a special server that acts similar to a DNS server, which maps names of websites to IP addresses. In particular, the authentication access information may be a coded representation of the memory location. In Fig. 1 , it is assumed that the private key signature unit 140 has knowledge about the above mentioned coded representation of the memory location, with which it generates corresponding authentication access information 162.

[0062] The authentication access information 162 has a specific size, which is essentially independent of the size of the corresponding / associated audio block but may nevertheless exceed the amount of information that can be integrated as watermark in the analyzed audio block 131. The authentication access information 162 is forwarded to an information em- bedder 160, which attempts to embed the authentication access information 162 as a watermark into the current audio block 131. In case the current payload capacity of the analyzed audio block 131 is not sufficient to allow the attempted embedding, the block divider 130 is requested to increase the size of the current audio block to a desired block size 165 and the attempt to embed the authentication access information 162 as a watermark into the current audio block 131 with the increased size 165 is repeated. These two steps, namely the incremental increase of the block size and the attempt to embed the watermark are carried out until the embedding is successful. Optionally, the finally selected block size 165 for the current audio block can be coded into the watermark additionally to the authentication access information in order to allow a simpler decoding.

[0063] The signed authentication information 141 itself comprises the authentication information, namely (a) audio features 151 , optionally (b) the further information 113 and (c) the signature 144. The individual audio blocks 131 are simultaneously provided to the information embedder 160 for embedding authentication access information 162 and to audio feature extractor 150.

[0064] In an alternative embodiment the information embedder 160 embeds the authentication access information 162 into an appropriate audio container next to the actual audio blocks 131 . Regardless, whetherthe authentication access information 162 is stored as watermark or in an audio container, the information embedder 160 outputs audio blocks 161 with embedded authentication access information 162. The output of the embedder 160 corresponds to the output audio signal 501 of the module 500 at the audio output terminal 190. The output audio signal 501 can be transmitted or stored. In particular, it can even be processed furtherto achieve sample rate conversion, perceptual compression and cutting. The embedded authentication access information 162 is not or at least only to a small extent influenced by the further processing.

[0065] In the case that the information embedder 160 embeds watermarks, its output 501 could alternatively be input to the audio feature extractor 150 instead of the original audio blocks 131 , which is indicated in Fig. 1 by the selector 102. This alternative offers the advantage that the audio features are extracted based on the final signal 501 that will be output by the audio signal signing module 500 rather than being based on an intermediate signal 131 that is not outputted. As the information embedder 160 is designed to perceptually alter an audio signal as little as possible, both input options (131 or 501) for the audio feature extractor 150 are reasonable.

[0066] Audio features relate to an audio representation trying to capture the relevant aspects for human auditory perception and at the same time preferably reduces the required data rate and hence storage space required on the server or memory 600. Although it could possibly be the original audio signal itself, preferably it can be a transparently coded version of the original signal requiring much less storage space. Another example for audio features could be the output of any model mimicking the human auditory perception. The audio signal 501 (with the embedded watermark or authentication access information included in an audio container) at the output of the audio signal module 500 can be stored or transmitted. As a watermark 163 can be embedded into e. g. substantially each audio block 131 , each audio block 131 can be individually authenticated. The watermark 163 can also be embedded only in some of the audio blocks. The watermark 163 can comprise apart from the authentication access information further information like the block boundaries. This information about block boundaries is useful for determining an adapted block length, which needs to be known when authenticating the audio signal.

[0067] In order to make watermarking a feasible option for authenticating an audio signal, the amount of data of the information to be embedded should be below the watermarking payload capacity rate, which is the average quantity of information per time that can be inserted into the audio signal while maintaining imperceptibility. Watermarking typically exploits effects in the human perception such as temporal and frequency masking to maintain imperceptibility. In particular, the actual amount of information that can be embedded per time highly depends on the short-term signal characteristics, and thus is usually highly time varying. Assuming a uniform block length over time for the above-mentioned authentication method causes the problem that for individual blocks the data size for the metadata may be too large compared to what may be embedded into the watermark, rendering the watermarking concept infeasible.

[0068] Hence, according to an example the block length is adapted to the watermarking requirements. The block length of an individual audio block or the block length of a group of audio blocks can be adapted according to the watermarking requirements. The block divider can analyze the audio characteristics to determine how much watermarking payload can be introduced.

[0069] There is a trade-off between the idea to use short blocks for the authenticating procedure, which are in the order of a low single digit number of seconds and the need to include watermarks into the audio blocks. It is a further advantage of the above-described example that a complete authentication of audio compositions that have been composed of various audio file excerpts is now possible.

[0070] According to an example, the block length is varied in dependence on the block specific watermarking payload capacity that can be embedded into the audio signal. The block divider 130 can also be able to analyze the input audio signal to determine audio characteristics of the audio signal that are relevant for determining the available watermark payload, i.e. the amount of data that can be embedded into an audio block. Alternatively, the audio feature extractor can be used to analyze the audio features of the audio signal.

[0071] Audio features relate to an audio representation trying to capture the relevant aspects for human auditory perception and at the same time preferably reduces the required data rate and hence storage space required on the server or memory 600. Although it could possibly be the original audio signal itself, preferably it can be a transparently coded version of the original signal requiring much less storage space. Another example for audio features could be the output of any model mimicking the human auditory perception.

[0072] Preferably, the watermark is embedded into the audio signal such that it becomes very difficult to remove or destroy the watermark. Preferably, if the audio signal with the embedded watermark is copied, stored or transmitted, the embedded watermark will not change. The same reasoning applies to authentication access information included in an audio container.

[0073] Fig. 2 shows a flow chart of a procedure to adapt the block length of the audio blocks. The flow starts with step S1 , where the smallest block length is selected for the audio blocks by the block divider 130. In step S2 the payload capacity for the watermarking is analysed for the selected block length. In other words, the capacity of an audio block with its block length to embed a watermark (i.e. the payload capacity) is determined. Optionally, the determination of the block specific payload capacity is based on a predetermined robustness of the watermark against manipulation. If the block length is not sufficient then in step S3 the block length is increased. If the block length is sufficient then the flow continues to step S4, where the block divider uses the selected block length at least for this specific audio block. In step S5 the watermark is embedded into the audio block with the specific block length.

[0074] This block length adaption can be used for each audio block. Alternatively, the block length can be determined for a set of blocks where the size of the watermark is expected not to be changed.

[0075] First, the smallest pre-defined block length is selected before analysing the signal properties if they allow the embedding of the required information, e.g., the link to the block-specific metadata as mentioned above. If the condition is not satisfied, the length of the considered audio block is increased and the test for the feasibility of embedding is carried out once again. These two steps are repeated until an embedding is feasible. In this case, the actual embedding is accomplished, and the next signal portion is processed. The information about the eventually used block length has to be prepended (possibly in a coded way) to the embedded data, in order to enable correct watermark decoding and thereby correct reconstruction of the block length for authentication. Usually, the audio signal properties are unknown a priori, such that a minimum watermarking payload capacity cannot be guaranteed. Hence, it is reasonable to code the block length (at least partially) with a variable bit rate into the watermark.

[0076] The reason why the proposed method works in the described context is that, on the one hand, the size of the data to be embedded does not (or only barely) change with the increase of the block length, but on the other hand, the increase of the block length increases the block-specific watermarking payload capacity for a successful embedding.

[0077] Fig. 3 shows a block diagram of an audio signal authenticator 200. The audio signal authenticator 200 receives an audio signal 171 with embedded authentication access information. The audio signal 171 is for instance the output audio signal 501 from the audio signal signing module 500 of Fig. 1. This audio signal 171 is input into a block boundary detector and block divider 205, which outputs a plurality of audio blocks 201 with embedded authentication access information 212.

[0078] The block boundary detector and block divider 205 detects the block boundaries e.g. by means of information in the block headers. As the block lengths of the audio blocks can vary, the block lengths should be determined for each audio block or for a group of audio blocks having the same block length.

[0079] If the authentication access information has been embedded by means of watermarking, it is assumed that the block boundaries have been previously coded into the watermark, from which they can be detected at this stage. In case the audio signal is transported by an audio container that contains the authentication access information as metadata, it is assumed that the authentication access information is available per block with the block boundaries being part of the metadata.

[0080] The audio blocks 201 with embedded authentication access information are input into the access information extractor 210, which extracts authentication access information 212 (corresponding to the authentication access information 162 from the signing modules if no manipulation has taken place) from the audio blocks 201 . The authentication access information interpreter 230 determines access information from the authentication access information 212 namely, where to find the signed authentication information 141 belonging to the current audio block 201 on a server or memory 600. The authentication information 141 retrieved from the server or memory 600 comprise firstly audio features 151 representing the audio block 201 , secondly the signature 144 and thirdly optional further information 113 like location and / or time where and / or when the signing has taken place.

[0081] In a perceptual similarity analyzer 240 the audio blocks 201 and the audio features 151 are perceptually compared to each other, providing a similarity statement 241 .

[0082] The similarity analyzer 240 the audio blocks 201 can be compared to the audio features 151 of the original audio signal. The audio features can be a true copy of the original audio signal. This will however require large storage space to store the original audio signal. If the audio features relate to other characteristics of the original audio signal, the similarity analyzer should compare the received audio features with the audio features of the audio signal to be authenticated. Hence, the required audio features can be extracted from the audio signal to be authenticated. This can be performed e.g. in the similarity analyzer 240. In other words, the similarity analyzer 240 should compare the audio features (as received via the authentication access information, i.e. the audio features of the original audio signal) and corresponding audio features of the audio signal to be authenticated.

[0083] In the simplest case this can be a binary decision about whether the compared quantities are perceptually sufficiently equal or not. Alternatively, the similarity statement 241 can be more detailed, proving for instance a similarity score.

[0084] An authentication access information interpreter 230 further determines from the authentication access information 212 which owner of the private key 142 has originally signed the audio signal 171 , e.g. represented by an ID. The owner of the private key 142 can be a natural person, an organization, or a registered microphone to name only a few possibilities. Using this ID, a public key 143 and the identity 270 of the owner of the private key can be retrieved from an external database 400. Optionally, the external database is a distributed ledger. The public key 143 is subsequently employed to verify the signature 144 within a signature verifier 250 to produce a signature verification result 251 . The signature verification result can take on two states, e.g. true or false. The perceptual similarity statement 241 and the verification result 251 are finally combined in a combiner 260 to a final authentication result 261 . It is noted that the processing that is here proposed for audio content can be similarly applied to any time dependent data. In particular, it can be also applied to video content, if the following modifications are applied, i.e. the microphone 100 is replaced by a video camera, the audio feature extractor 150 is replaced by a video feature extractor, the authentication access information embedding unit 160 uses a visual watermark instead an audio watermark, or uses a video container instead of an audio container and the auditory perceptual comparison 240 is replaced by a visual perceptual comparison. Instead of repeatedly attempting to embed the information access information 162 as watermark into the current audio block 131 and increasing the block size 165 in case the embedding fails, the required block length might be directly determined by analyzing the size of a potential watermark payload that might be embedded for different sizes of the current audio block 131 and selecting the smallest block size that allows a successful embedding of the required information access information.

[0085] According to an example, each audio signal which is captured by the microphone 100 and which is outputted by the microphone can comprise a watermark by means of which it is possible to determine the actual microphone which has captured the audio signal. If the microphone is used by several users, several user IDs can be associated to the microphone. If the microphone is registered, then it is possible to identify the respective microphone. The user IDs can also be registered. The user IDs can be dedicated IDs or sub IDs of the microphone ID. The microphone ID and / or the user IDs can be part of the watermark. By means of the sequence numbers of the audio blocks, it can be determined when part of the audio signal has been removed.

[0086] Optionally, the audio signal could be part of a video file.

[0087] Optionally, the signed authentication information can be included in an audio file (e.g., similar to ADM file format) instead of including it into a watermark. In this case the audio signal may not be modified.

[0088] Optionally, the process of generating a signed audio signal could be performed in a software solution based on a pre-existing audio recording or an audio stream. The private key can be entered to the software e.g., by dongle, text input, and authorized e.g., by biometric signal like fingerprint or face detection. The authentication process can calculate a similarity score between the audio features of the signed audio signal and the analyzed signal. This could allow to determine a likelihood that the signal is still authentic even if minor signal processing like e.g., gain has happened.

[0089] For authentication the audio signal as well as the signed authentication information is needed. The signed authentication information might be provided via a side channel. In case watermarking is used the authentication access information must be read out from the audio signal and the authentication information must be retrieved accordingly. In case the watermarking readout needs the block boundaries one approach to achieve this is to try out different offsets of block boundaries until a successful watermark readout is possible. Another approach could be to embed some synchronization signal with a watermark into the audio signal.

[0090] To enhance the security of the system a process is provided that allows to withdraw keys e.g., in case they are stolen.

[0091] The public keys for authentication can be provided in various ways. One option is to store all public keys in one centralized database to make it easy to find the needed key. To tackle potential trust issues in this central instance the database with all public keys could be provided on a distributed ledger technology. Another option is that the organization / person that uses the technology provides the public key on their own website.

[0092] Further examples of the audio features are described by Torsten Dau, Dirk Pueschel, and Armin Kohlrausch. A quantitative model of the "effective" signal processing in the auditory system. I. model structure. The Journal of the Acoustical Society of America, 99(6):3615- 3622, 1996, in the following referred to as the perception model (PEMO). This paper describes a quantitative model aimed at describing how the auditory system processes acoustic signals. The focus is on developing a model that mimics the functional processing of auditory stimuli as perceived by humans, particularly in the context of complex sounds like speech or music.

[0093] Key components of the model can include peripheral processing, envelope extraction, modulation filtering and decision stage. The peripheral processing part of the model captures the initial stages of auditory processing, including the transformation of acoustic signals into neural representations through mechanisms such as the outer and middle ear filtering, as well as nonlinearities in the cochlea. The envelope extraction part relates to the extraction of the temporal envelope of sound, which is crucial for understanding amplitude modulation and speech processing. The modulation filtering part relates to a system that filters amplitude modulations, mimicking how the auditory system is sensitive to different modulation frequencies. At the decision stage the processed auditory signal is used to make decisions about the nature of the sound, representing higher-level auditory perception.

[0094] The model was designed to be consistent with experimental psychoacoustic data, particularly in its ability to simulate the auditory system's response to amplitude-modulated sounds. This approach provides a framework for understanding the effective signal processing in human auditory perception, especially for tasks involving the detection and discrimination of complex auditory patterns.

[0095] Carrying out the comparison of the received audio blocks and the extracted or retrieved authentication information in a transformed domain (namely regarding the human auditory perception audio features) ensures that the identified differences are perceptually meaningful. In principle any similarity measure, e.g., the empirical cross-correlation coefficient or the relative error with respect to the reference, may be used to map the result of the comparison onto a numerical value.

[0096] When applying further transformations to the Mel-frequency power spectrum of an audio signal, Mel Frequency Cepstral Coefficients (MFCCs) are. MFCCs use the Mel scale, which is a perceptual scale of pitches which serves to approximate the way humans perceive sound. The Mel scale is nonlinear, such that it spaces out lower frequencies more finely and higher frequencies more coarsely, aligning more closely with human auditory perception.

[0097] Mel Frequency Cepstral Coefficients use the cepstrum as a transformation that converts a signal from the frequency domain back into a domain where the signal's rate of change can be analyzed. The idea is to capture the spectral properties of a signal in a way that emphasizes the information-carrying components.

[0098] The process to compute MFCCs from an audio signal can involve several steps: Pre-em- phasis: Boosting the high-frequency components of the signal to balance the spectrum, usually by applying a high-pass filter, Framing: The audio signal is divided into short overlapping frames, typically 20-40 milliseconds long. This is because speech signals are non- stationary, but they can be considered quasi-stationary within these short frames, and Windowing: Each frame is multiplied by a window function, such as a Hamming window, to reduce edge effects and smooth the signal. Fourier Transform: A Fast Fourier Transform (FFT) is applied to each windowed frame to convert the time-domain signal into the frequency domain. Mel Filter Bank: The resulting spectrum is passed through a series of triangular bandpass filters spaced according to the Mel scale. This step approximates how humans perceive sound frequencies. Logarithm: The logarithm of the power of each Mel- filtered spectrum is computed. This simulates the human ear's response to loudness, Discrete Cosine Transform (DCT): Finally, the logarithmic Mel spectrum is transformed using the DCT. The result is a set of coefficients called MFCCs. Typically, only the first 12-13 coefficients are kept, as these contain the most important information.

[0099] MFCCs can capture the broad spectral shape of the audio signal in a compact form, making them highly useful for machine learning algorithms in speech and audio processing. They help in distinguishing different phonemes, speaker identities, and other audio characteristics by providing a representation that is robust to variations in pitch, volume, and other factors.

[0100] The perception audio feature can also comprise:

[0101] - Linear Predictive Coding (LPC) which is a method that represents the spectral envelope of a digital signal of speech in compressed form, using the information from a linear predictive model. It estimates the parameters of a filter that can be used to recreate the signal. LPCs are highly effective for modeling the formants (resonant frequencies) of speech sounds.

[0102] - Perceptual Linear Predictive (PLP) Coefficients which is similar to Linear Predictive Coding but incorporates aspects of human auditory perception, such as the critical-band spectral resolution, equal-loudness curve, and intensity-loudness power law. Perceptual Linear Predictive (PLP) Coefficients aims to mimic the non-linear perception of loudness and frequency by the human ear.

[0103] - Gammatone Filterbank Features relate to filterbanks which simulate the filtering that occurs in the human cochlea. They are similar to Mel filterbanks but use Gammatone filters instead of triangular filters. The Gammatone Filterbank Features are useful for capturing the detailed frequency structure of audio, particularly in tasks involving environmental sound classification or hearing aid design. - Chroma Features, which represent the 12 different pitch classes (C, C#, D, etc.) of a musical octave. They capture harmonic and melodic characteristics of music. These are particularly useful in music information retrieval, key detection, and chord recognition tasks.

[0104] - Mel Spectrogram: While similar to MFCCs, a Mel spectrogram is the result of directly applying the Mel filterbank to a power spectrogram without further transformation (like DCT). The Mel spectrogram retains more detailed frequency information and is often used as input to deep learning models and is used in conjunction with Convolutional Neural Networks (CNNs) in tasks like audio event detection, speech recognition, and music genre classification.

[0105] - Constant-Q Transform (CQT), which is a time-frequency representation that provides a logarithmic frequency scale, similar to the Mel scale, but with a variable time resolution that matches the frequency resolution. It is particularly useful for musical applications because it provides a better representation of musical pitch than the linear FFT or even the Mel scale.

[0106] - Deep Learning-Based Features: Learned features from deep learning models, such as embeddings from a trained neural network, can also be used as an alternative to traditional hand-crafted features like MFCCs. These features are often more robust and can capture complex patterns that are not easily modeled by traditional methods.

[0107] - Spectral Subband Centroids (SSC) which captures the centroid of energy distribution in different subbands of the signal, which can be interpreted as the "center of mass" of the spectrum within each subband. It provides information about the distribution of energy across frequency bands and is sometimes used as a complement to MFCCs.

[0108] - RelAtive SpecTrAI (RASTA) Features which involves filtering the log energy of the speech signal to emphasize modulation frequencies important for speech recognition. RASTA- PLP is a combination of RASTA filtering and PLP analysis, providing robust features against noise and channel variations.

[0109] Audio features relate to an audio representation of an audio signal trying to capture relevant aspects for human auditory perception of the audio signal. The audio features should be selected to reduce the required data rate (for the transmission of the audio features) and hence storage space required on a server. Although the audio features could possibly be the original audio signal itself, preferably the audio features can be a transparently coded version of the original signal requiring much less storage space. Another example for audio features could be the output of any model mimicking the human auditory perception. The audio features can be a Mel-frequency power spectrum of an audio signal with Mel Frequency Cepstral Coefficients (MFCCs).

[0110] In particular the audio features can be acoustical characteristics, spectral characteristics, and statistical characteristics of the audio signal.

[0111] Acoustical features can include pitch (Fundamental Frequency); prosody (Rhythm, Stress, Intonation) and / or duration and silence patterns (inconsistencies in pauses, speech timing, or breathing sounds can be used as audio features

[0112] Spectral Features can include formant frequencies (like resonant frequencies in speech); Mel-Frequency Cepstral Coefficients (MFCCs) (the short-term power spectrum of sound and can reveal artifacts introduced during synthesis or manipulation); spectrogram analysis (a manipulation may exhibit unusual or smeared energy distribution in the time-frequency representation); and / or phase information.

[0113] Acoustical features can include statistical and signal processing features, like noise and residuals (like subtle background noises or artifacts introduced during synthesis may not match natural recordings); high-frequency content; and / or phase distortions (some manipulations may introduce phase anomalies, which can be detected with signal processing techniques)

[0114] Acoustical features can include temporal features, like jitter and shimmer (like frequency and amplitude variations); temporal coherence (like sudden transitions or changes in speech characteristics, like unnatural breaks in the signal)

[0115] Acoustical features can include behavioral or semantic Inconsistencies, like content coherence (logical inconsistencies or unnatural flow in the spoken content might be indicative of manipulation); emotion and naturalness (The emotional tone may not align with the content, or the voice may lack the natural nuances of human emotion).

[0116] The audio features can be any one of the above mentioned audio characteristics of the audio signal or a combination of them. By combining these features with machine learning or signal processing techniques, models can be trained to detect artifacts and inconsistencies indicative of deepfake audio.

[0117] List of reference signs

[0118] 100 microphone

[0119] 101 selector switch

[0120] 102 selector 110 microphone capsule

[0121] 111 microphone analog audio signal

[0122] 113 metadata

[0123] 120 AD converter

[0124] 121 digital audio signal 130 block divider

[0125] 131 audio blocks

[0126] 140 private key signature unit

[0127] 141 signed authentication information

[0128] 142 private key 143 public key

[0129] 144 signature

[0130] 150 audio feature extractor

[0131] 151 audio features

[0132] 160 information embedder / watermark generator 161 audio blocks with embedded information

[0133] 162 authentication access information

[0134] 163 watermark

[0135] 165 audio block size 171 audio signal

[0136] 190 audio output

[0137] 200 audio signal authenticator

[0138] 205 block bounds detector and block divider

[0139] 210 authentication access information extractor 212 authentication access information

[0140] 230 authentication access information interpreter

[0141] 240 perceptual similarity analyzer

[0142] 241 similarity result

[0143] 250 signature verifier 251 signature verification result

[0144] 260 combiner

[0145] 261 authentication result

[0146] 270 identity

[0147] 400 external data base audio signal signing module authenticable audio signal audio input server or memory

Claims

Claims1 . Audio signal signing module (500), comprising- an input (502) configured to receive a digital audio signal (121),- a block divider (130) configured to divide the received digital audio signal (121) into a sequence of audio blocks (131) having a block length and a block specific payload capacity,- an audio feature extractor (150) configured to extract audio features (151) from a current audio block (131),- a signature unit (140) configured to generate a signature (144) associated to the current audio block (131) by applying a private key (142) to the extracted audio features (151) and configured to provide signed authentication information (141), which comprises the extracted audio features (151) and the signature (144), wherein the signed authentication information (141) is outputted to a memory (600),- an information embedder (160) configured to generate an audio block (131) with embedded information (161) by embedding authentication access information (162) into the current audio block (131), wherein the authentication access information (162) comprises information on how and / or where the signed authentication information (141) is retrievable from the memory (600), and- an audio output (190) configured to output an authenticable audio signal (501) comprising a sequence of the audio blocks (131) with embedded authentication access information (162), wherein the block divider (130) is configured to adapt the block length of the audio block (131) depending on the block specific payload capacity and the size of the authenticating access information (162) to be embedded into the audio block (131).

2. Audio signal signing module (500) according to claim 1 , wherein the information embedder (160) is configured to introduce the authentication access information (162) as a watermark (163) into the current audio block (131) such that the authenticable audio signal (501) consists of a sequence of audio blocks with embedded watermarks (163).

3. Audio signal signing module (500) according to claim 1 or 2, wherein the signed authentication information (141) further comprises metadata (113) belonging to the current audio block (131), wherein the metadata (113) comprises at least one of: an identifier (145) of the user or entity who or which has signed the audio signal, a location, and a time.

4. Audio signal signing module (500) according to claim 1 , 2 or 3, wherein the information embedder (160) is further configured to determine the block specific payload capacity of the current audio block (131) to decide whether the block specific payload capacity is sufficient to embed the authentication access information (161) and to increase the block length, if the block specific payload capacity is not sufficient to embed the authentication access information.

5. Microphone (100), comprising:- a microphone capsule (110) adapted to capture an audio signal (111),- an AD converter (120) configured to convert the captured audio signal (111) into a digital audio signal (121), and- an audio signal signing module (500) according to claim 1 , 2, 3 or 4.

6. Method of generating an authenticable audio signal (501), comprising the steps of:- receiving a digital audio signal (121),- dividing the digital audio signal (121) into a sequence of a plurality of audio blocks (131) by a block divider (130), wherein each audio block (131) has a block length and a block specific payload capacity,- extracting audio features (151) from a current audio block (131),- generating a signature (144) associated to the current audio block (131) by applying a private key (142) to the extracted audio features (151),- providing signed authentication information (141), which comprises the audio features (151) and the signature (144),- outputting the signed authentication information (141) to a memory (600),- generating an audio block (131) with embedded information by embedding authentication access information (162) into the current audio block (131), wherein the authentication access information (162) comprises information on how and / or where the signed authentication information (141) is retrievable from the memory (600),- outputting the authenticable audio signal (501) comprising of a sequence of audio blocks with embedded authentication access information (161), and- adapting the block length of the audio block (131) depending on the block specific payload capacity and the size of the authenticating access information to be embedded into the audio block (131).

7. Method according to claim 6, comprisingdetermining the block specific payload capacity of the current audio block (131) to decide whether the block specific payload capacity is sufficient to embed the authentication access information and increasing the block length if the block specific payload capacity is not sufficient to embed the authentication access information.

8. Audio signal authenticator (200), comprising:- a block bounds detector and block divider (205) configured to receive an audio signal (171) and to output a sequence of a plurality of audio blocks (201) with embedded authentication access information (212, 162),- an access information extractor (210) configured to extract the authentication access information (212, 162) from a current audio block (201),- an authentication access information interpreter (230) configured to extract access information on how and / or where the signed authentication information (141) belonging to the current audio block (201) is retrievable from a memory (600), wherein the signed authentication information (141) retrieved from the memory (600) comprises a signature (144) and audio features (151) belonging to the current audio block (201), and- a perceptual similarity analyzer (240) configured to compare the audio blocks (201) of the received audio signal and the retrieved audio features (151) or to compare audio features extracted from the audio blocks (201) of the received audio signal and the retrieved audio features (151) to each other providing a similarity result (241).

9. Audio signal authenticator (200) according to claim 8, further comprising:- a signature verifier (250) configured to verify a combination of the signature (144) and the audio features (151) using a public key (143) and to provide a signature verification result (251), wherein the public key (143) is associated to a private key (142) that can be used by an audio signal signing module (500) for generating an authenticable audio signal (501).

10. Audio signal authenticator (200) according to claim 9, wherein the authentication access information interpreter (230) is configured to determine an identifier of the user or entity who or which has originally signed the audio signal, wherein the public key (143) and an identity (270) of the user or entity can be retrieved from an external database (400) based on the determined identifier.11 . Audio signal authenticator (200) according to claim 9 or 10, further comprising a combiner (260) configured to combine the similarity result (241) and the signature verification result (251) to a final authentication result (261).

12. Method of authenticating an audio signal (171) by an authenticator (200), comprising the steps of:- receiving an audio signal (171),- outputting a sequence of a plurality of audio blocks (201) with embedded authentication access information (162),- extracting the authentication access information (212, 162) from a current audio block (201),- extracting access information (212, 162) on how and / or where signed authentication information (141) belonging to the current audio block (201) is retrievable from a memory (600),- retrieving from the memory (600) signed authentication information (141) comprising a signature (144) and audio features (151) belonging to the current audio block (201), and-comparing the audio blocks (201) of the received audio signal and the retrieved audio features (151) to each other providing a similarity result (241), or- comparing audio features extracted from the audio blocks (201) of the received audio signal (171) and the retrieved audio features (151) to each other providing a similarity result (241).

13. Method according to claim 12, further comprising the steps of:- verifying a combination of the signature (144) and the audio features (151) using a public key (143), and- providing a signature verification result (251), wherein the public key (143) is associated to a private key (142) that can be used by an audio signal signing module (500) for generating an authenticable audio signal (501).

14. Computer program product for generating an authenticable audio signal, wherein the computer program product comprises program code, which, when executed by a processor, causes an audio signal signing module according to claim 1 to carry out the method according to claims 5 to 7.

15. Computer program product for authenticating an audio signal,wherein the computer program product comprises program code which, when executed by a processor, causes the audio signal authenticator according to claim 8 to carry out the method of claim 12.

Images