Audio watermark for indicating post-processing

Abstract

A system for avoiding double processing using audio watermarking. A decoder inserts an audio watermark during a transient in an audio signal. This avoids the drawbacks of using out-of-band control signals or metadata. The decoder performs: detecting a transient in a first audio signal; transforming a portion related to the transient into a frequency domain to compare a first frequency band of the frequency domain data to a second frequency band of the frequency domain data; when the first frequency band is not related to the second frequency band, the decoder performs processing on the first audio data to generate second audio data; when the first frequency band is related to the second frequency band, the first audio data is used as the second audio data without performing any processing.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to European Patent Application No. 20177393.4, filed May 29, 2020, U.S. Provisional Patent Application No. 63 / 027,286, filed May 19, 2020, and PCT Application No. PCT / CN2020 / 088816, filed May 6, 2020, each of which is incorporated by reference herein in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates to audio processing, and in particular to using audio watermarks to indicate audio processing. BACKGROUND

[0004] Unless otherwise indicated herein, the methods described in this section are not prior art to the claims in this application, and are not admitted to be prior art by virtue of their inclusion in this section.

[0005] Media players, including those implemented in mobile phones, are becoming more and more configurable. Media players can include various decoders, pre-processors, post-processors, etc. that perform various types of audio processing (e.g., according to user selections, preferences, machine learning, etc.).

[0006] The selected type of audio processing can be performed by components at more than one point in the audio processing chain. This leads to the possibility that more than one component can perform the processing, leading to double processing of the audio. One problem with double processing is that it consumes additional resources (power, processor cycles, battery life, etc.), which is particularly undesirable in mobile devices. Another problem with double processing is that the double-processed audio can have perceptible differences from the (single) processed audio, leading to a negative user experience.

[0007] One way to avoid double processing is to communicate between components that the processing has already been performed. Such communication can be via control signals, control messages, metadata, etc. SUMMARY

[0008] One problem with using control signals, control messages, metadata, etc., for communication between components is that this communication must comply with the inter-component communication requirements of the mobile device's operating system. For example, for security purposes, the operating system may not allow communication to be passed directly between components, but may instead require communication to be intermediated by a security component. This involves additional effort in many ways. First, in addition to focusing on audio processing, audio component developers need to maintain expertise in operating system security. Second, if the operating system modifies its security system, even if it does not affect the operational details of audio processing, audio component developers need to update the audio processing component to comply. As a concrete example, in Android... TM In the operating system, due to Android TM The architecture design prevents audio metadata from being directly accessed through Android. TM Audio chain.

[0009] Given the above, it is necessary to transmit information about dual processing between audio processing components in a manner different from using control signals, control messages, metadata, etc. This article describes techniques related to using audio watermarking to detect dual processing.

[0010] According to an embodiment, an audio processing method includes detecting transients in first audio data by a processing unit. The method further includes transforming a transient-related portion of the first audio data into frequency domain data. The method further includes comparing a first frequency band of the frequency domain data with a second frequency band of the frequency domain data. When the first frequency band and the second frequency band are uncorrelated, the method further includes performing processing on the first audio data by the processing unit to generate second audio data. When the first frequency band and the second frequency band are correlated, the method further includes using the first audio data as the second audio data without performing processing by the processing unit. In this way, the method uses a detected audio watermark to determine whether to perform processing.

[0011] Audio watermarks can be inserted as follows. Before detecting transients in the first audio data (see above), the method further includes decoding the third audio data by a decoder component to generate fourth audio data. The method further includes detecting transients in the fourth audio data, wherein the transients in the fourth audio data correspond to transients in the first audio data. The method further includes transforming a first portion of the fourth audio data associated with the transients in the fourth audio data into first frequency domain data. The method further includes copying a first frequency band of the first frequency domain data into a second frequency band of the first frequency domain data to generate second frequency domain data. The method further includes transforming the second frequency domain data to generate a second portion. The method further includes generating fifth audio data, wherein the fifth audio data corresponds to the fourth audio data in which the first portion is replaced by the second portion. (The fifth audio data corresponds to the first audio data discussed above.)

[0012] According to another embodiment, an apparatus for audio processing includes a processor and a memory. The processor is configured to control the apparatus to perform one or more of the method steps discussed above. The apparatus may additionally include details similar to those of one or more of the methods described herein.

[0013] According to another embodiment, a non-transitory computer-readable medium stores a computer program that, when executed by a processor, controls means to perform processing including one or more of the methods described herein.

[0014] The following detailed description and accompanying drawings provide a further understanding of the nature and advantages of the various embodiments. Attached Figure Description

[0015] Figure 1 This is a block diagram of mobile device 100.

[0016] Figure 2 This is a block diagram of the audio processing framework 200.

[0017] Figure 3 This is a block diagram of decoder component 300.

[0018] Figure 4 This is a block diagram of the processing component 400.

[0019] Figures 5A-5B It is a graph illustrating the spectral copy of the audio watermark.

[0020] Figure 6 This is a flowchart of audio processing method 600. Detailed Implementation

[0021] This document describes techniques related to audio watermarking. In the following description, numerous examples and specific details are set forth for illustrative purposes in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure, as defined by the claims, may include some or all of these examples, either alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0022] The following description details various methods, processes, and procedures. While specific steps may be described in a particular order, this order is primarily for convenience and clarity. A particular step may be performed more than once, may occur before or after other steps (even if these steps are described in a different order), and may occur in parallel with other steps. A second step is only necessary if the first step must be completed before the second step begins. This will be specifically indicated when it is unclear from the context.

[0023] In this document, the terms “and,” “or,” and “and / or” are used. These terms will be understood to have inclusive meanings. For example, “A and B” can at least mean: “both A and B,” or “at least both A and B.” As another example, “A or B” can at least mean: “at least A,” “at least B,” “both A and B,” or “at least both A and B.” As yet another example, “A and / or B” can at least mean: “A and B,” or “A or B.” When intended as XOR, this will be specified (e.g., “either A or B,” or “at most one of A and B”).

[0024] Figure 1 This is a block diagram of mobile device 100. Mobile device 100 can be a media player (e.g., an MP3 player, iPod). TM Devices, etc.), mobile phones (e.g., Android) TM Devices, iOS TM The mobile device 100 includes a processor 102, memory 104, radio 106, speaker 108, microphone 110, and bus 112. The mobile device 100 may include other components (e.g., display, battery, input or output interfaces for data communication or charging), which are not discussed in detail for the sake of brevity.

[0025] Processor 102 typically controls the operation of mobile device 100. Processor 102 can be one or more processors. Processor 102 can execute one or more computer programs, such as an operating system (e.g., Android). TM Operating system, iOS TMThe processor 102 may include a digital signal processor (DSP) or a computer program that can execute DSP functions. This includes operating systems, various application programs (e.g., media player programs, audio effects programs, etc.).

[0026] Memory 104 typically stores instructions executed by processor 102 and data manipulated by processor 102. These instructions and data may include various computer programs (e.g., operating systems, application programs, etc.), media data (audio data, video data, audiovisual data, etc.), configuration data (e.g., user settings and preferences, etc.), etc. Memory 104 may include volatile components (e.g., random access memory (RAM), etc.) and non-volatile components (e.g., read-only memory (ROM), flash memory, etc.).

[0027] Radio 106 typically controls the wireless data exchange between mobile device 100 and other wireless devices and networks. Radio 106 can be one or more radios of various types, such as cellular radio, IEEE 802.11 standard radio (e.g., WiFi). TM Radio), IEEE 802.15.1 standard radio (e.g., Bluetooth) TM Radio 106 can be used to acquire media content, such as streaming content (e.g., for processing by processor 102), downloaded content (e.g., for storage by memory 104), etc. In some embodiments of mobile device 100 (e.g., when mobile device 100 is a recorder with media player functionality), radio 106 may be omitted.

[0028] Speaker 108 typically outputs sound corresponding to audio data. For example, speaker 108 may output streaming audio data received by mobile device 100, stored audio data stored by mobile device 100, etc. In some embodiments of mobile device 100 (e.g., when mobile device 100 is connected to an external speaker via a wired or wireless connection), speaker 108 may be omitted. For example, mobile device 100 may be connected to wireless earbuds via radio 106.

[0029] Microphone 110 typically receives sound that mobile device 100 can use for various purposes. For example, microphone 110 can receive background noise, which mobile device 100 can use to adjust how it processes audio data. As another example, microphone 110 can receive voice commands, which mobile device 100 can use to control media player functions or set user configuration preferences. In some embodiments of mobile device 100 (e.g., when mobile device 100 is connected to an external microphone via a wired or wireless connection), microphone 110 may be omitted.

[0030] Bus 112 typically connects to other components of mobile device 100. Bus 112 may include one or more buses of one or more types, such as internal integrated circuits (I... 2 C) Bus, internal integrated circuit sound (I) 2 Serial Peripheral Interface (SPI) bus, etc.

[0031] Typically, mobile device 100 implements media player functionality to output media data including audio data. Mobile device 100 can also perform audio post-processing on the audio data, such as audio effects. As discussed in more detail below, mobile device 100 implements audio watermarking to avoid double processing of the audio signal. Mobile device 100 may also implement additional functions (e.g., telephone functionality, web browser functionality, camera functionality, two-factor authentication functionality, etc.), which will not be described in detail for the sake of brevity.

[0032] Figure 2 This is a block diagram of the audio processing framework 200. The audio processing framework 200 can be provided by the mobile device 100 (see [link]). Figure 1 For example, it can be implemented based on the function of processor 102 in executing one or more computer programs or controlling dedicated circuit components (DSP, decoder, etc.). Example mobile device operating systems that can be used to implement the audio processing framework 200 include Android. TM Mobile operating system, iOS TM Mobile operating systems, etc. The audio processing framework 200 includes an application layer 202, a framework layer 204, and a vendor layer 206. Dashed lines indicate control signals. The audio processing framework 200 may include (for brevity) additional layers or components not described in detail.

[0033] Application layer 202 typically includes applications executed by mobile device 100 to implement various functions. For example, applications in application layer 202 can interact with operating system components in framework layer 204 to implement media player functionality. This arrangement enables mobile device 100 to work with multiple applications, each with different functions, which can be selected by the user according to their preferences. Application layer 202 includes media player application 210 and user interface application 212.

[0034] Media player application 210 typically implements the media player functionality of mobile device 100. Media player application 210 may be one of multiple media player applications on mobile device 100, each with different functions. Example functions implemented by media player application 210 include media file organization (playlists, shuffle, etc.), media playback (play, pause, skip, etc.), etc. Media player application 210 is typically built as a collection of low-level operating system functions provided by framework layer 204.

[0035] User interface application 212 typically implements user interface functions associated with the media player functionality of mobile device 100. Specifically, for audio processing framework 200, user interface application 212 can be used to select various post-processing effects and audio effects that can be implemented outside of media player application 210. (Depending on the implementation, media player application 210 itself may also implement audio effects.) These audio effects are discussed in more detail below.

[0036] Framework layer 204 typically includes framework components, operating system components, services, and programming interfaces used by applications in application layer 202 to implement applications. For example, a specific media player application 210 in application layer 202 can be built using specific components in framework layer 204 to implement media file organization functions, media playback functions, etc. Specific to audio processing framework 200, framework layer 204 includes media player service 220 and effects service 222.

[0037] Media player service 220 typically includes framework components that implement media player functionality. Media player service 220 interacts with the file system of mobile device 100 to access audio file 230 (e.g., stored audio data, streaming audio data, etc.). Media player service 220 interacts with various components in vendor layer 206 (e.g., to perform decoding, etc.), as discussed further below. Media player service 220 processes audio file 230 and outputs audio signal 232 to effects service 222.

[0038] Effects service 222 typically includes components that implement post-processing functions, including audio effects. Effects service 222 interacts with various components in vendor layer 206 (e.g., to apply various effects, etc.), as discussed further below. Effects service 222 applies audio effects to audio signal 232 output from media player service 220 and generates audio signal 234.

[0039] Effects service 222 can also interact with a mixer component (not shown) in frame layer 204. The mixer component typically mixes system audio (e.g., alarms, notifications, etc.) with other audio signals. For example, while a user is listening to audio, the mixer can mix a ringtone to indicate that mobile device 100 is receiving a phone call. The mixer component can mix system audio before effects service 222 (e.g., mixed with audio signal 232) or after effects service 222 (e.g., mixed with audio signal 234).

[0040] As described above, the components of the frame layer 204 can interact with the components in the supplier layer 206.

[0041] The vendor layer 206 typically includes components developed by entities other than those developing components in the framework layer 204. For example, the framework layer 204 may implement Android from Google LLC. TM The operating system or iOS from Apple Inc. TM Operating system; vendor layer 206 can implement components from Dolby Laboratories, Inc., Apple, Sony Corp., Fraunhofer Society, etc. This arrangement allows components in vendor layer 206 to extend the functionality of mobile device 100 beyond the basic functions provided by frame layer 204, while still remaining under the control of frame layer 204. Specific to audio processing frame 200, vendor layer 206 includes decoder component 240 and processing component 242.

[0042] Decoder component 240 typically performs decoding of audio file 230. For example, media player service 220 may invoke a specific decoder component 240 to decode audio file 230 of a specific type. Decoder component 240 may also be referred to as a codec component, where "codec" represents a combination of encoder and decoder (although the term "codec" can often be used even when the component does not perform encoding). As mentioned above, decoder component 240 may be one or more decoder components that implement one or more different decoding processes. For example, when audio file 230 is an MP3 file, media player service 220 may interact with an MP3 decoder that is decoder component 240. As another example, when audio file 230 is Dolby Digital Plus... TM When accessing files, the media player service 220 can work with Dolby Digital Plus, which functions as a decoder component 240. TM Decoder interaction. Other example decoders include the Advanced Audio Coding (AAC) decoder, Apple...TM Lossless audio codec (ALAC) decoder, etc. Specific decoder components 240 can also apply audio effects, as discussed further below.

[0043] Processing unit 242 typically performs post-processing on audio signal 232 to generate audio signal 234, for example, to apply audio effects. For instance, effects service 222 can invoke a specific processing unit 242 to apply a specific audio effect to audio signal 232. Processing unit 242 may also be referred to as an effects processing unit or a post-processing unit. Processing unit 242 can be one or more processing units that implement one or more audio effects. Audio effects include volume adjustment, volume modeling, dialogue enhancement, and intelligent equalization. Audio effects are discussed in more detail below.

[0044] As discussed above, audio effects can be applied by multiple components in frame layer 204. For example, media player service 220 can generate an audio signal 232 with audio effects by processing audio file 230 using selected decoder component 240. As another example, effects service 222 can generate an audio signal 234 with audio effects by processing audio signal 232 using selected processing component 242. When audio signal 232 has audio effects applied by media player service 220, it is expected that effects service 222 will suppress the application of audio effects to avoid double processing.

[0045] Unfortunately, in the audio processing framework 200, the audio path is limited to the audio signal. The term "audio path" generally refers to the audio input to and output from the media player service 220 and effects service 222. For example, the audio path might only accept two channels of pulse-code modulation (PCM) samples represented by 16-bit integers. The audio path itself does not allow additional control signals, metadata, etc., for the media player service 220 to indicate that audio effects have been applied to that media player service.

[0046] To overcome this limitation, decoder component 240 inserts an audio watermark into audio signal 232 to indicate that audio effects have been applied to the audio signal. When processing component 242 detects the audio watermark, it does not apply audio effects itself; otherwise, it applies audio effects. In this way, using audio watermarking avoids double processing without passing additional control signals, metadata, etc., outside the audio chain.

[0047] Figure 3 This is a block diagram of decoder component 300. Decoder component 300 is decoder component 240 (see [link]). Figure 2 Examples of ), and can be implemented by one or more computer programs as vendor layer 206 (seeFigure 2 The decoder component 300 typically performs decoding on the audio file and selectively inserts an audio watermark when applying audio effects to the audio signal. The decoder component 300 includes a decoder component 302, a transient detector 304, a transform component 306, a copy component 308, an inverse transform component 310, a reassembler component 312, and a selection component 314. The decoder component 300 may include other components not discussed in detail (for brevity).

[0048] Decoder component 302 receives audio file 230 (see...) Figure 2 The decoder component 302 decodes the audio file 230 and generates an audio signal 320. The decoder component 302 can also selectively apply audio effects when generating the audio signal 320. When the decoder component 302 applies audio effects, subsequent components of the decoder component 300 (e.g., 304-310) operate to insert an audio watermark. The decoder component 302 can do so based on user preferences (e.g., according to...). Figure 2 The user interface application 212 sets up audio effects, and machine learning (e.g., analyzing audio files 230 based on decoder component 302) is used to apply audio effects.

[0049] Decoder component 302 can implement one or more decoding processes. Typically, the decoding process performed will depend on the format of the audio file 230. For example, media player service 220 (in frame layer 204, see...) Figure 2 The appropriate decoder component 302 (in vendor layer 206) can be selected based on the audio file 230. An example decoding process includes Dolby DigitalPlus. TM (DD+) decoding, Dolby Digital Plus TM Joint Object Encoding (DD+JOC) Decoding, Dolby AC-4 TM Decoding, Dolby Atmos TM Decoding, etc. Dolby Digital Plus TM Decoding can also be referred to as Enhanced Dolby Digital AC-3 TM (E-AC-3), and conforms to the standards set forth in Annex E of ATSC A / 52:2012 issued by the Advanced Television Systems Committee and Annex E of ETSI TS 102 366 VI.2.1 (2008-08).

[0050] Transient detector 304 detects transients in audio signal 320. Typically, a transient is a high-amplitude, short-duration sound at the beginning of a waveform in phenomena such as music, noise, or speech. A transient can also be described as a short-duration signal representing the non-harmonic band of a sound source. Transients usually occur at the onset of a sound, but can also occur at the end. Transients can contain a high degree of aperiodic components and high-frequency amplitudes greater than the harmonic content of the sound. A transient does not need to be directly dependent on the frequency of the tone it initiates (or terminates).

[0051] Transient detector 304 can use one or more processes to detect transients. For example, audio signal 320 can be a time-domain signal composed of samples, where the samples are grouped into units such as blocks, sub-blocks, frames, etc. Transient detector 304 can detect transients in specific blocks of audio signal 320. Transient detector 304 can examine each sample block to obtain an energy increase (above a defined threshold) from one block to the next. The block size and threshold can be adjusted as needed. For example, block sizes of 256 samples, 128 samples, 64 samples, etc., can be used. The threshold can be based on the relative peak levels of adjacent blocks; a threshold between 1.5 and 2.5 can be used, where 2.0 provides good results. The threshold can be lowered to detect more transients (e.g., making detection more likely over time), or the threshold can be increased to detect fewer transients (e.g., making it less necessary to detect subsequent transients in the near future once a detection has occurred). Transient detector 304 can dynamically adjust the threshold to achieve a target rate of detecting transients within a given time period (e.g., detecting one transient per second). As a specific example, transient detector 304 can implement transient detection as described in ATSC document A / 52B, "Digital Audio Compression Standard (AC-3, E-AC-3) Revision B". When transient detector 304 does not detect a transient, decoder unit 300 continues processing audio file 230. (In this case, the output of decoder unit 300 can be considered audio signal 320.) When transient detector 304 detects a transient, the process continues in units 306-312. When transient detector 304 does not detect a transient, the process can jump to selection unit 314.

[0052] The transformation unit 306 transforms a transient-related portion 328 of the audio signal 320 into frequency domain data 330. For example, when the transient detector 304 detects a transient in a specific block of the audio signal 320, that specific block corresponds to portion 328.

[0053] Transformer 306 can transform portion 328 using one or more transform procedures. As an example, when portion 328 is a block of 256 samples, the transformer can perform a Fast Fourier Transform (FFT) using a block size of 512 points and a block size of 256 overlapping points (referred to as a window); alternatively, a block size of 1024 points or 2048 points can be used.

[0054] The transformation component 306 can use a Hanning window. Other types of windows can be used as needed. For example, a Hamming window can be used, where parameters a0 = 0.54 and a1 = 0.46. As another example, a Blackman window can be used, where parameter α = 0.16. As yet another example, a Gaussian window can be used, where parameter Δ = 0.1.

[0055] The copying unit 308 receives frequency domain data 330 and copies one frequency band (source band) to another frequency band (target band) to generate frequency domain data 332. This copying process can also be referred to as replication or copying. In frequency domain data 330, the target frequency band can be referred to as the original target frequency band, and in frequency domain data 332, the target frequency band can be referred to as the copied target frequency band. This substitution of one frequency band for another serves as an audio watermark. Because replication is performed relative to a detected transient (e.g., after the transient), perceptual masking can improve fidelity.

[0056] The copying unit 308 can also perform energy scaling in the target frequency band, so that the energy level of the copied target frequency band matches the energy level of the original target frequency band (rather than the energy level of the source frequency band). For example, although the spectral shape of the source frequency band is copied, the energy level of the target frequency band is maintained. The energy level can be expressed in decibels (dB).

[0057] The copying component 308 can operate across a variety of spectral bands and ranges. For example, the frequency domain data 330 can range from 0 to 12 kHz, and the source and target bands can have bandwidths between 500 and 1500 Hz. The bandwidth can be increased (to make audio watermarks easier to detect) or decreased (to reduce the likelihood of the watermark affecting the listener's experience) as needed. Experiments show that a bandwidth of 1000 Hz is suitable for detecting [the watermark]. Figure 2The processing unit 242 provides a good balance between perceptibility (by the listener) and the ability to detect transients. The center frequencies of the source and target frequency bands can be located anywhere within 0 to 12 kHz (although copying bands below 3 kHz may result in watermarked audibility due to inaccurate copying of low-frequency content with harmonic content); the source and target frequency bands do not have to be adjacent and there can be other frequency bands between them. Transients can coexist with music and sound content and can be detected along with music and sound content.

[0058] The center frequencies of the frequency bands used as the source and target bands can be adjusted as needed. Experiments show that a reasonable option is a source band including 3500Hz (e.g., a center frequency of 3500Hz) and a target band including 5500Hz (e.g., a center frequency of 5500Hz). Another reasonable option is a source band including 4500Hz and a target band including 6500Hz.

[0059] Considering these bandwidths and center frequencies, a reasonable option is a source band of 3-4 kHz and a target band of 5-6 kHz. Another reasonable option is a source band of 4-5 kHz and a target band of 6-7 kHz.

[0060] Although the duplication occurs within the perceptible audio range (e.g., between 3 and 12 kHz), because duplication is associated with transients, the audio watermark may be imperceptible to the average listener. Therefore, this duplication is used as a watermark to indicate that an audio effect has been applied. This watermark is called an audio watermark because it occurs within the perceptible audio range, rather than being transmitted out of band using metadata, control signals, etc.

[0061] The following is for reference. Figures 5A-5B A specific example illustrating the copying of the source frequency band to the target frequency band is discussed.

[0062] The inverse transform unit 310 performs an inverse transform on the frequency domain data 332 to generate portion 338. Therefore, portion 338 corresponds to portion 328 but has an audio watermark (e.g., the source frequency band is copied into the target frequency band). Typically, the inverse transform unit 310 performs the inverse transform of the transform performed by the transform unit 306. For example, the inverse transform unit can perform a 512-point inverse FFT using a 256-point window to generate a block of 256 time-domain samples.

[0063] The reconstructor component 312 receives audio signal 320 and portion 338, and generates audio signal 340. Audio signal 340 corresponds to audio signal 320, but portion 328 is replaced by portion 338. For example, when the portion corresponds to a sample block, the reconstructor 312 replaces the block containing the transient (portion 328) with portion 338.

[0064] Selection unit 314 receives audio signal 340 and audio signal 320, selects an audio signal based on whether a transient has been detected, and outputs the selection as audio signal 232 (see also). Figure 2 When the transient detector 304 has not yet detected a transient, the selection unit selects audio signal 320 (i.e., without audio watermark) as audio signal 232. When the transient detector 304 has detected a transient, the selection unit 314 selects audio signal 340 (i.e. with audio watermark) as audio signal 232.

[0065] In summary, since the audio watermark is inserted into the audio signal 320 in association with transients, the presence of transients serves to reduce the listener's perception that the audio signal 232 has been modified to include an audio watermark.

[0066] Figure 4 This is a block diagram of processing unit 400. Processing unit 400 is processing unit 242 (see...). Figure 2 Examples of ), and can be implemented by one or more computer programs as vendor layer 206 (see Figure 2 The component is typically processed by component 400 for detection (by...). Figure 2 Decoder component 240 Figure 3 The decoder component 300 inserts an audio watermark and selectively applies audio effects based on the detection. The processing component 400 includes a transient detector 402, a transformation component 404, a comparison component 406, a processing component 408, and a selection component 410. The processing component 400 may include other components not discussed in detail (for brevity).

[0067] Transient detector 402 detects transients in audio signal 232 (see also...) Figure 2 and Figure 3 Typically, transient detector 402 performs the same function as transient detector 304 (see...). Figure 3 The detection process performed is similar to that of transient detector 304. However, transient detector 402 can use a lower threshold than transient detector 304. This allows transient detector 304 to have a higher threshold so that the audio quality is not degraded, and allows transient detector 402 to have a lower threshold to improve the detection rate. For example, when transient detector 304 uses a threshold of 2.0, transient detector 402 can use a threshold between 3.0 and 4.0. When transient detector 402 does not detect a transient, the process continues in parts 408-410. When transient detector 402 detects a transient, the process continues in parts 404-410.

[0068] Transformation unit 404 transforms a transient-related portion 428 of the audio signal 232 into frequency domain data 430. For example, when transient detector 402 detects a transient in a specific block of the audio signal 232, that specific block corresponds to portion 428. Typically, transformation unit 404 performs the same function as transformation unit 306 (see also transformation unit 306). Figure 3 A similar transformation process to the transformation process performed.

[0069] Comparison unit 406 receives frequency domain data 430 and decodes it by decoder unit 300 (see [link]). Figure 3 The two frequency bands being copied are (potentially) compared. For example, when the decoder unit uses 3-4 kHz as the source band and 5-6 kHz as the target band, the comparison unit compares these two bands. Typically, comparison unit 406 calculates the correlation between the two bands to generate result 432. When result 432 is below a threshold, the two bands are uncorrelated (indicating the audio watermark does not exist), and the process continues in units 408-410. When result 432 is above the threshold, the two bands are correlated (indicating the audio watermark exists), and the process continues in selection unit 410.

[0070] Since the transient detector 402 did not detect a transient or the result 432 indicated that the frequency band was irrelevant, the processing unit 408 selectively processes the audio signal 232 to generate an audio signal 434. This processing typically corresponds to applying audio effects to the audio signal 232, as discussed in more detail below. The processing unit 408 operates in three modes.

[0071] In the first mode, when the transient detector 402 does not detect a transient, the processing unit 408 processes the audio signal 232 to generate the audio signal 434. In this mode, since there is no transient that provides an audio watermark, the processing unit 408 assumes that the decoder unit 300 has not applied audio effects, and therefore the processing unit 408 applies audio effects to generate the audio signal 434.

[0072] In the second mode, when the transient detector 402 detects a transient and the result 432 is irrelevant, the processing unit 408 processes the audio signal 232 to generate the audio signal 434. In this mode, the irrelevant frequency band indicates that the decoder unit 300 has not applied an audio watermark and therefore has not applied any audio effects; therefore, the processing unit 408 applies audio effects to generate the audio signal 434.

[0073] In the third mode, when the transient detector 402 detects a transient and the result 432 is correlated, the processing unit 408 does not process the audio signal 232. In this mode, the correlated frequency band indicates that the decoder unit 300 has applied an audio watermark and therefore an audio effect has been applied; therefore, to avoid double processing, the processing unit 408 can suppress operation on the audio signal 232.

[0074] In summary, detecting audio watermarks allows the processing unit 408 to selectively apply audio effects to avoid double processing.

[0075] Based on the fact that transient detector 402 did not detect a transient or that result 432 indicates that the frequency band is relevant, selection component 410 selects audio signal 434 or audio signal 232 to generate audio signal 234 (see also...). Figure 2 Select the component to operate in three modes.

[0076] In the first mode, when transient detector 402 does not detect a transient, selection unit 410 selects audio signal 434 as audio signal 234. In this mode, since there is no transient, processing module 408 applies audio effects to audio signal 232 to generate audio signal 434. Because this mode may result in double processing, transient detector 402 (and Figure 3 The transient detectors (304) can adjust their thresholds so that transients are detected at a desired rate (and audio watermark insertion occurs).

[0077] In the second mode, when transient detector 402 detects a transient and result 432 is correlated, selection unit 410 selects audio signal 232 as audio signal 234. In this mode, the result correlation indicates decoder unit 300 (see [link]). Figure 3 The audio effect is applied to the audio signal 232, so the audio signal can be used. In this way, double processing of the audio signal is avoided.

[0078] In the third mode, when transient detector 402 detects a transient and result 432 is irrelevant, selection unit 410 selects audio signal 434 as audio signal 234. In this mode, irrelevant result indicates to decoder unit 300 (see [link to decoder]). Figure 3 Since the audio effect is not applied to the audio signal 232, the audio signal 434 (with the audio effect applied by the processing unit 408) can be used. In this way, the audio effect can be reliably applied while avoiding double processing without the need for metadata or other out-of-band control signals between components.

[0079] Audio effects

[0080] As discussed above, audio effects can be achieved using the audio processing framework 200 (see [link]). Figure 2 Various components (including decoder 302 (see)) Figure 3 Processing component 408 (see) Figure 4Audio effects are typically applied after decoding or other audio processing; therefore, audio effects can also be referred to as post-processing. Audio effects can modify audio signals based on cognitive and psychoacoustic models of human audio perception. Multiple audio effects can be packaged together; an example effects package is DolbyAudio Processing. TM Audio effects can include volume adjustment, volume modeling, dialogue enhancement, and intelligent equalization.

[0081] Volume adjustment describes the ability to maintain a consistent playback level regardless of source selection and content. For example, the volume should remain constant when a user switches between different songs in a playlist or switches from listening to music to watching a movie. This feature allows for continuous audio analysis based on a psychoacoustic model of loudness perception to assess the listener's perceived loudness. This information is then used to automatically adjust the perceived loudness to a consistent playback level. Volume adjustment can be performed using auditory scene analysis, a cognitive model of audio perception developed by analyzing data about the audio source. This ensures that the audio loudness in the signal is not adjusted at inappropriate times, such as during naturally decaying notes in a song. Volume adjusters can adjust individual channels and frequency bands within each channel to prevent unwanted compression-based "pickle" and "breathing" artifacts. The result is consistent, adjusted audio free of artifacts associated with traditional volume adjustment solutions.

[0082] Volume modeling describes the effect of compensating for a reference level used in audio mixing. In a recording studio, audio is mixed at what audio professionals call a reference level (typically around 85 dB). While this reference level is generally considered loud, it's the volume level at which most people can perceive the entire spectrum of the audio in the mix and hear the expected tonal balance. This is important in terms of how well we actually hear it. Generally, the lower the volume, the less clearly we can hear high and low frequency frequencies—treble and bass. However, traditional volume controls treat all frequencies the same. Therefore, when you turn the volume down, you lose perception of high and low frequency frequencies, and tonal balance is affected. To compensate for this, a volume modeler analyzes the incoming audio, groups similar frequencies into critical bands, and applies an appropriate amount of gain to each band.

[0083] Dialogue enhancement describes the effect of dynamically applying processing to improve the intelligibility of spoken audio. This post-processing feature is designed to enhance the listener's perception and understanding of the dialogue. This involves monitoring the audio track to detect the presence of dialogue. The dialogue enhancer analyzes features from the audio signal and applies pattern recognition to detect the presence of dialogue at all times. When dialogue is detected, the dialogue enhancer can perform two types of dynamic audio processing: dynamic spectral rebalancing of the dialogue and dynamic suppression of interference signals (although other techniques can also be used).

[0084] The dynamic spectral rebalancing of the dialogue enhances the mid-to-high frequencies, which are crucial for comprehensibility. In short, the dialogue's audio spectrum is altered where necessary to highlight the content, allowing listeners to distinguish it more clearly.

[0085] Dynamic suppression of interfering signals reduces the level of mid- to high-frequency sounds unrelated to dialogue in the audio mix. These sounds are identified as interfering with the intelligibility of the dialogue.

[0086] Intelligent equalization describes an effect designed to provide a consistent spectral balance (also known as timbre). This is achieved by continuously monitoring the spectral balance of the audio and comparing it to a specified spectral profile (or timbre), called the reference spectral profile. The equalization filter dynamically transforms the original audio tone to the specified reference spectral profile. This process differs from existing equalization presets found on many audio systems (such as presets for jazz, rock, or voice), where the preset applies the same frequency variation regardless of the content. Typically, when a user sets the bass boost level in a traditional equalizer, the setting may become inappropriate as the bass content in the source audio increases; too much bass can cause distortion. Intelligent equalizers do not adjust bass when there is sufficient bass in the signal. When the source audio lacks sufficient bass, intelligent equalizers appropriately boost the bass. The result is the desired sound, without over-processing or distortion.

[0087] Figures 5A-5B It is a graph illustrating the spectral copy of the audio watermark. Figure 5A Graph 500 shows loudness in dB on the y-axis and frequency in Hz on the x-axis. Spectrum 502 corresponds to the audio signal before the audio watermark was inserted (e.g., Figure 3 The audio signal is 320. Band 504 (3-4kHz) is the source band, and band 506 (5-6kHz) is the target band.

[0088] Figure 5B The spectrum 552 is shown to correspond to the audio signal after the audio watermark has been inserted (e.g., Figure 3The audio signal 340) is plotted in graph 550. In spectrum 552, source band 554 is the same as source band 504 in spectrum 550, but target band 556 corresponds to a copy of source band 554, not a target band 506 in spectrum 550. It should be further noted that target band 556 is scaled so that the energy is continuous in adjacent bands of spectrum 552, rather than simply copying the loudness of source band 554.

[0089] Figure 6 This is a flowchart of audio processing method 600. Method 600 typically inserts an audio watermark to transmit the effects that have been added to the audio, allowing subsequent components to avoid performing double processing. Method 600 can be executed by mobile device 100 (see...). Figure 1 ) is executed, for example, as controlled by one or more computer programs. Method 600 can be executed by an audio processing framework (see... Figure 2 One or more components (decoder component 300 (see...) Figure 3 Processing component 400 (see) Figure 4 (etc.) to implement.

[0090] At 602, the encoded audio data is decoded to generate decoded audio data. For example, decoder component 302 (see...) Figure 3 The audio file 230 can be decoded to generate an audio signal 320. The encoded audio data may include metadata, and generating the decoded audio data may include processing the metadata as part of the decoding process. Generating the decoded audio data may also include applying audio effects. Because method 600 aims to avoid double processing when applying audio effects to the decoder component 302, the remainder of this discussion of method 600 assumes that audio effects have already been applied.

[0091] At position 604, transients are detected in the decoded audio data. For example, transient detector 304 (see...) Figure 3 Transients can be detected. Since method 600 is for inserting an audio watermark in a transient, the remainder of this discussion about method 600 assumes that the transient has already been detected.

[0092] At 606, a first portion of the decoded audio data, related to transients in the decoded audio data, is transformed into first frequency domain data. This first portion may correspond to a sample block. For example, transformation unit 306 (see...) Figure 3 It can transform part 328 of the audio signal 320 to generate frequency domain data 330.

[0093] At 608, a first frequency band of the first frequency domain data is copied to a second frequency band of the first frequency domain data to generate second frequency domain data. This copying may also include scaling the energy in the target frequency band to match the energy of the original target frequency band. For example, copying component 308 (see...) Figure 3 The source bandwidth can be 554 (see...) Figure 5B The energy in target frequency band 556 is copied to the original target frequency band 506 (see [link to target frequency band 506]). Figure 5A The energy in ) matches.

[0094] At 610, the second frequency domain data is transformed to generate the second part. For example, the inverse transform unit 310 (see...) Figure 3 It can transform the frequency domain data 332 to generate part 338.

[0095] At position 612, watermarked audio data is generated, where the watermarked audio data corresponds to the decoded audio data in which the first part is replaced by the second part. For example, recombination component 312 (see...) Figure 3 It can generate an audio signal 340 corresponding to the audio signal 320, but part 328 is replaced by part 338.

[0096] At position 614, a transient is detected in the first audio data. (Typically, the first audio data corresponds to the watermarked audio data at position 612; however, at position 614, the presence of the watermark is unknown, so the label "first audio data" is used.) For example, transient detector 402 (see...) Figure 4 It can detect transients in audio signal 232. Since method 600 is aimed at detecting audio watermarks in transients, the remainder of this discussion about method 600 assumes that transients have already been detected.

[0097] At 616, a portion of the first audio data associated with the transient is transformed into frequency domain data. For example, transformation unit 404 (see...) Figure 4 It can transform part 428 of the audio signal 232 to generate frequency domain data 430.

[0098] At position 618, the first frequency band of the frequency domain data is compared with the second frequency band of the frequency domain data. For example, comparison unit 406 (see...) Figure 4 It can compare two frequency bands in frequency domain data 430 to generate result 432.

[0099] At 620, when the first frequency band and the second frequency band are uncorrelated, processing is performed on the first audio data to generate the second audio data. Uncorrelated frequency bands indicate the absence of an audio watermark. In this case, the first audio data does not have audio effects, therefore audio effects need to be applied. For example, when the frequency bands are uncorrelated, processing unit 408 (see...)Figure 4 It can process audio signal 232 to generate audio signal 434.

[0100] At point 622, when the first frequency band is related to the second frequency band, the first audio data is used as the second audio data without processing. The related frequency band indicates the presence of an audio watermark. In this case, the first audio data has an audio effect, therefore (to avoid double processing) the first audio data is used as the second audio data without applying the audio effect. For example, select component 410 (see...) Figure 4 The audio signal 232 can be selected as the audio signal 234 based on the correlation between the frequency bands indicated by result 432.

[0101] Variations and Options

[0102] exist Figure 2 In the diagram, decoder component 240 and processing component 242 are shown as components of vendor layer 206 in mobile device 100. However, these components can be in separate devices. For example, the decoder component could be located in a server that streams audio signal 232 to the mobile device containing the processing component. In this embodiment, a watermark (when the server applies an effect) enables the mobile device to avoid double processing of the audio.

[0103] Implementation details

[0104] Embodiments may be implemented in hardware, as executable modules stored on a computer-readable medium, or a combination of both (e.g., a programmable logic array). Unless otherwise stated, the steps performed by the embodiments do not need to be inherently associated with any particular computer or other device, although they may be relevant in some embodiments. Specifically, various general-purpose machines may be used with programs written in accordance with the teachings herein, or more specialized devices (e.g., integrated circuits) may be more readily constructed to perform the desired method steps. Therefore, embodiments may be implemented in one or more computer programs that execute on one or more programmable computer systems, each including at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices in a known manner.

[0105] Each such computer program is preferably stored or downloaded to a storage medium or device (e.g., solid-state memory or medium, or magnetic or optical medium) readable by a general-purpose or special-purpose programmable computer, for configuring and operating the computer to execute the program described herein when the computer system reads the storage medium or device. The system of the present invention can also be considered as an embodiment of a computer-readable storage medium configured with a computer program, wherein such a configuration causes the computer system to operate in a specific and predefined manner to perform the functions described herein. (Software itself and intangible or transient signals are excluded in the sense that they are not patentable subject matter.)

[0106] The foregoing description illustrates various embodiments of this disclosure and examples of how aspects of this disclosure may be implemented. The foregoing examples and embodiments should not be considered as limited embodiments, but are presented to illustrate the flexibility and advantages of this disclosure as defined by the appended claims. Other arrangements, embodiments, implementations, and equivalents will be apparent to those skilled in the art based on the foregoing disclosure and the appended claims, and may be employed without departing from the spirit and scope of this disclosure as defined by the claims.

[0107] Various aspects of the invention can be understood from the following enumerated example embodiments (EEE):

[0108] EEE 1. An audio processing method, the method comprising:

[0109] Transients in the first audio data are detected by the processing unit;

[0110] Transform the portion of the first audio data related to the transient into frequency domain data;

[0111] Compare the first frequency band of the frequency domain data with the second frequency band of the frequency domain data;

[0112] When the first frequency band and the second frequency band are unrelated, the processing unit processes the first audio data to generate second audio data; and

[0113] When the first frequency band is related to the second frequency band, the first audio data is used as the second audio data, and the processing unit does not perform processing.

[0114] EEE 2. The method as described in EEE 1, wherein, prior to detecting transients in the first audio data, the method further comprises:

[0115] The decoder component decodes the third audio data to generate the fourth audio data;

[0116] Detect transients in the fourth audio data, wherein the transients in the fourth audio data correspond to transients in the first audio data;

[0117] The first portion of the fourth audio data that is related to transients in the fourth audio data is transformed into first frequency domain data;

[0118] The first frequency band of the first frequency domain data is copied into the second frequency band of the first frequency domain data to generate the second frequency domain data;

[0119] Transform the second frequency domain data to generate the second part; and

[0120] Generate fifth audio data, wherein the fifth audio data corresponds to the fourth audio data in which the first part is replaced by the second part.

[0121] The fifth audio data corresponds to the first audio data.

[0122] EEE 3. The method as described in EEE 2, wherein the third audio data includes an audio signal and metadata, wherein decoding the third audio data includes decoding the audio signal and the metadata to generate the fourth audio data.

[0123] EEE 4. The method of any one of EEE 2 to 3, wherein decoding the third audio data further comprises:

[0124] Apply audio effects to generate the fourth audio data.

[0125] EEE 5. The method of any one of EEE 2 to 4, wherein, before copying the first frequency band into the second frequency band, the first frequency band has a first energy level, and the second frequency band has a second energy level.

[0126] Copying the first frequency band to the second frequency band includes scaling the first energy level to the second energy level.

[0127] EEE 6. The method of any one of EEE 1 to 5, wherein processing the first audio data by the processing unit to generate the second audio data comprises:

[0128] Apply audio effects to the first audio data.

[0129] EEE 7. The method as described in EEE 6, wherein the audio effect is at least one of the following: volume adjuster effect, volume modeler effect, dialogue enhancer effect, and smart equalizer effect.

[0130] EEE 8. The method of any one of EEE 1 to 7, wherein the first portion comprises a plurality of samples of the first audio data, the first audio data including the transients.

[0131] EEE 9. The method of any one of EEE 1 to 8, wherein the first frequency band is a frequency band including 3500 Hz and the second frequency band is a frequency band including 5500 Hz.

[0132] EEE 10. The method of any one of EEE 1 to 8, wherein the first frequency band is a frequency band including 4500 Hz and the second frequency band is a frequency band including 6500 Hz.

[0133] EEE 11. The method of any one of EEE 1 to 10, wherein the first frequency band and the second frequency band each have a bandwidth between 500 Hz and 1500 Hz.

[0134] EEE 12. The method of any one of EEE 1 to 10, wherein the first frequency band and the second frequency band each have a bandwidth of 1000 Hz.

[0135] EEE 13. The method of any one of EEE 1 to 12, wherein the frequency domain data includes a third frequency band, wherein the third frequency band is between the first frequency band and the second frequency band.

[0136] EEE 14. The method of any one of EEE 1 to 13, wherein the frequency domain data is within the perceptible audio range.

[0137] EEE 15. The method of any one of EEE 1 to 14, wherein the frequency domain data is between 3 and 12 kHz.

[0138] EEE 16. A non-transitory computer-readable medium storing a computer program that, when executed by a processor, controls a device to perform processing including the method described in any one of EEE 1 to 15.

[0139] EEE 17. An apparatus for audio processing, the apparatus comprising:

[0140] Processor; and

[0141] memory,

[0142] The processor is configured to control the device to detect transients in the first audio data by the processing unit;

[0143] The processor is configured to control the device to transform a portion of the first audio data related to the transient into frequency domain data;

[0144] The processor is configured to control the device to compare a first frequency band of the frequency domain data with a second frequency band of the frequency domain data;

[0145] Wherein, when the first frequency band and the second frequency band are unrelated, the processor is configured to control the device to perform processing on the first audio data by the processing unit to generate second audio data; and

[0146] Wherein, when the first frequency band is related to the second frequency band, the processor is configured to control the device to use the first audio data as the second audio data, instead of having the processing unit perform the processing.

[0147] EEE 18. The apparatus as described in EEE 17, wherein, prior to detecting transients in the first audio data:

[0148] The processor is configured to control the device to decode the third audio data by the decoder component to generate the fourth audio data;

[0149] The processor is configured to control the device to detect transients in the fourth audio data, wherein the transients in the fourth audio data correspond to transients in the first audio data;

[0150] The processor is configured to control the device to transform a first portion of the fourth audio data that is related to transients in the fourth audio data into first frequency domain data;

[0151] The processor is configured to control the device to copy a first frequency band of the first frequency domain data into a second frequency band of the first frequency domain data to generate second frequency domain data;

[0152] The processor is configured to control the device to transform the second frequency domain data to generate a second portion; and

[0153] The processor is configured to control the device to generate fifth audio data, wherein the fifth audio data corresponds to the fourth audio data in which the first portion is replaced by the second portion.

[0154] The fifth audio data corresponds to the first audio data.

[0155] EEE 19. The apparatus of any one of EEE 17 to 18, wherein, before copying the first frequency band into the second frequency band, the first frequency band has a first energy level and the second frequency band has a second energy level.

[0156] Copying the first frequency band to the second frequency band includes scaling the first energy level to the second energy level.

[0157] EEE 20. The apparatus as described in any one of EEE 17 to 19, wherein the frequency domain data is within the perceptible audio range.

Claims

1. An audio processing method, the method comprising: Transients in the first audio data are detected by the processing unit; Transform the portion of the first audio data related to the transient into frequency domain data; Compare the frequency domain data of the first frequency band with the frequency domain data of the second frequency band; When the frequency domain data of the first frequency band is not related to the frequency domain data of the second frequency band, the processing unit performs processing on the first audio data to generate the second audio data; as well as When the frequency domain data of the first frequency band is related to the frequency domain data of the second frequency band, the first audio data is used as the second audio data, and the processing unit does not perform processing.

2. The method as described in claim 1, wherein, Before detecting transients in the first audio data, the method further includes: The decoder component decodes the third audio data to generate the fourth audio data; Detect transients in the fourth audio data, wherein the transients in the fourth audio data correspond to transients in the first audio data; The first portion of the fourth audio data that is related to transients in the fourth audio data is transformed into first frequency domain data; Copy the first frequency domain data of the first frequency band to the first frequency domain data of the second frequency band to generate the second frequency domain data; Transform the second frequency domain data to generate the second part; and Generate fifth audio data, wherein the fifth audio data corresponds to the fourth audio data in which the first part is replaced by the second part. The fifth audio data corresponds to the first audio data.

3. The method as described in claim 2, wherein, The third audio data includes an audio signal and metadata, wherein decoding the third audio data includes decoding the audio signal and the metadata to generate the fourth audio data.

4. The method according to any one of claims 2 to 3, wherein, Decoding the third audio data further includes: Apply audio effects to generate the fourth audio data.

5. The method according to any one of claims 2 to 3, wherein, Before copying the frequency domain data of the first frequency band into the frequency domain data of the second frequency band, the frequency domain data of the first frequency band has a first energy level and the frequency domain data of the second frequency band has a second energy level. Copying the frequency domain data of the first frequency band to the frequency domain data of the second frequency band includes scaling the first energy level to the second energy level.

6. The method according to any one of claims 1 to 3, wherein, The processing unit performs processing on the first audio data to generate the second audio data, including: Apply audio effects to the first audio data.

7. The method of claim 6, wherein, The audio effect is at least one of the following: volume adjuster effect, volume modeler effect, dialogue enhancer effect, and smart equalizer effect.

8. The method according to any one of claims 1 to 3, wherein, The portion of the first audio data includes multiple samples of the first audio data, and the first audio data includes the transients.

9. The method according to any one of claims 1 to 3, wherein, The first frequency band is a frequency band including 3500 Hz, and the second frequency band is a frequency band including 5500 Hz.

10. The method according to any one of claims 1 to 3, wherein, The first frequency band is a frequency band including 4500 Hz, and the second frequency band is a frequency band including 6500 Hz.

11. The method according to any one of claims 1 to 3, wherein, The first frequency band and the second frequency band each have a bandwidth between 500 Hz and 1500 Hz.

12. The method according to any one of claims 1 to 3, wherein, The first frequency band and the second frequency band each have a bandwidth of 1000 Hz.

13. The method according to any one of claims 1 to 3, wherein, The frequency domain data includes frequency domain data of a third frequency band, wherein the third frequency band is located between the first frequency band and the second frequency band.

14. The method according to any one of claims 1 to 3, wherein, The frequency domain data is within the perceptible audio range.

15. The method according to any one of claims 1 to 3, wherein, The frequency domain data is between 3 kHz and 12 kHz.

16. A non-transitory computer-readable medium storing a computer program that, when executed by a processor, controls a device to perform processing including the method as described in any one of claims 1 to 15.

17. An apparatus for audio processing, the apparatus comprising: processor; as well as memory, The processor is configured to control the device to detect transients in the first audio data by the processing unit; The processor is configured to control the device to transform a portion of the first audio data related to the transient into frequency domain data; The processor is configured to control the device to compare frequency domain data of the first frequency band with frequency domain data of the second frequency band. Wherein, when the frequency domain data of the first frequency band is unrelated to the frequency domain data of the second frequency band, the processor is configured to control the device to perform processing on the first audio data by the processing unit to generate second audio data; and Wherein, when the frequency domain data of the first frequency band is related to the frequency domain data of the second frequency band, the processor is configured to control the device to use the first audio data as the second audio data, instead of having the processing unit perform the processing.

18. The apparatus of claim 17, wherein, Before detecting transients in the first audio data: The processor is configured to control the device to decode the third audio data by the decoder component to generate the fourth audio data; The processor is configured to control the device to detect transients in the fourth audio data, wherein the transients in the fourth audio data correspond to transients in the first audio data; The processor is configured to control the device to transform a first portion of the fourth audio data that is related to transients in the fourth audio data into first frequency domain data; The processor is configured to control the device to copy first frequency domain data of a first frequency band into first frequency domain data of a second frequency band to generate second frequency domain data; The processor is configured to control the device to transform the second frequency domain data to generate a second portion; and The processor is configured to control the device to generate fifth audio data, wherein the fifth audio data corresponds to the fourth audio data in which the first portion is replaced by the second portion. The fifth audio data corresponds to the first audio data.

19. The apparatus as claimed in any one of claims 17 to 18, wherein, Before copying the frequency domain data of the first frequency band into the frequency domain data of the second frequency band, the frequency domain data of the first frequency band has a first energy level and the frequency domain data of the second frequency band has a second energy level. Copying the frequency domain data of the first frequency band to the frequency domain data of the second frequency band includes scaling the first energy level to the second energy level.

20. The apparatus of any one of claims 17 to 18, wherein, The frequency domain data is within the perceptible audio range.

21. A computer program product comprising a computer program including instructions that, when executed by a processor, cause a device to perform the method as described in any one of claims 1 to 15.

Images