Method and apparatus for extracting tone-independent timbre attributes from media signals

Abstract

A method and apparatus for extracting tone-independent timbre attributes from a media signal are disclosed. An example apparatus includes an interface to receive a media signal; and an audio feature extractor for determining a spectrum of audio corresponding to the media signal; and determining a tone-independent timbre attribute of the audio based on an inverse transform of the amplitude of the transform of the spectrum.

Description

[0001] This application is a divisional application of the invention patent application with the original application number 201980018804.5 (international application number PCT / US2019 / 021865) (application date: March 12, 2019, invention name: Method and device for extracting pitch-independent timbre attributes from media signals). Technical Field

[0002] The present disclosure relates generally to audio processing and, more particularly, to methods and apparatus for extracting pitch-independent timbre attributes from media signals. Background Art

[0003] Timbre (e.g., timbre / timbral attributes) is a quality / characteristic of audio that is independent of the audio pitch or loudness. Timbre makes two different sounds different from each other, even if they have the same pitch and loudness. For example, a guitar and a flute playing the same note with the same amplitude sound different because the guitar and flute have different timbres. Timbre corresponds to the frequency and time envelope of an audio event (e.g., the distribution of energy along time and frequency). Audio features that correspond to timbre perception include spectrum and envelope. Summary of the invention

[0004] According to one aspect, the present invention provides a device for extracting pitch-independent timbre attributes from a media signal, the device comprising: an interface for receiving a media signal; and an audio feature extractor, the audio feature extractor being used to: determine a spectrum of audio corresponding to the media signal; and determine the pitch-independent timbre attributes of the audio based on an inverse transform of the transformed amplitude of the spectrum.

[0005] According to another aspect, the present invention also provides a device for extracting pitch-independent timbre attributes from a media signal, the device comprising: a first unit, the first unit being used to determine the frequency spectrum of audio corresponding to the received media signal by executing instructions using a processor; and a second unit, the second unit being used to determine the pitch-independent timbre attributes of the audio based on an inverse transformation of the transformed amplitude of the spectrum by executing instructions using the processor.

[0006] The present invention also provides a method for extracting pitch-independent timbre attributes from a media signal, the method comprising: determining the frequency spectrum of audio corresponding to the received media signal by executing instructions using a processor; and determining the pitch-independent timbre attributes of the audio based on an inverse transformation of the transformed amplitude of the frequency spectrum by executing instructions using the processor.

[0007] In addition, the present invention also relates to a non-transitory computer-readable storage medium comprising instructions, which when executed cause a machine to at least: access a media signal; determine a spectrum of audio corresponding to the media signal; and determine pitch-independent timbre properties of the audio based on an inverse transform of a transformed amplitude of the spectrum. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a diagram of an exemplary meter for extracting pitch-independent timbre attributes from a media signal.

[0009] Figure 2 yes Figure 1 Block diagram of an exemplary audio analyzer and an exemplary audio determiner.

[0010] Figure 3 is a flowchart representing example machine readable instructions that may be executed to implement Figure 1 and Figure 2 An exemplary audio analyzer is provided to extract pitch-independent timbre attributes from a media signal and / or to extract timbre-independent pitch from a media signal.

[0011] Figure 4 is a flowchart representing example machine readable instructions that may be executed to implement Figure 1 and Figure 2 An exemplary audio determiner is provided to characterize audio and / or identify media based on a tonal timbre logarithmic spectrum.

[0012] Figure 5 Shows that you can use Figure 1 and Figure 2 An exemplary audio signal, an exemplary pitch of the audio signal, and an exemplary timbre of the audio signal determined by an exemplary audio analyzer.

[0013] Figure 6 is constructed to execute Figure 3 Example machine readable instructions to control Figure 1 and Figure 2 A block diagram of a processor platform of an exemplary audio analyzer.

[0014] Figure 7 is constructed to execute Figure 4 Example machine readable instructions to control Figure 1 and Figure 2 Block diagram of a processor platform of an exemplary audio determiner.

[0015] These drawings are not drawn to scale. Wherever possible, the same reference numbers will be used throughout the drawings and accompanying written description to refer to the same or like parts. DETAILED DESCRIPTION

[0016] An audio meter is a device that captures audio signals (e.g., directly or indirectly) to process those audio signals. For example, when panelists sign up to have their exposure to media monitored by an audience measurement entity, the audience measurement entity may send a technician to the panelist's home to install a meter (e.g., a media monitor) that is capable of collecting media exposure data from a media output device (e.g., a television, radio, computer, etc.). In another example, a meter may correspond to instructions executed on a processor in a smartphone to, for example, process received audio and / or video data to determine characteristics of the media.

[0017] Typically, the meter includes or is otherwise connected to an interface to receive media signals directly from a media source or indirectly (e.g., collecting ambient audio from a microphone and / or a magnetic coupling device). For example, when a media output device is "on," a microphone may receive an acoustic signal sent by the media output device. The meter may process the received acoustic signal to determine audio features that may be used to characterize and / or identify the audio or audio source. When the meter corresponds to an instruction to operate within and / or with the media output device to receive an audio and / or video signal to be output by the media output device, the meter may process / analyze the input audio and / or video signal to directly determine data associated with the signal. For example, the meter may operate in a set-top box, a receiver, a mobile phone, etc. to receive and process input audio / video data before, during, or after output by the media output device.

[0018] In some examples, the audio metering device / instructions utilize various features of the audio to classify and / or identify the audio and / or audio source. Such features may include energy of the media signal, energy of a frequency band of the media signal, discrete cosine transform (DCT) coefficients of the media signal, etc. Examples disclosed herein classify and / or identify media based on the timbre of the audio corresponding to the media signal.

[0019] Timbre (e.g., timbre / timbre attributes) is a quality / characteristic of audio that is independent of the audio pitch or loudness. For example, a guitar and a flute playing the same note with the same amplitude sound different because the guitar and the flute have different timbres. Timbre corresponds to the frequency and time envelope of an audio event (e.g., the distribution of energy along time and frequency). Traditionally, timbre is characterized by various features. However, timbre has not been extracted from audio independently of other aspects of the audio (e.g., pitch). Therefore, identifying media based on pitch-dependent timbre measurements would require a large database of reference pitch-dependent timbres for each category and each pitch. The examples disclosed herein extract a pitch-independent timbre log spectrum from pitch-independent measured audio, thereby reducing the resources required to classify and / or identify media based on timbre.

[0020] As described above, the extracted timbre that is independent of pitch can be used to classify and / or identify media and / or can be used as part of a signature algorithm. For example, the extracted timbre attribute that is independent of pitch (e.g., log spectrum) can be used to determine that the measured audio (e.g., audio sample) corresponds to a violin, regardless of the notes that the violin is playing. In some examples, the feature audio can be used to adjust the audio settings of the media output device to provide a better audio experience for the user. For example, some audio equalizer settings may be more suitable for audio from a specific instrument and / or style. Therefore, the examples disclosed herein can adjust the audio equalizer settings of the media output device based on the instrument / style corresponding to the extracted timbre identification. In another example, the extracted timbre that is independent of pitch can be used to identify the media being output by a media presentation device (e.g., a television, a computer, a radio, a smart phone, a tablet, etc.) by comparing the extracted timbre attribute that is independent of pitch with a reference timbre attribute in a database. In this manner, the extracted timbre and / or pitch may be used to provide audience measurement entities with more detailed media exposure information than conventional techniques that consider only the pitch of the received audio.

[0021] Figure 1 An exemplary audio analyzer 100 for extracting pitch-independent timbre properties from a media signal is shown. Figure 1 An exemplary audio analyzer 100 , an exemplary media output device 102 , exemplary speakers 104 a , 104 b , an exemplary media signal 106 , and an exemplary audio determiner 108 are included.

[0022] Figure 1The exemplary audio analyzer 100 of the embodiment receives a media signal from a device (e.g., an exemplary media output device 102 and / or exemplary speakers 104a, 104b) and processes the media signal to determine timbre properties (e.g., logarithmic spectrum) that are independent of pitch and pitch properties that are independent of timbre. In some examples, the audio analyzer 100 may include (or otherwise be connected to) a microphone to receive the exemplary media signal 106 by sensing ambient audio. In such examples, the audio analyzer 100 may be implemented in a meter or other computing device (e.g., a computer, tablet, smart phone, smart watch, etc.) that utilizes a microphone. In some examples, the analyzer 100 includes an interface for directly (e.g., via a wired or wireless connection) receiving the exemplary media signal 106 from the exemplary media output device 102 and / or a media presentation device that presents media to the media output device 102. For example, the audio analyzer 100 may receive the media signal 106 directly from a set-top box, a mobile phone, a gaming device, an audio receiver, a DVD player, a Blu-ray player, a tablet computer, and / or any other device that provides media to be output by the media output device 102 and / or the exemplary speakers 104a, 104b. Figure 2 As further described, the exemplary audio analyzer 100 extracts pitch-independent timbre attributes and / or timbre-independent pitch attributes from the media signal 106. If the media signal 106 is a video signal having an audio component, the exemplary audio analyzer 100 extracts the audio component from the media signal 106 before extracting the pitch and / or timbre.

[0023] Figure 1 The exemplary media output device 102 is a device that outputs media. Figure 1 The exemplary media output device 102 is shown as a television, but the exemplary media output device 102 may be a radio, an MP3 player, a video game console, a stereo system, a mobile device, a tablet, a computing device, a tablet computer, a laptop computer, a projector, a DVD player, a set-top box, a set-top device, and / or any device capable of outputting media (e.g., video and / or audio). The exemplary media output device may include a speaker 104a and / or may be coupled or connected to a portable speaker 104b via a wired or wireless connection. The exemplary speakers 104a, 104b output the audio portion of the media output by the exemplary media output device. Figure 1In the example shown, media signal 106 represents the audio output by exemplary speakers 104a, 104b. Additionally or alternatively, exemplary media signal 106 can be an audio signal and / or a video signal, which is sent to exemplary media output device 102 and / or exemplary speakers 104a, 104b to be output by exemplary media output device 102 and / or exemplary speakers 104a, 104b. For example, exemplary media signal 106 can be a signal from a game console, which is sent to exemplary media output device 102 and / or exemplary speakers 104a, 104b to output the audio and video of a video game. Exemplary audio analyzer 100 can receive media signal 106 directly from a media presentation device (e.g., a game console) and / or from ambient audio. In this way, audio analyzer 100 can even classify and / or identify audio from a media signal when speakers 104a, 104b are closed, not working or turned off.

[0024] Figure 1The exemplary audio determiner 108 of the exemplary audio analyzer 100 characterizes audio and / or identifies media based on the pitch-independent timbre attributes received from the exemplary audio analyzer 100. For example, the audio determiner 108 may include a database of reference pitch-independent timbre attributes corresponding to the classification and / or identification. In this way, the exemplary audio determiner 108 can compare the received pitch-independent timbre attributes with the reference pitch-independent attributes to identify a match. If the exemplary audio determiner 108 identifies a match, the exemplary audio determiner 108 classifies the audio and / or identifies the media on the information corresponding to the matched reference timbre attributes. For example, if the received timbre attributes match the reference attributes corresponding to a speaker, the exemplary audio determiner 108 classifies the audio corresponding to the received timbre attributes as audio from a speaker. In such an example, if the audio analyzer 100 is part of a mobile phone, the exemplary audio analyzer 100 can receive an audio signal of a speaker playing a song (e.g., via an interface that receives an audio / video signal or via a microphone of a mobile phone that receives an audio signal). In this way, the audio determiner 108 can recognize that the instrument corresponding to the received audio is a trumpet and identify the trumpet to the user (e.g., using a user interface of a mobile device). In another example, if the received timbre attribute matches a reference attribute corresponding to a specific video game, the exemplary audio determiner 108 can identify the audio corresponding to the received timbre attribute as coming from a specific video game. The exemplary audio determiner 108 can generate a report to identify the audio. In this way, the audience measurement entity can trust exposure to the video game based on the report. In some examples, the audio determiner 108 receives the timbre directly from the audio analyzer 100 (e.g., the audio analyzer 100 and the audio determiner 108 are both located in the same device). In some examples, the audio determiner 108 is located in a different location and receives the timbre from the exemplary audio analyzer 100 via wireless communication. In some examples, the audio determiner 108 sends instructions to the example audio media output device 102 and / or the example audio analyzer 100 (e.g., when the example audio analyzer 100 is implemented in the example media output device 102) to adjust the audio equalizer settings based on the audio classification. For example, if the audio determiner 108 classifies the audio output by the media output device 102 as coming from a speaker, the example audio determiner 108 may send instructions to adjust the audio equalizer settings to settings corresponding to the speaker audio. Figure 2 An exemplary audio determiner 108 is further described.

[0025] Figure 2 include Figure 1 1 is a block diagram of an example implementation of an example audio analyzer 100 and an example audio determiner 108. Figure 2The exemplary audio analyzer 100 includes an exemplary media interface 200 , an exemplary audio extractor 202 , an exemplary audio feature extractor 204 , and an exemplary device interface 206 . Figure 2 The example audio determiner 108 of the example audio analyzer 100 includes an example device interface 210, an example timbre processor 212, an example timbre database 214, and an example audio setting adjuster 216. In some examples, elements of the example audio analyzer 100 may be implemented in the example audio determiner 108 and / or elements of the example audio determiner 108 may be implemented in the example audio analyzer 100.

[0026] Figure 2 The exemplary media interface 200 receives (eg, samples) Figure 1 In some examples, media interface 200 may be a microphone for collecting media signal 106 by sensing ambient audio to obtain media signal 106 as audio. In some examples, media interface 200 may be an interface for directly receiving audio and / or video signals (e.g., digital representations of media signals) to be output by exemplary media output device 102. In some examples, media interface 200 may include two interfaces, a microphone for detecting and sampling ambient audio and an interface for directly receiving and / or sampling audio and / or video signals.

[0027] Figure 2 The exemplary audio extractor 202 extracts audio from the received / sampled media signal 106. For example, the audio extractor 202 determines whether the received media signal 106 corresponds to an audio signal or a video signal having an audio component. If the media signal corresponds to a video signal having an audio component, the exemplary audio extractor 202 extracts the audio component to generate an audio signal / sample for further processing.

[0028] Figure 2 The exemplary audio feature extractor 204 processes the audio signal / sample to extract a pitch-independent timbre log spectrum and / or a timbre-independent pitch log spectrum. The log spectrum is the convolution between the pitch-independent (e.g., pitchless) timbre log spectrum and the timbre-independent (e.g., pitchless) pitch log spectrum (e.g., X=T*P, where X is the log spectrum of the audio signal, T is the pitch-independent log spectrum, and P is the timbre-independent pitch log spectrum). Therefore, in the Fourier domain, the amplitude of the Fourier transform (FT) of the log spectrum on the audio signal can correspond to an approximation of the FT of the timbre (e.g., F(X)=F(T)×F(P), where F(.) is the Fourier transform, F(T)≈|F(X)|, and F(P)≈e j arg(F(X))). The complex independent variable is a combination of amplitude and phase (e.g., corresponding to energy and offset). Therefore, the FT of timbre can be approximated by the amplitude of the FT of the log spectrum. Therefore, in order to determine the timbre-independent log spectrum of timbre and / or the timbre-independent log spectrum of pitch of the audio signal, the exemplary audio feature extractor 204 determines the log spectrum of the audio signal (e.g., using a constant Q transform (CQT)) and transforms the log spectrum to the frequency domain (e.g., using an FT). In this manner, the exemplary audio feature extractor 204 (A) extracts the amplitude of the FT based on an inverse transform (e.g., an inverse Fourier transform (FFT) of the transform output). -1 )(For example, T = F -1 (|F(X)|)) to determine the pitch-dependent timbre log spectrum, and (B) based on the inverse transform of the complex argument of the transform output (e.g., P = F -1 (e j arg(F(X)) )) to determine the logarithmic spectrum of the timbre-free pitch. The logarithmic frequency scale of the audio spectrum of the audio signal allows the pitch shift to be equal to the vertical translation. Therefore, the exemplary audio feature extractor 204 uses CQT to determine the logarithmic spectrum of the audio signal.

[0029] In some examples, if Figure 2 If the exemplary audio feature extractor 204 determines that the resulting timbre and / or pitch is not satisfactory, the audio feature extractor 204 filters the result to improve the decomposition. For example, the audio feature extractor 204 can filter the result by emphasizing specific harmonics in the timbre or by forcing a single peak / line in the pitch and updating other components of the result. The exemplary audio feature extractor 204 can filter once or can perform an iterative algorithm while updating the filter / pitch at each iteration, thereby ensuring that the overall convolution of the pitch and timbre produces the original logarithmic spectrum of the audio. The audio feature extractor 204 can determine whether the result is satisfactory based on user and / or manufacturer preferences.

[0030] Figure 2The exemplary device interface 206 of the exemplary audio analyzer 100 is connected to the exemplary audio determiner 108 and / or other devices (e.g., user interfaces, processing devices, etc.) interface. For example, when the audio feature extractor 204 determines a timbre attribute that is independent of pitch, the exemplary device interface 206 can send the attribute to the exemplary audio determiner 108 to classify the audio and / or identify the media. In response, the device interface 206 can receive the classification and / or identification (e.g., an identifier corresponding to the source of the media signal 106) from the exemplary audio determiner 108 (e.g., in a signal or report). In such an example, the exemplary device interface 206 can send the classification and / or identification to other devices (e.g., user interfaces) to display the classification and / or identification to the user. For example, if the audio analyzer 100 is being used in conjunction with a smart phone, the device interface 206 can output the results of the classification and / or identification to the user of the smart phone via the interface (e.g., screen) of the smart phone.

[0031] Figure 2 The exemplary device interface 210 of the exemplary audio determiner 108 receives pitch-independent timbre attributes from the exemplary audio analyzer 100. In addition, the exemplary device interface 210 outputs a signal / report representing a classification and / or identification determined by the exemplary audio determiner 108. The report can be a signal corresponding to the classification and / or identification based on the received timbre. In some examples, the device interface 210 sends the report (e.g., including an identification of the media corresponding to the timbre) to a processor (e.g., a processor of an audience measurement entity) for further processing. For example, a processor of the receiving device can process the report to generate media exposure metrics, audience measurement metrics, etc. In some examples, the device interface 210 sends the report to the exemplary audio analyzer 100.

[0032] Figure 2The exemplary timbre processor 212 of the exemplary audio analyzer 100 processes the received timbre attributes to characterize the audio and / or identify the source of the audio. For example, the timbre processor 212 may compare the received timbre attributes with the reference attributes in the exemplary timbre database 214. In this manner, if the exemplary timbre processor 212 determines that the received timbre attributes match the reference attributes, the exemplary timbre processor 212 classifies and / or identifies the source of the audio based on the data corresponding to the matched reference timbre attributes. For example, if the timbre processor 212 determines that the received timbre attributes match the reference timbre attributes corresponding to a specific commercial, the timbre processor 212 identifies the source of the audio as a specific commercial. In some examples, the classification may include a type classification. For example, if the exemplary timbre processor 212 determines multiple instruments based on timbre, the exemplary timbre processor 212 may identify the style of the audio (e.g., classical, rock, hip-hop, etc.) based on the identified instruments and / or based on the timbre itself. In some examples, when the timbre processor 212 does not find a match, the exemplary timbre processor 212 stores the received timbre attributes in the timbre database 214 to become new reference timbre attributes. If the exemplary timbre processor 212 stores the new reference timbre in the exemplary timbre database 214, the exemplary device interface 210 sends an instruction to the exemplary audio analyzer 100 to prompt the user for identification information (e.g., what is the audio classification, what is the media source, etc.). In this way, if the audio analyzer 100 responds to additional information, the timbre database 214 can store the additional information in conjunction with the new reference timbre. In some examples, the technician analyzes the new reference timbre to determine the additional information. The exemplary timbre processor 212 generates a report based on the classification and / or identification.

[0033] Figure 2 The exemplary audio settings adjuster 216 determines the audio equalizer settings based on the classified audio. For example, if the classified audio corresponds to one or more instruments and / or styles, the exemplary audio settings adjuster 216 can determine the audio equalizer settings corresponding to the one or more instruments and / or styles. In some examples, if the audio is classified as classical music, the exemplary audio settings adjuster 216 can select a classical audio equalizer setting corresponding to classical music (e.g., based on bass level, vibrato level, etc.). In this way, the exemplary device interface 210 can send the audio equalizer settings to the exemplary media output device 102 and / or the exemplary audio analyzer 100 to adjust the audio equalizer settings of the exemplary media output device 102.

[0034] Although in Figure 2 The implementation is shown in Figure 1The exemplary audio analyzer 100 and the exemplary audio determiner 108 are exemplary embodiments, but Figure 2 One or more of the elements, processes, and / or devices shown in the example may be combined, divided, rearranged, omitted, removed, and / or implemented in any other manner. In addition, the example media interface 200, the example audio extractor 202, the example audio feature extractor 204, the example device interface 206, the example audio setting adjuster 216, and / or (more generally) Figure 2 The exemplary audio analyzer 100 and / or the exemplary device interface 210, the exemplary timbre processor 212, the exemplary timbre database 214, the exemplary audio setting adjuster 216 and / or (more generally) Figure 2 The example audio determiner 108 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, the example media interface 200, the example audio extractor 202, the example audio feature extractor 204, the example device interface 206, and / or (more generally) Figure 2 The exemplary audio analyzer 100 and / or the exemplary device interface 210, the exemplary timbre processor 212, the exemplary timbre database 214, the exemplary audio setting adjuster 216 and / or (more generally) Figure 2 The exemplary audio determiner 108 may be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field programmable logic devices (FPLDs). When any apparatus or system claim of this patent is read to cover pure software and / or firmware implementations, the exemplary media interface 200, the exemplary audio extractor 202, the exemplary audio feature extractor 204, the exemplary device interface 206, and / or (more generally) Figure 2 The exemplary audio analyzer 100 and / or the exemplary device interface 210, the exemplary timbre processor 212, the exemplary timbre database 214, the exemplary audio setting adjuster 216 and / or (more generally) Figure 2 At least one of the exemplary audio determiners 108 is hereby expressly defined as comprising a non-transitory computer-readable storage device or storage disk, such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc., including software and / or firmware. In addition, Figure 1 The exemplary audio analyzer 100 and / or the exemplary audio determiner 108 may also include Figure 2One or more elements, processes and / or devices in addition to or in place of those shown in, and / or may include more than one of any or all of the shown elements, processes and devices. As used herein, the phrase "in communication," including variations thereof, encompasses direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but additionally includes selective communication at periodic intervals, predetermined intervals, non-periodic intervals, and / or one-time events.

[0035] Figure 3 The diagram shows a method for implementing Figure 2 A flowchart of exemplary hardware logic or machine-readable instructions of the audio analyzer 100, and Figure 4 The diagram shows a method for implementing Figure 2 108 of the audio determiner 108. The machine-readable instructions may be a flowchart of an exemplary hardware logic or machine-readable instructions .... The machine-readable instructions may be a flowchart of an exemplary hardware logic or machine-readable instructions of the audio determiner 108. The machine-readable instructions may be a flowchart of an exemplary hardware logic or machine-readable instructions of the audio determiner 108. Figure 6 and / or Figure 7 The program or portion of the program is executed by the processor 612, 712 shown in the exemplary processor platform 600, 700 discussed above. The program may be implemented in software stored on a non-transitory computer-readable storage medium such as a CD-ROM, floppy disk, hard drive, DVD, Blu-ray disk, or memory associated with the processor 612, 712, but the entire program and / or its portion may alternatively be executed by a device 712 other than the processor 612, 712 and / or implemented in firmware or dedicated hardware. In addition, although reference is made to Figure 3 and Figure 4 The flowcharts shown in describe exemplary procedures, but many other methods of implementing the exemplary audio analyzer 100 and / or the exemplary audio determiner 108 may alternatively be used. For example, the order of execution of the blocks may be changed, and / or some of the blocks described may be changed, removed, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are configured to perform the corresponding operations without executing software or firmware.

[0036] As mentioned above, Figure 3 and Figure 4The exemplary processes of are implemented using executable instructions (e.g., computer and / or machine readable instructions) stored on a non-transitory computer and / or machine readable medium, such as a hard drive, flash memory, read-only memory, compact disk, digital versatile disk, cache, random access memory, and / or any other storage device or storage disk in which information is stored for any duration (e.g., extended period of time, permanent for brief instances, temporary buffering, and / or caching of information). As used herein, the term non-transitory computer readable medium is expressly defined to include any type of computer readable storage device and / or storage disk, and excludes propagating signals and transmission media.

[0037] "Include" and "comprising" are used as open-ended terms in this document. Thus, whenever a claim uses any form of "include" or "comprising" as a preamble or within any type of claim statement, it should be understood that additional elements, terms, etc. may be present without falling outside the scope of the corresponding claim or statement. As used herein, when the phrase "at least" is used as a transitional term, such as in the preamble of a claim, it is open-ended in the same way as the terms "include" and "comprising". When used, for example, in a form such as A, B, and / or C, the term "and / or" refers to any combination or subset of A, B, and C, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, and (6) B and C.

[0038] Figure 3 is an exemplary flow chart 300 representing exemplary machine-readable instructions that may be executed by Figure 1 and Figure 2 The exemplary audio analyzer 100 of FIG. 1 is implemented to extract pitch-independent timbre properties from a media signal (eg, an audio signal of the media signal). Figure 1 The exemplary audio analyzer 100 describes Figure 3 , but the example instructions can be used by the audio analyzer in any environment.

[0039] At block 302, the exemplary media interface 200 receives one or more media signals or samples of media signals (e.g., the exemplary media signal 106). As described above, the exemplary media interface 200 may receive the media signal 106 directly (e.g., as a signal to / from the media output device 102) or indirectly (e.g., as a microphone that detects the media signal by sensing ambient audio). At block 304, the exemplary audio extractor 202 determines whether the media signal corresponds to video or audio. For example, if the media signal is received using a microphone, the audio extractor 202 determines that the media corresponds to audio. However, if the media signal is a received signal, the audio extractor 202 processes the received media signal to determine whether the media signal corresponds to audio or video with an audio component. If the exemplary audio extractor 202 determines that the media signal corresponds to audio (block 304: audio), the process continues to block 308. If the exemplary audio extractor 202 determines that the media signal corresponds to video (block 306: video), the exemplary audio extractor 202 extracts the audio component from the media signal (block 306).

[0040] At block 308, the example audio feature extractor 204 determines a log spectrum of the audio signal (e.g., X). For example, the audio feature extractor 204 may determine the log spectrum of the audio signal by performing a CQT. At block 310, the example audio feature extractor 204 transforms the log spectrum to the frequency domain. For example, the audio feature extractor 204 performs an FT on the log spectrum (e.g., F(X)). At block 312, the example audio feature extractor 204 determines a magnitude of the transform update (e.g., |F(X)|). At block 314, the example audio feature extractor 204 determines a pitch-independent timbre log spectrum of the audio (e.g., T=F) based on an inverse transform (e.g., inverse FT) of the magnitude of the transform output. -1 |F(X)|). At block 316, the example audio feature extractor 204 determines the complex argument (e.g., e j arg(F(X)) At block 318, the example audio feature extractor 204 determines the uncolored pitch log spectrum (eg, P=F) of the audio based on an inverse transform (eg, inverse FT) of the complex argument of the transform output. -1 (e j arg (F(X)) )).

[0041] At block 320, the example audio feature extractor 204 determines whether the result (e.g., the determined pitch and / or the determined timbre) is satisfactory. Figure 2As described above, the example audio feature extractor 204 determines whether the result is satisfactory based on user and / or manufacturer result preferences. If the example audio feature extractor 204 determines that the result is satisfactory (block 320: yes), the process continues to block 324. If the example audio feature extractor 204 determines that the result is not satisfactory (block 320: no), the example audio feature extractor 204 filters the result (block 322). Figure 2 As described, the exemplary audio feature extractor 204 may filter the results (eg, once or iteratively) by emphasizing harmonics in the timbre or forcing individual peaks / lines in the pitch.

[0042] At box 324, the exemplary device interface 206 sends the result to the exemplary audio determiner 108. At box 326, the exemplary audio feature extractor 204 receives the classification and / or identification data corresponding to the audio signal. Alternatively, if the audio determiner 108 cannot match the timbre of the audio signal to the benchmark, the device interface 206 can send instructions for additional data corresponding to the audio signal. In such an example, the device interface 206 can send a prompt to the user interface to provide the additional data to the user. Therefore, the exemplary device interface 206 can provide the additional data to the exemplary audio determiner 108 to generate a new reference timbre attribute. At box 328, the exemplary audio feature extractor 204 sends the classification and / or identification to other connected devices. For example, the audio feature extractor 204 can send the classification to the user interface to provide the classification to the user.

[0043] Figure 4 is an exemplary flow chart 400 representing example machine-readable instructions that may be executed by Figure 1 and Figure 2 The exemplary audio determiner 108 of the embodiment performs to classify audio and / or identify media based on the pitch-independent timbre attributes of the audio. Figure 1 The exemplary audio determiner 108 describes Figure 4 , but the example instructions may be used by an audio determiner in any environment.

[0044] At block 402, the example device interface 210 receives a measured (e.g., determined or extracted) atonal timbre logarithmic spectrum from the example audio analyzer 100. At block 404, the example timbre processor 212 compares the measured atonal timbre logarithmic spectrum to a reference atonal timbre logarithmic spectrum in the example timbre database 214. At block 406, the example timbre processor 212 determines whether a match is found between the received atonal timbre attributes and the reference atonal timbre attributes. If the example timbre processor 212 determines that a match is determined (block 406: yes), the example timbre processor 212 classifies the audio based on the match (e.g., identifies an instrument and / or style) and / or identifies media corresponding to the audio using additional data stored in the example timbre database 214 corresponding to the matched reference timbre attributes (block 408).

[0045] At block 410, the example audio settings adjuster 216 determines whether the audio settings of the media output device 102 can be adjusted. For example, there may be an enable setting to allow the audio settings of the media output device 102 to be adjusted based on the classification of the audio output by the example media output device 102. If the example audio settings adjuster 216 determines not to adjust the audio settings of the media output device 102 (block 410: No), the process continues to block 414. If the example audio settings adjuster 216 determines to adjust the audio settings of the media output device 102 (block 410: Yes), the example audio settings adjuster 216 determines the media output device setting adjustment based on the classified audio. For example, the example audio settings adjuster 216 may select an audio equalizer setting based on one or more identified instruments and / or identified styles (e.g., from timbre or based on the identified instrument) (block 412). At block 414, the example device interface 210 outputs a report corresponding to the classification, identification, and / or media output device setting adjustment. In some examples, device interface 210 outputs the report to another device for further processing / analysis. In some examples, device interface 210 outputs the report to exemplary audio analyzer 100 to display the results to a user via a user interface. In some examples, device interface 210 outputs the report to exemplary media output device 102 in order to adjust the audio settings of media output device 102.

[0046] If the example timbre processor 212 determines that a match is not determined (block 406: No), the example device interface 210 prompts for additional information corresponding to the audio signal (block 416). For example, the device interface 210 may send instructions to the example audio analyzer 100 to (A) prompt the user to provide information corresponding to the audio, or (B) prompt the audio analyzer 100 to respond with the complete audio signal. At block 418, the example timbre database 214 stores the measured untimbre tonal log spectrum in conjunction with corresponding data that may have been received.

[0047] Figure 5 An exemplary FT of a logarithmic spectrum 500 of an audio signal, an exemplary untoned tonal logarithmic spectrum 502 of an audio signal, and an exemplary untoned tonal logarithmic spectrum 504 of an audio signal are shown.

[0048] As combined Figure 2 As described, when the example audio analyzer 100 receives the example media signal 106 (e.g., or a sample of a media signal), the example audio analyzer 100 determines an example log spectrum of the audio signal / sample (e.g., if the media sample corresponds to a video signal, the audio analyzer 100 extracts an audio component). In addition, the example audio analyzer 100 determines the FT of the log spectrum. Figure 5 The exemplary FT log spectrum 500 corresponds to an exemplary transformed output of the log spectrum of the audio signal / sample. The exemplary uncolored tonal log spectrum 502 corresponds to the inverse FT of the complex argument of the exemplary FT of the log spectrum 500 (e.g., P=F -1 (e j arg(F(X)) )), while the untoned timbre log spectrum 504 corresponds to the inverse FT of the magnitude of the exemplary FT of the log spectrum 500 (e.g., T=F -1 (|F(X)|)). Figure 5 As shown, the exemplary FT of log spectrum 500 corresponds to the convolution of the exemplary untoned pitched log spectrum 502 with the exemplary untoned pitched log spectrum 504. The convolution of the exemplary pitched log spectrum 502 with the peak adds an offset.

[0049] Figure 6 is a block diagram of an exemplary processor platform 600 configured to perform Figure 3 Instructions to achieve Figure 2 The processor platform 600 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, an iPad, etc.). TMtablet computer), personal digital assistant (PDA), Internet appliance, DVD player, CD player, digital video recorder, Blu-ray player game console, personal video recorder, set-top box, headset or other wearable device, or any other type of computing device.

[0050] The processor platform 600 of the illustrated example includes a processor 612. The processor 612 of the illustrated example is hardware. For example, the processor 612 may be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements Figure 2 An exemplary media interface 200, an exemplary audio extractor 202, an exemplary audio feature extractor 204, and / or an exemplary device interface.

[0051] The processor 612 of the illustrated example includes a local memory 613 (e.g., a cache). The processor 612 of the illustrated example communicates with a main memory including a volatile memory 614 and a non-volatile memory 616 via a bus 618. The volatile memory 614 may be comprised of a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 616 may be implemented by a flash memory and / or any other desired type of memory device. Access to the main memory 614, 616 is controlled by a memory controller.

[0052] The processor platform 600 of the illustrated example also includes an interface circuit 620. The interface circuit 620 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), interface, near field communication (NFC) interface, and / or PCI Express interface.

[0053] In the example shown, one or more interface circuits 622 are connected to the interface circuit 620. The input devices 622 allow a user to enter data and / or commands into the processor 612. The input devices may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a track pad, a track ball, isopoint, and / or a voice recognition system.

[0054] One or more output devices 624 are also connected to the interface circuit 620 of the illustrated example. The output device 624 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit 620 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0055] The interface circuit 620 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces, in order to facilitate the exchange of data with external machines (e.g., any type of computing device) via the network 626. Communication can be achieved through, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, etc.

[0056] The processor platform 600 of the illustrated example also includes one or more mass storage devices 628 for storing software and / or data. Examples of such mass storage devices 628 include floppy disk drives, hard disk drives, optical disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0057] Figure 3 The machine-executable instructions 632 may be stored in the mass storage device 628, in the volatile memory 614, in the non-volatile memory 616, and / or on a removable, non-transitory computer-readable storage medium such as a CD or DVD.

[0058] Figure 7 is a block diagram of an exemplary processor platform 700 configured to perform Figure 4 Instructions to achieve Figure 2 The processor platform 700 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, such as an iPad TM tablet computer), personal digital assistant (PDA), Internet appliance, DVD player, CD player, digital video recorder, Blu-ray player, game console, personal video recorder, set-top box, headset or other wearable device, or any other type of computing device.

[0059] The processor platform 700 of the illustrated example includes a processor 712. The processor 712 of the illustrated example is hardware. For example, the processor 712 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements the exemplary device interface 210, the exemplary timbre processor 212, the exemplary timbre database 214, and / or the exemplary audio setting adjuster 216.

[0060] The processor 712 of the illustrated example includes a local memory 713 (e.g., a cache). The processor 712 of the illustrated example communicates with a main memory including a volatile memory 714 and a non-volatile memory 716 via a bus 718. The volatile memory 714 may be comprised of a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 716 may be implemented by a flash memory and / or any other desired type of memory device. Access to the main memory 714, 716 is controlled by a memory controller.

[0061] The processor platform 700 of the illustrated example also includes an interface circuit 720. The interface circuit 720 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), interface, near field communication (NFC) interface, and / or PCI Express interface.

[0062] In the example shown, one or more input devices 722 are connected to the interface circuit 720. The input devices 722 allow a user to enter data and / or commands into the processor 712. The input devices may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a track pad, a track ball, isopoint, and / or a voice recognition system.

[0063] One or more output devices 724 are also connected to the interface circuit 720 of the illustrated example. The output device 724 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit 720 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0064] The interface circuit 720 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces, in order to facilitate the exchange of data with external machines (e.g., any type of computing device) via the network 726. Communication can be achieved through, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, etc.

[0065] The processor platform 700 of the illustrated example also includes one or more mass storage devices 728 for storing software and / or data. Examples of such mass storage devices 728 include floppy disk drives, hard disk drives, optical disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0066] Figure 4 The machine-executable instructions 732 may be stored in the mass storage device 728, in the volatile memory 714, in the non-volatile memory 716, and / or on a removable, non-transitory computer-readable storage medium such as a CD or DVD.

[0067] It can be understood from the above that the above disclosed methods, devices and products extract timbre attributes that are independent of pitch from media signals. The examples disclosed herein determine the pitch-independent timbre logarithmic spectrum based on audio received directly or indirectly from a media output device. The examples disclosed herein also include classifying audio based on timbre (e.g., identifying musical instruments) and / or identifying media sources of audio based on timbre (e.g., songs, video games, advertisements, etc.). Using the examples disclosed herein, timbre can be used to classify and / or identify audio with much fewer resources than traditional techniques because the extracted timbre is independent of pitch. Therefore, audio can be classified and / or identified without the need for multiple reference timbre attributes of multiple pitches. In contrast, timbre independent of pitch can be used to classify audio regardless of pitch.

[0068] Although certain exemplary methods, devices, and articles have been described herein, other implementations are possible. The coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, devices, and articles that fully fall within the scope of the claims of this patent.

Claims

1. A method, comprising: determining the log spectrum of an audio signal; transforming the logarithmic spectrum of the audio signal into the frequency domain to generate a transformed output; determining a magnitude of the transformed output; as well as A timbral attribute of the audio signal is determined based on an inverse transform of the amplitude.

2. The method according to claim 1, wherein: The audio signal is part of the media signal.

3. The method according to claim 1, wherein: The audio signal is an audio component of a video signal, and the method further comprises extracting the audio signal from the video signal.

4. The method according to claim 1, wherein: Determining the logarithmic spectrum of the audio signal includes using a constant-Q transform.

5. The method according to claim 1, wherein: Determining the logarithmic spectrum of the audio signal comprises using a Fourier transform, and determining the inverse transform using an inverse Fourier transform.

6. The method according to claim 1, wherein: The timbre attributes include timbre attributes that are not related to pitch.

7. The method according to claim 6, wherein: Determining the pitch-independent timbre property of the audio signal is based on an inverse transformation of a complex argument of the transformation of the log spectrum.

8. The method according to claim 6, wherein: The pitch-independent timbre attribute comprises a pitch-independent timbre logarithmic spectrum.

9. The method according to claim 1, wherein: The method also includes receiving an audio signal via ambient audio through a microphone.

10. The method according to claim 1, wherein: The audio signal corresponds to a media signal to be output by a media output device.

11. A non-transitory computer-readable storage medium comprising instructions that, when executed, cause one or more processors to perform a set of operations comprising: determining the log spectrum of an audio signal; transforming the logarithmic spectrum of the audio signal into the frequency domain to generate a transformed output; determining a magnitude of the transformed output; as well as A timbral attribute of the audio signal is determined based on an inverse transform of the amplitude.

12. The non-transitory computer-readable storage medium of claim 11, wherein: The audio signal is part of the media signal.

13. The non-transitory computer-readable storage medium of claim 11, wherein: The audio signal is an audio component of a video signal, and the operations further include extracting the audio signal from the video signal.

14. The non-transitory computer-readable storage medium of claim 11, wherein: Determining the logarithmic spectrum of the audio signal includes using a constant-Q transform.

15. The non-transitory computer-readable storage medium of claim 11, wherein: Determining the logarithmic spectrum of the audio signal comprises using a Fourier transform, and determining the inverse transform using an inverse Fourier transform.

16. The non-transitory computer-readable storage medium of claim 11, wherein: The timbre attributes include timbre attributes that are not related to pitch.

17. The non-transitory computer-readable storage medium of claim 16, wherein: Determining the pitch-independent timbre property of the audio signal is based on an inverse transformation of a complex argument of the transformation of the log spectrum.

18. The non-transitory computer-readable storage medium of claim 16, wherein: The pitch-independent timbre attribute comprises a pitch-independent timbre logarithmic spectrum.

19. The non-transitory computer-readable storage medium of claim 11, wherein: The audio signal corresponds to a media signal to be output by a media output device.

20. A computing device, the computing device comprising: one or more processors; as well as A non-transitory computer-readable storage medium comprising instructions that, when executed, cause the one or more processors to perform a set of operations comprising: determining the log spectrum of an audio signal; transforming the logarithmic spectrum of the audio signal into the frequency domain to generate a transformed output; determining a magnitude of the transformed output; and A timbral attribute of the audio signal is determined based on an inverse transform of the amplitude.

Images