Method for audio fingerprint recognition, computer readable medium and computing device

Abstract

The invention discloses a method for audio fingerprint recognition, a computer readable medium and a computing device. The method comprises: transforming an audio signal to a frequency domain, the transformed audio signal comprising a plurality of time-frequency bins, each time-frequency bin corresponding to an intersection of a frequency bin and a time bin and comprising a portion of the audio signal; determining an audio characteristic of an audio region comprising a set of a plurality of time-frequency bins, the set of the plurality of time-frequency bins comprising a first time-frequency bin and a time-frequency bin within a predefined distance of the first time-frequency bin; normalizing the audio regions to generate normalized energy values, the normalized audio regions including normalizing each portion of the audio signal for each time-frequency bin based on the determined audio characteristics associated with the audio regions; selecting one of the normalized energy values based on the category of the audio signal; a fingerprint of the audio signal is generated using the selected one normalized energy value.

Description

[0001] This application is a divisional application of the invention patent application with the original application number 201980072112.9 (International application number: PCT / US2019 / 049953, international application date; September 6, 2019, invention name: Method and device for fingerprinting audio signals via normalization). Technical Field

[0002] The present disclosure relates generally to audio signals, and more particularly, to methods and apparatus for fingerprinting audio signals via normalization. Background Art

[0003] Audio information (e.g., sound, speech, music, etc.) can be represented as digital data (e.g., electronic, optical, etc.). Audio captured (e.g., via a microphone) can be digitized, electronically stored, processed, and / or classified. One method of classifying audio information is by generating an audio fingerprint. An audio fingerprint is a digital summary of the audio information created by sampling a portion of an audio signal. Audio fingerprints have historically been used to identify audio and / or verify the authenticity of audio. Summary of the invention

[0004] According to one aspect, a method for audio fingerprinting is provided, the method comprising the following steps:

[0005] transforming an audio signal into a frequency domain, wherein the transformed audio signal comprises a plurality of time-frequency bins, wherein each time-frequency bin of the plurality of time-frequency bins corresponds to an intersection of a frequency bin and a time bin and comprises a portion of the audio signal;

[0006] determining audio characteristics of an audio region comprising a set of a plurality of time-frequency bins, wherein the set of a plurality of time-frequency bins comprises a first time-frequency bin and a time-frequency bin within a predefined distance of the first time-frequency bin;

[0007] normalizing the audio region to generate a normalized energy value, wherein normalizing the audio region comprises normalizing each portion of the audio signal for each time-frequency bin in the set of a plurality of time-frequency bins based on the determined audio characteristic associated with the audio region;

[0008] Selecting one of the normalized energy values ​​based on the category of the audio signal; and

[0009] A fingerprint of the audio signal is generated using the selected one of the normalized energy values.

[0010] The determined audio characteristics of the audio area include at least one of the following items: (i) mean energy value, (ii) mode energy value, (iii) average power value, (iv) mode power value; or (v) mean amplitude of the set of multiple time-frequency bins.

[0011] The method further comprises:

[0012] determining a plurality of audio characteristics, each audio characteristic corresponding to an audio region of a plurality of audio regions, wherein each audio region of the plurality of audio regions comprises a set of time-frequency bins, the set of time-frequency bins comprising a particular time-frequency bin and a time-frequency bin within a predefined distance of the particular time-frequency bin; and

[0013] Each audio region of the plurality of audio regions is normalized based on the associated determined audio characteristic of the audio region.

[0014] wherein a first audio region of the plurality of audio regions comprises a first set of time-frequency bins, wherein a second audio region of the plurality of audio regions comprises a second set of time-frequency bins, wherein at least a portion of the first set of time-frequency bins overlaps with at least a portion of the second set of time-frequency bins.

[0015] The category of the audio signal includes at least one of music, human voice, sound effect, or advertisement.

[0016] The selecting of one of the normalized energy values ​​is based on an energy extreme value of the normalized audio region.

[0017] Wherein, selecting one of the normalized energy values ​​comprises:

[0018] Determining a category of the audio signal;

[0019] weighting each time-frequency bin in the set of the plurality of time-frequency bins based on the determined class of the audio signal; and

[0020] The selection of one of the normalized energy values ​​is weighted by the category of the audio signal.

[0021] According to another aspect, a tangible, non-transitory computer-readable medium is provided, the tangible, non-transitory computer-readable medium comprising instructions that, when executed, cause at least one processor to perform a set of operations comprising:

[0022] transforming an audio signal into a frequency domain, wherein the transformed audio signal comprises a plurality of time-frequency bins, wherein each time-frequency bin of the plurality of time-frequency bins corresponds to an intersection of a frequency bin and a time bin and contains a portion of the audio signal;

[0023] determining audio characteristics of an audio region comprising a set of a plurality of time-frequency bins, wherein the set of a plurality of time-frequency bins comprises a first time-frequency bin and a time-frequency bin within a predefined distance of the first time-frequency bin;

[0024] normalizing the audio region to generate a normalized energy value, wherein normalizing the audio region comprises normalizing each portion of the audio signal for each time-frequency bin in the set of a plurality of time-frequency bins based on the determined audio characteristic associated with the audio region;

[0025] selecting one of the normalized energy values ​​based on the category of the audio signal; and

[0026] A fingerprint of the audio signal is generated using the selected one of the normalized energy values.

[0027] The determined audio characteristics of the audio area include at least one of the following: (i) mean energy value, (ii) mode energy value, (iii) average power value, (iv) mode power value, or (v) mean amplitude of the set of multiple time-frequency bins.

[0028] The tangible, non-transitory computer-readable medium further includes:

[0029] determining a plurality of audio characteristics, each audio characteristic corresponding to an audio region of a plurality of audio regions, wherein each audio region of the plurality of audio regions comprises a set of time-frequency bins, the set of time-frequency bins comprising a particular time-frequency bin and a time-frequency bin within a predefined distance of the particular time-frequency bin; and

[0030] Each audio region of the plurality of audio regions is normalized based on an associated determined audio characteristic of the audio region.

[0031] wherein a first audio region of the plurality of audio regions comprises a first set of time-frequency bins, wherein a second audio region of the plurality of audio regions comprises a second set of time-frequency bins, wherein at least a portion of the first set of time-frequency bins overlaps with at least a portion of the second set of time-frequency bins.

[0032] The category of the audio signal includes at least one of music, human speech, sound effects, or advertisements.

[0033] The selecting of one of the normalized energy values ​​is based on an energy extreme value of the normalized audio region.

[0034] Wherein, selecting one of the normalized energy values ​​comprises:

[0035] Determining a category of the audio signal;

[0036] weighting each time-frequency bin in the set of the plurality of time-frequency bins based on the determined class of the audio signal; and

[0037] The selection of one of the normalized energy values ​​is weighted by the class of the audio signal.

[0038] According to yet another aspect, there is provided a computing device, the computing device comprising:

[0039] at least one processor; and

[0040] A tangible, non-transitory computer-readable medium comprising instructions that, when executed, cause the at least one processor to perform a set of operations comprising:

[0041] transforming an audio signal into a frequency domain, wherein the transformed audio signal comprises a plurality of time-frequency bins, wherein each time-frequency bin of the plurality of time-frequency bins corresponds to an intersection of a frequency bin and a time bin and contains a portion of the audio signal;

[0042] determining audio characteristics of an audio region comprising a set of a plurality of time-frequency bins, wherein the set of a plurality of time-frequency bins comprises a first time-frequency bin and a time-frequency bin within a predefined distance of the first time-frequency bin;

[0043] normalizing the audio region to generate a normalized energy value, wherein normalizing the audio region comprises normalizing each portion of the audio signal for each time-frequency bin in the set of a plurality of time-frequency bins based on the determined audio characteristic associated with the audio region;

[0044] selecting one of the normalized energy values ​​based on the category of the audio signal; and

[0045] A fingerprint of the audio signal is generated using the selected one of the normalized energy values.

[0046] The determined audio characteristics of the audio area include at least one of the following: (i) mean energy value, (ii) mode energy value, (iii) average power value, (iv) mode power value, or (v) mean amplitude of the set of multiple time-frequency bins.

[0047] The set of operations further includes:

[0048] determining a plurality of audio characteristics, each audio characteristic corresponding to an audio region of a plurality of audio regions, wherein each audio region of the plurality of audio regions comprises a set of time-frequency bins, the set of time-frequency bins comprising a particular time-frequency bin and a time-frequency bin within a predefined distance of the particular time-frequency bin; and

[0049] Each audio region of the plurality of audio regions is normalized based on the associated determined audio characteristic of the audio region.

[0050] wherein a first audio region of the plurality of audio regions comprises a first set of time-frequency bins, wherein a second audio region of the plurality of audio regions comprises a second set of time-frequency bins, wherein at least a portion of the first set of time-frequency bins overlaps with at least a portion of the second set of time-frequency bins.

[0051] The category of the audio signal includes at least one of music, human voice, sound effect, or advertisement.

[0052] Wherein, selecting one of the normalized energy values ​​comprises:

[0053] Determining a category of the audio signal;

[0054] weighting each time-frequency bin in the set of the plurality of time-frequency bins based on the determined class of the audio signal; and

[0055] The selection of one of the normalized energy values ​​is weighted by the class of the audio signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is an example system in which the teachings of the present disclosure may be implemented.

[0057] Figure 2 yes Figure 1 An example implementation of an audio processor.

[0058] Figure 3A and Figure 3B Depicted by Figure 2 Example unprocessed spectrogram generated by the frequency range separator.

[0059] Figure 3C Depicted by Figure 2 The signal normalizer is based on Figure 3A and Figure 3B Example of a normalized spectrogram generated from an unprocessed spectrogram of .

[0060] Figure 4 is divided into fixed audio signal frequency components, Figure 3A and Figure 3B An example of an unprocessed spectrogram.

[0061] Figure 5 Is Figure 2 The signal normalizer is based on Figure 4 An example of a normalized spectrogram generated for a fixed audio signal frequency component.

[0062] Figure 6 Is Figure 2 The point selector is based on Figure 5 Example of a normalized and weighted spectrogram generated by a normalized spectrogram of .

[0063] Figure 7 and Figure 8 It means that it can be executed to achieve Figure 2 A flow chart of machine readable instructions for an audio processor.

[0064] Fig. 9 is constructed to execute Figure 7 and Figure 8 Instructions to achieve Figure 2 A block diagram of an example processing platform for an audio processor.

[0065] The drawings are not drawn to scale. In general, the same reference numbers will be used throughout the drawings and accompanying written description to refer to the same or like parts. DETAILED DESCRIPTION

[0066] Fingerprint or signature-based media monitoring techniques typically exploit one or more inherent characteristics of the media being monitored during a monitoring time interval to generate a substantially unique proxy for the media. Such a proxy is called a signature or fingerprint, and can take any form (e.g., a series of digital values, a waveform, etc.) that represents any aspect of a media signal (e.g., an audio signal and / or a video signal that forms a presentation of the media being monitored). A signature can be a series of signatures collected continuously over a time interval. The terms "fingerprint" and "signature" are used interchangeably herein and are defined herein to mean a proxy generated based on one or more inherent characteristics of the media for identifying the media.

[0067] Signature-based media monitoring generally involves determining (e.g., generating and / or collecting) a signature representing a media signal (e.g., an audio signal and / or a video signal) output by a monitored media device, and comparing the monitored signature with one or more reference signatures corresponding to a known (e.g., reference) media source. Various comparison criteria (such as cross-correlation values, Hamming distances, etc.) may be evaluated to determine whether the monitored signature matches a particular reference signature.

[0068] When a match is found between a monitored signature and one of the reference signatures, the monitored media can be identified as corresponding to a particular reference media represented by the reference signature that matches the monitored signature. Since attributes such as media identifier, presentation time, broadcast channel, etc. are collected for the reference signatures, these attributes can then be associated with the monitored media whose monitored signature matches the reference signature. Example systems for identifying media based on codes and / or signatures are already known and were first disclosed in U.S. Pat. No. 5,481,294 to Thomas, the entire contents of which are incorporated herein by reference.

[0069] Historically, audio fingerprinting techniques have used the loudest parts of an audio signal (e.g., the parts with the most energy, etc.) to create a fingerprint over a period of time. However, in some cases, this approach has several serious limitations. In some examples, the loudest parts of an audio signal may be associated with noise (e.g., unwanted audio) rather than from audio of interest. For example, if a user attempts to fingerprint a song in a noisy restaurant, the loudest parts of the captured audio signal may be conversations between restaurant patrons rather than the song or media to be identified. In this example, many sampled portions of the audio signal will have background noise rather than music, which reduces the usefulness of the generated fingerprint.

[0070] Another potential limitation of previous fingerprinting techniques is that, particularly in music, audio in the bass frequency range tends to be the loudest. In some examples, the dominant bass frequency energy results in the sampled portion of the audio signal being primarily in the bass frequency range. As a result, fingerprints generated using existing methods typically do not include samples from all parts of the audio spectrum that can be used for signature matching, especially samples in higher frequency ranges (e.g., treble range, etc.).

[0071] The example methods and devices disclosed herein overcome the above problems by generating fingerprints from audio signals using mean normalization. An example method includes normalizing one or more time-frequency bins of an audio signal according to the audio characteristics of the surrounding audio area. As used herein, a "time-frequency bin" is a portion of an audio signal that corresponds to a specific frequency bin (e.g., an FFT bin) at a specific time (e.g., three seconds into the audio signal). In some examples, the normalization is weighted by the audio category of the audio signal. In some examples, a fingerprint is generated by selecting points from the normalized time-frequency bins.

[0072] Another example method disclosed herein includes dividing an audio signal into two or more audio signal frequency components. As used herein, an "audio signal frequency component" is a portion of an audio signal corresponding to a frequency range and a time period. In some examples, the audio signal frequency component may be composed of a plurality of time-frequency bins. In some examples, audio characteristics are determined for some of the audio signal frequency components. In this example, each of the audio signal frequency components is normalized by an associated audio characteristic (e.g., an audio mean, etc.). In some examples, a fingerprint is generated by selecting a point from the normalized audio signal frequency component.

[0073] Figure 1 1 is an example system 100 in which the teachings of the present disclosure may be implemented. Example system 100 includes example audio source 102, example microphone 104 that captures sound from audio source 102 and converts the captured sound into example audio signal 106. Example audio processor 108 receives audio signal 106 and generates example fingerprint 110.

[0074] An example audio source 102 emits audible sounds. An example audio source may be a speaker (e.g., an electroacoustic transducer, etc.), a live performance, a conversation, and / or any other suitable audio source. An example audio source 102 may include desired audio (e.g., audio to be fingerprinted, etc.) and may also include undesired audio (e.g., background noise, etc.). In the example shown, the audio source 102 is a speaker. In other examples, the audio source 102 may be any other suitable audio source (e.g., a person, etc.).

[0075] The example microphone 104 is a transducer that converts sound emitted by the audio source 102 into an audio signal 106. In some examples, the microphone 104 can be a component of a computer, a mobile device (smartphone, tablet, etc.), a navigation device, or a wearable device (e.g., a smart watch, etc.). In some examples, the microphone can include an audio-to-digital conversion to digitize the audio signal 106. In other examples, the audio processor 108 can digitize the audio signal 106.

[0076] The example audio signal 106 is a digitized representation of the sound emitted by the audio source 102. In some examples, the audio signal 106 can be stored on a computer before being processed by the audio processor 108. In some examples, the audio signal 106 can be transmitted to the example audio processor 108 via a network. Additionally or alternatively, any other suitable method can be used to generate audio (e.g., digital synthesis, etc.).

[0077] The example audio processor 108 converts the example audio signal 106 into an example fingerprint 110. In some examples, the audio processor 108 divides the audio signal 106 into frequency bins and / or time bins and then determines the mean energy of one or more of the created audio signal frequency components. In some examples, the audio processor 108 may normalize the audio signal frequency components using the associated mean energy of the audio region surrounding each time-frequency bin. In other examples, any other suitable audio characteristics may be determined and used to normalize each time-frequency bin. In some examples, the fingerprint 110 may be generated by selecting the highest energy among the normalized audio signal frequency components. Additionally or alternatively, any suitable means may be used to generate the fingerprint 110. The following description will now be provided in conjunction with the description of the example audio processor 108. Figure 2 An example implementation of the audio processor 108 is described.

[0078] The example fingerprint 110 is a concise digital summary of the audio signal 106, which can be used to identify and / or verify the audio signal 106. For example, the fingerprint 110 can be generated by sampling multiple parts of the audio signal 106 and processing these parts. In some examples, the fingerprint 110 can include samples of the highest energy parts of the audio signal 106. In some examples, the fingerprint 110 can be indexed in a database, which can be used for comparison with other fingerprints. In some examples, the fingerprint 110 can be used to identify the audio signal 106 (e.g., determine what song is playing, etc.). In some examples, the fingerprint 110 can be used to verify the authenticity of the audio.

[0079] Figure 2 yes Figure 1The example audio processor 108 includes an example frequency range separator 202, an example audio characteristic determiner 204, an example signal normalizer 206, an example point selector 208, and an example fingerprint generator 210.

[0080] The example frequency range separator 202 separates the audio signal (e.g., Figure 1 The digitized audio signal 106) is divided into time-frequency bins and / or audio signal frequency components. For example, the frequency range separator 202 can perform a fast Fourier transform (FFT) on the audio signal 106 to transform the audio signal 106 into the frequency domain. Additionally, the example frequency range separator 202 can divide the transformed audio signal 106 into two or more frequency bins (e.g., using a Hamming function, a Hann function, etc.). In this example, each audio signal frequency component is associated with a frequency bin in two or more frequency bins. Additionally or alternatively, the frequency range separator 202 can aggregate the audio signal 106 into one or more time periods (e.g., the duration of the audio, a six-second period, a 1-second period, etc.). In other examples, the frequency range separator 202 can use any suitable technology to transform the audio signal 106 (e.g., discrete Fourier transform, sliding time window Fourier transform, wavelet transform, discrete Hadamard transform, discrete Walsh Hadamard, discrete cosine transform, etc.). In some examples, the frequency range separator 202 can be implemented by one or more band pass filters (BPF). In some examples, the output of the example frequency range separator 202 can be represented by a spectrum diagram. FIG. 3A to FIG. 3B as well as Figure 4 Example outputs of the frequency range separator 202 are discussed.

[0081] The example audio characteristic determiner 204 determines audio characteristics of a portion of the audio signal 106 (e.g., an audio signal frequency component, an audio region around a time-frequency bin, etc.). For example, the audio characteristic determiner 204 may determine a mean energy (e.g., average power, etc.) of one or more of the audio signal frequency components. Additionally or alternatively, the audio characteristic determiner 204 may determine other characteristics of a portion of the audio signal (e.g., mode energy, median energy, mode power, median energy, mean energy, mean amplitude, etc.).

[0082] The example signal normalizer 206 normalizes one or more time-frequency bins by the associated audio characteristics of the surrounding audio area. For example, the signal normalizer 206 may normalize the time-frequency bins by the mean energy of the surrounding audio area. In other examples, the signal normalizer 206 normalizes some of the audio signal frequency components by the associated audio characteristics. For example, the signal normalizer 206 may use the mean energy associated with the audio signal frequency component to normalize each time-frequency bin of the audio signal frequency component. In some examples, the output of the signal normalizer 206 (e.g., normalized time-frequency bins, normalized audio signal frequency components, etc.) may be represented as a spectrogram. The following is a schematic diagram of the output of the signal normalizer 206 in conjunction with the audio signal normalizer 206. Figure 3C and Figure 5 Example outputs of the signal normalizer 206 are discussed.

[0083] The example point selector 208 selects one or more points from the normalized audio signal to be used to generate the fingerprint 110. For example, the example point selector 208 can select a plurality of energy maxima of the normalized audio signal. In other examples, the point selector 208 can select any other suitable point of the normalized audio.

[0084] Additionally or alternatively, the point selector 208 can weight the selection of the points based on the category of the audio signal 106. For example, if the category of the audio signal is music, the point selector 208 can focus the selection of the points on common frequency ranges of music (e.g., bass, treble, etc.). In some examples, the point selector 208 can determine the category of the audio signal (e.g., music, speech, sound effects, advertisements, etc.). The example fingerprint generator 210 generates a fingerprint (e.g., fingerprint 110) using the points selected by the example point selector 208. The example fingerprint generator 210 can generate the fingerprint based on the selected points using any suitable method.

[0085] Although Figure 2 Demonstrates the implementation Figure 1 An example of an audio processor 108, but Figure 2 One or more of the illustrated elements, processes and / or devices may be combined, divided, rearranged, omitted, eliminated and / or implemented in any other manner. In addition, the example frequency range separator 202, the example audio characteristics determiner 204, the example signal normalizer 206, the example point selector 208 and the example fingerprint generator 210 and / or more generally Figure 1 and Figure 2The example audio processor 108 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, any of the example frequency range separator 202, the example audio characteristic determiner 204, the example signal normalizer 206, the example point selector 208, and the example fingerprint generator 210, and / or more generally the example audio processor 108 may be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field programmable logic devices (FPLDs). When any of the apparatus or system claims of this patent is understood to cover pure software and / or firmware implementations, at least one of the example frequency range separator 202, the example audio characteristics determiner 204, the example signal normalizer 206, the example point selector 208, and the example fingerprint generator 210 is thereby expressly defined to include a non-transitory computer-readable storage device or storage disk (such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc.) having software and / or firmware. Further, Figure 1 and Figure 2 An example audio processor 106 may include in addition to Figure 2 Elements, processes and / or devices other than or in place of the illustrated elements, processes and / or devices Figure 2 One or more of the illustrated elements, processes and / or devices, and / or may include more than one of any or all of the illustrated elements, processes and devices. As used herein, the phrase "communicating" (including variations thereof) encompasses direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or continuous communication, but additionally includes selective communication at regular intervals, scheduled intervals, non-periodic intervals and / or one-time events.

[0086] FIG. 3A to FIG. 3B Depicted by Figure 2 An example of an unprocessed spectrogram generated by an example frequency range separator is shown in Figure 300. Figure 3A In the illustrated example of , the example unprocessed spectrogram 300 includes an example first time-frequency bin 304A surrounded by an example first audio region 306A. Figure 3B In the illustrated example of , the example unprocessed spectrogram includes an example second time-frequency bin 304B surrounded by an example audio region 306B. Figure 3A and Figure 3B The example unprocessed spectrogram 300 and the normalized spectrogram 302 each include an example vertical axis 308 representing frequency bins and an example horizontal axis 310 representing time bins. Figure 3A and Figure 3B Example audio regions 306A and 306B are illustrated from which the audio characteristic determiner 204 obtains normalized audio characteristics and the signal normalizer 206 uses the normalized audio characteristics to normalize the first time-frequency bin 304A and the second time-frequency bin 304B, respectively. In the illustrated example, each time-frequency bin of the unprocessed spectrogram 300 is normalized to generate the normalized spectrogram 302. In other examples, any suitable number of time-frequency bins of the unprocessed spectrogram 300 may be normalized to generate the normalized spectrogram 302. Figure 3C A normalized spectrum diagram 302 of .

[0087] The example vertical axis 308 has units of frequency bins generated by a Fast Fourier Transform (FFT) and has a length of 1024 FFT bins. In other examples, the example vertical axis 308 may be measured by any other suitable technique for measuring frequency (e.g., Hertz, another transform algorithm, etc.). In some examples, the vertical axis 308 covers the entire frequency range of the audio signal 106. In other examples, the vertical axis 308 may cover a portion of the audio signal 106.

[0088] In the example shown, the example horizontal axis 310 represents a time period of 11.5 seconds of the total length of the unprocessed spectrogram 300. In the example shown, the horizontal axis 310 has sixty-four millisecond (ms) intervals as units. In other examples, the horizontal axis 310 can be measured in any other suitable units (e.g., 1 second, etc.). For example, the horizontal axis 310 covers the complete duration of the audio. In other examples, the horizontal axis 310 can cover a portion of the duration of the audio signal 106. In the example shown, the size of each time-frequency bin of the spectrogram 300, 302 is 64ms×1FFT bin.

[0089] exist Figure 3AIn the illustrated example of , the first time-frequency bin 304A is associated with the intersection of the frequency bin and the time bin of the unprocessed spectrogram 300 and a portion of the audio signal 106 associated with the intersection. The example first audio region 306A includes time-frequency bins within a predefined distance from the example first time-frequency bin 304A. For example, the audio characteristic determiner 204 can determine the vertical length of the first audio region 306A (e.g., the length of the first audio region 306A along the longitudinal axis 308, etc.) based on a set number of FFT bins (e.g., 5 bins, 11 bins, etc.). Similarly, the audio characteristic determiner 204 can determine the horizontal length of the first audio region 306A (e.g., the length of the first audio region 306A along the horizontal axis 310, etc.). In the illustrated example, the first audio region 306A is square. Alternatively, the first audio region 306A can be any suitable size and shape and can contain any suitable combination of time-frequency bins within the unprocessed spectrogram 300 (e.g., any suitable time-frequency bin group, etc.). The example audio characteristic determiner 204 may then determine audio characteristics (eg, mean energy, etc.) of the time-frequency bins contained within the first audio region 306A. Using the determined audio characteristics, Figure 2 The example signal normalizer 206 may normalize the associated values ​​of the first time-frequency bin 304A (eg, the energy of the first time-frequency bin 304A may be normalized by the mean energy of each time-frequency bin within the first audio region 306A).

[0090] exist Figure 3BIn the illustrated example of , the second time-frequency bin 304B is associated with the intersection of the frequency bin and the time bin of the unprocessed spectrogram 300 and a portion of the audio signal 106 associated with the intersection. The example second audio region 306B includes time-frequency bins within a predefined distance from the example second time-frequency bin 304B. Similarly, the audio characteristic determiner 204 can determine the horizontal length of the second audio region 306B (e.g., the length of the second audio region 306B along the horizontal axis 310, etc.). In the illustrated example, the second audio region 306B is square. Alternatively, the second audio region 306B can be any suitable size and shape, and can include any suitable combination of time-frequency bins within the unprocessed spectrogram 300 (e.g., any suitable time-frequency bin group, etc.). In some examples, the second audio region 306B can overlap with the first audio region 306A (e.g., include some of the same time-frequency bins, shift on the horizontal axis 310, shift on the vertical axis 308, etc.). In some examples, the second audio region 306B may have the same size and shape as the first audio region 306A. In other examples, the second audio region 306B may have a different size and shape than the first audio region 306A. The example audio characteristics determiner 204 may then determine audio characteristics (e.g., mean energy, etc.) of the time-frequency bins contained in the second audio region 306B. Using the determined audio characteristics, Figure 2 The example signal normalizer 206 may normalize the associated values ​​of the second time-frequency bin 304B (eg, the energy of the second time-frequency bin 304B may be normalized by the mean energy of the bins located within the second audio region 306B).

[0091] Figure 3C Depicted by Figure 2 The signal normalizer is FIG. 3A to FIG. 3B 304A and 304B are normalized. Figure 7 An example process 700 for generating a normalized spectrogram is described. Figure 3C The resulting frequency bins are normalized. As a result, the darker regions are the regions with the maximum energy in their respective local regions. This enables the fingerprint to contain relevant audio features in regions with even lower energy than the typically louder bass frequency regions.

[0092] Figure 4An example of a fixed audio signal divided into multiple frequency components FIG. 3A to FIG. 3C The example unprocessed spectrogram 300 is generated by processing the audio signal 106 using a fast Fourier transform (FFT). In other examples, any other suitable method may be used to generate the unprocessed spectrogram 300. In this example, the unprocessed spectrogram 300 is divided into a plurality of example audio signal frequency components 402. The example unprocessed spectrogram 400 includes FIG. 3A to FIG. 3C An example vertical axis 308 and FIG. 3A to FIG. 3C 310. In the illustrated example, the example audio signal frequency components 402 each have an example frequency range 408 and an example time period 410. The example audio signal frequency components 402 include an example first audio signal frequency component 412A and an example second audio signal frequency component 412B. In the illustrated example, the darker portions of the unprocessed spectrogram 300 represent portions of the audio signal 106 that have higher energy.

[0093] The example audio signal frequency components 402 are each associated with a unique combination of a continuous frequency range (e.g., frequency bins, etc.) and a continuous time period. In the illustrated example, each audio signal frequency component in the audio signal frequency components 402 has a frequency bin of equal size (e.g., frequency range 408). In other examples, some or all of the audio signal frequency components 402 may have frequency bins of different sizes. In the illustrated example, each audio signal frequency component in the audio signal frequency components 402 has a time period of equal duration (e.g., time period 410). In other examples, some or all of the audio signal frequency components 402 may have time periods of different durations. In the illustrated example, the audio signal frequency components 402 constitute the entire audio signal 106. In other examples, the audio signal frequency components 402 may include a portion of the audio signal 106.

[0094] In the illustrated example, the first audio signal frequency component 412A is located in the high frequency range of the audio signal 106 and has no visible energy points. The example first audio signal frequency component 412A is associated with a frequency bin between the 768 FFT bin and the 896 FFT bin and a time period between 10024 ms and 11520 ms. In some examples, multiple portions of the audio signal 106 exist within the first audio signal frequency component 412A. In this example, since the audio within the bass spectrum of the audio signal 106 (e.g., the audio in the second audio signal frequency component 412B, etc.) has relatively high energy, the portion of the audio signal 106 located within the audio signal frequency component 412A is not visible. The second audio signal frequency component 412B is located in the bass range of the audio signal 106 and has a visible energy point. The example second audio signal frequency component 412B is associated with a frequency bin between the 128 FFT bin and the 256 FFT bin and a time period between 10024 ms and 11520 ms. In some examples, because portions of the audio signal 106 that are within the bass spectrum (e.g., the second audio signal frequency component 412B, etc.) have relatively high energy, a fingerprint generated from the unprocessed spectrogram 300 will include a disproportionate number of samples from the bass spectrum.

[0095] Figure 5 Is Figure 2 The signal normalizer is based on Figure 4 An example of a normalized spectrogram 500 generated from a fixed audio signal frequency component. The example normalized spectrogram 500 includes FIG. 3A to FIG. 3C An example vertical axis 308 and FIG. 3A to FIG. 3C 408. The example normalized spectrogram 500 is divided into a plurality of example audio signal frequency components 502. In the example shown, the audio signal frequency components 502 each have an example frequency range 408 and an example time period 410. The example audio signal frequency components 502 include an example first audio signal frequency component 504A and an example second audio signal frequency component 504B. In some examples, the first audio signal frequency component 504A and the second audio signal frequency component 504B correspond to FIG. 3A to FIG. 3C The first audio signal frequency component 412A and the second audio signal frequency component 412B are in the same frequency bin and time period. In the example shown, the darker portions of the normalized spectrogram 500 represent areas of the audio spectrum with higher energy.

[0096] By sorting by associated audio characteristics Figure 4The unprocessed spectrogram 300 may be normalized by normalizing the individual audio signal frequency components 402 of the example normalized spectrogram 500. For example, the audio characteristic determiner 204 may determine the audio characteristic (e.g., mean energy, etc.) of the first audio signal frequency component 412A. In this example, the signal normalizer 206 may then normalize the first audio signal frequency component 412A according to the determined audio characteristic to create the example audio signal frequency component 402A. Similarly, the example normalized spectrogram 500 may be normalized by normalizing the first audio signal frequency component 412A according to the determined audio characteristic. Figure 4 The example second audio signal frequency component 402B is generated by normalizing the second audio signal frequency component 412B according to the audio characteristics associated with the second audio signal frequency component 412B. In other examples, the normalized spectrogram 500 can be generated by normalizing a portion of the audio signal frequency component 402. In other examples, any other suitable method can be used to generate the example normalized spectrogram 500.

[0097] exist Figure 5 In the example shown in FIG. 1 , the first audio signal frequency component 504A (eg, processed by the signal normalizer 206) Figure 4 The first audio signal frequency component 412A of the first audio signal 106 (e.g., the first audio signal frequency component 412A of the first audio signal 106) has a visible energy point on the normalized spectrogram 500. For example, because the first audio signal frequency component 504A has been normalized by the energy of the first audio signal frequency component 412A, a previously hidden portion of the audio signal 106 (e.g., when compared to the first audio signal frequency component 412A) is visible on the normalized spectrogram 500. The second audio signal frequency component 504B (e.g., after being processed by the signal normalizer 206) Figure 4 The second audio signal frequency component 412B, etc.) corresponds to the bass range of the audio signal 106. For example, because the second audio signal frequency component 504B has been normalized by the energy of the second audio signal frequency component 412B, the number of visible energy points has been reduced (e.g., when compared to the second audio signal frequency component 412B). Figure 4 Compared with the fingerprint generated from the unprocessed spectrogram 300, the fingerprint generated from the normalized spectrogram 500 (e.g., Figure 1 The fingerprint 110) will include samples that are more evenly distributed in the audio spectrum.

[0098] Figure 6 Is Figure 2 The point selector 208 is based on Figure 5 An example of a normalized and weighted spectrogram 600 generated from the normalized spectrogram 500 of FIG. The example spectrogram 600 includes FIG. 3A to FIG. 3C An example vertical axis 308 and FIG. 3A to FIG. 3C310. The example normalized and weighted spectrogram 600 is divided into a plurality of example audio signal frequency components 502. In the example shown, the example audio signal frequency components 502 each have an example frequency range 408 and an example time period 410. The example audio signal frequency components 502 include an example first audio signal frequency component 604A and an example second audio signal frequency component 604B. In some examples, the first audio signal frequency component 604A and the second audio signal frequency component 604B correspond to FIG. 3A to FIG. 3C The first audio signal frequency component 412A and the second audio signal frequency component 412B are in the same frequency bin and time period. In the example shown, the darker portions of the normalized and weighted spectrogram 600 represent areas of the audio spectrum with higher energy.

[0099] The example normalized and weighted spectrogram 600 is generated by weighting the normalized spectrogram 600 with a value range from zero to one based on the category of the audio signal 106. For example, if the audio signal 106 is music, then Figure 2 The point selector 208 will weight the area of ​​the audio spectrum associated with the music along each column. In other examples, the weighting can be applied to multiple columns and can adopt different ranges from zero to one.

[0100] Figure 7 and Figure 8 shows a representation for implementing Figure 2 108 of the audio processor 108, a flowchart of example hardware logic, machine readable instructions, a hardware implemented state machine, and / or any combination thereof. The machine readable instructions may be a flowchart for a computer processor (such as the following in conjunction with Fig. 9 The program may be embodied in software stored on a non-transitory computer-readable storage medium such as a CD-ROM, floppy disk, hard drive, DVD, Blu-ray disk, or memory associated with the processor 912, but the entire program and / or portions thereof may alternatively be executed by a device other than the processor 912, and / or embodied in firmware or dedicated hardware. In addition, although reference is made to Figure 7 and Figure 8The illustrated flowchart describes an example program, but many other methods of implementing the example audio processor 108 may alternatively be used. For example, the order of execution of the blocks may be changed, and / or some of the blocks may be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are configured to perform corresponding operations without executing software or firmware.

[0101] As described above, the present invention may be implemented using executable instructions (e.g., computer and / or machine readable instructions) stored on a non-transitory computer and / or machine readable medium (such as a hard drive, flash memory, read-only memory, compact disk, digital versatile disk, cache, random access memory, and / or any other storage device or storage disk in which information is stored for any duration (e.g., for an extended period of time, permanently, for a simple instance, for temporary buffering, and / or for caching information)). Figure 7 and Figure 8 As used herein, the term non-transitory computer readable medium is expressly defined to include any type of computer readable storage devices and / or storage disks and to exclude propagating signals and to exclude transmission media.

[0102] "Include" and "comprising" (and all forms and tenses thereof) are used herein as open-ended terms. Thus, whenever a claim employs any form of "includes" or "comprising" (e.g., comprises, includes, comprising, including, having, etc.) as a preamble or within any type of claim recitation, it will be understood that additional elements, terms, etc. may be present without falling outside the scope of the corresponding claim or recitation. As used herein, when the phrase "at least" is used as a transitional term in, for example, a preamble of a claim, it is open-ended in the same manner as the terms "includes" and "comprising" are open-ended. When used, for example, in a form such as A, B, and / or C, the term "and / or" refers to any combination or subset of A, B, C, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, and (7) A and B and C. As used herein, in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A and B” is intended to mean implementations that include any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein, in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A or B” is intended to mean implementations that include any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. As used herein, in the context of describing the execution of processes, instructions, actions, activities, and / or steps, the phrase “at least one of A and B” is intended to mean implementations that include any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein, in the context of describing the execution of a process, instruction, action, activity, and / or step, the phrase "at least one of A or B" is intended to mean an implementation that includes any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B.

[0103] Figure 7 The processing of begins at block 702. At block 702, the audio processor 108 receives the digitized audio signal 106. For example, the audio processor 108 may receive the digitized audio signal 106 captured by the microphone 104 (e.g., Figure 1 In this example, the microphone may include an analog-to-digital converter to convert the audio into a digitized audio signal 106. In other examples, the audio processor 108 may receive audio stored in a database (e.g., Fig. 9 Volatile memory 914, Fig. 9 Non-volatile memory 916, Fig. 9 In other examples, the digitized audio signal 106 may be sent to the audio processor 108 via a network (e.g., the Internet, etc.). Additionally or alternatively, the audio processor 108 may receive the audio signal 106 via any other suitable means.

[0104] At block 704, the frequency range separator 202 performs windowing on the audio signal 106 and transforms the audio signal 106 to the frequency domain. For example, the frequency range separator 202 may perform a fast Fourier transform to transform the audio signal 106 to the frequency domain, and may perform a windowing function (e.g., a Hamming function, a Hann function, etc.). Additionally or alternatively, the frequency range separator 202 may aggregate the audio signal 106 into two or more time bins. In these examples, the time-frequency bin corresponds to the intersection of the frequency bin and the time bin and contains a portion of the audio signal 106.

[0105] At block 706, the audio characteristics determiner 204 selects a time-frequency bin for normalization. For example, the audio characteristics determiner 204 may select Figure 3A In some examples, the audio characteristics determiner 204 can select a time-frequency bin adjacent to the previously selected first time-frequency bin.

[0106] At block 708, the audio characteristic determiner 204 determines the audio characteristics of the surrounding audio region. For example, if the audio characteristic determiner 204 selected the first time-frequency bin 304A, the audio characteristic determiner 204 may determine the audio characteristics of the first audio region 306A. In some examples, the audio characteristic determiner 204 may determine the mean energy of the audio region. In other examples, the audio characteristic determiner 204 may determine any other suitable audio characteristic (e.g., mean amplitude, etc.).

[0107] At block 710, the audio characteristics determiner 204 determines that if another time-frequency bin is to be selected, the process 700 returns to block 706. If another time-frequency bin is not to be selected, the process 700 proceeds to block 712. In some examples, blocks 706 to 710 are repeated until every time-frequency bin of the unprocessed spectrogram 300 has been selected. In other examples, blocks 706 to 710 may be repeated for any suitable number of iterations.

[0108] At block 712, the signal normalizer 206 normalizes the time-frequency bins based on the associated audio characteristics. For example, the signal normalizer 206 may utilize the associated audio characteristics determined at block 708 to normalize the time-frequency bins in the time-frequency bins selected at block 706. For example, the signal normalizer may normalize the first time-frequency bin 304A and the second time-frequency bin 304B by the audio characteristics (e.g., mean energy) of the first audio region 306A and the second audio region 306B, respectively. In some examples, the signal normalizer 206 generates a normalized spectrogram (e.g., a frequency spectrogram) based on the normalization of the time-frequency bins. Figure 3C Normalized spectrum diagram 302).

[0109] At block 714, the point selector 208 determines if the fingerprint generation is to be weighted based on the audio category, and the process 700 proceeds to block 716. If the fingerprint generation is not to be weighted based on the audio category, the process 700 proceeds to block 720. At block 716, the point selector 208 determines the audio category of the audio signal 106. For example, the point selector 208 may present a prompt to the user to indicate the category of the audio (e.g., music, speech, sound effects, advertisements, etc.). In other examples, the audio processor 108 may determine the audio category using an audio category determination algorithm. In some examples, the audio category may be the voice of a particular person, general human speech, music, sound effects, and / or advertisements.

[0110] At block 718, the point selector 208 weights the time-frequency bins based on the determined audio category. For example, if the audio category is music, the point selector 208 may weight the audio signal frequency components in association with the high and low ranges typically associated with music. In some examples, if the audio category is a particular person's voice, the point selector 208 may weight the audio signal frequency components in association with the person's voice. In some examples, the output of the signal normalizer 206 may be represented as a spectrogram.

[0111] At block 720, the fingerprint generator 210 generates a fingerprint of the audio signal 106 by selecting energy extrema of the normalized audio signal (eg, Figure 110). For example, the fingerprint generator 210 may use frequency time bins and energies associated with one or more energy extremes (e.g., one extreme, twenty extremes, etc.). In some examples, the fingerprint generator 210 may select the energy maximum of the normalized audio signal 106. In other examples, the fingerprint generator 210 may select any other suitable feature of the normalized audio signal frequency components. In some examples, the fingerprint generator 210 may generate the fingerprint 110 representing the audio signal 106 using any suitable means (e.g., an algorithm, etc.). Once the fingerprint 110 is generated, the process 700 ends.

[0112] Figure 8 The process 800 begins at block 802. At block 802, the audio processor 108 receives a digitized audio signal. For example, the audio processor 108 may receive (e.g., Figure 1 In this example, the microphone may include an analog-to-digital converter to convert the audio signal into a digitized audio signal 106. In other examples, the audio processor 108 may receive audio data stored in a database (e.g., Fig. 9 Volatile memory 914, Fig. 9 Non-volatile memory 916, Fig. 9 In other examples, the digitized audio signal 106 may be sent to the audio processor 108 via a network (e.g., the Internet, etc.). Additionally or alternatively, the audio processor 108 may receive the audio signal 106 by any suitable means.

[0113] At block 804, the frequency range separator 202 separates the audio signal into two or more audio signal frequency components (e.g., FIG. 3A to FIG. 3C402, etc.). For example, the frequency range separator 202 can perform a fast Fourier transform to transform the audio signal 106 into the frequency domain, and can perform a windowing function (e.g., a Hamming function, a Hann function, etc.) to create frequency bins. In these examples, each audio signal frequency component is associated with one or more frequency bins in the frequency bins. Additionally or alternatively, the frequency range separator 202 can also divide the audio signal 106 into two or more time periods. In these examples, each audio signal frequency component corresponds to a unique combination of a time period in two or more time periods and a frequency bin in two or more frequency bins. For example, the frequency range separator 202 can divide the audio signal 106 into a first frequency bin, a second frequency bin, a first time period, and a second time period. In this example, the first audio signal frequency component corresponds to a portion of the audio signal 106 within a first frequency bin and a first time period, the second audio signal frequency component corresponds to a portion of the audio signal 106 within the first frequency bin and a second time period, the third audio signal frequency component corresponds to a portion of the audio signal 106 within a second frequency bin and the first time period, and the fourth audio signal frequency component corresponds to a component of the audio signal 106 within the second frequency bin and the second time period. In some examples, the output of the frequency range separator 202 can be represented as a spectrogram (e.g., FIG. 3A to FIG. 3C Unprocessed spectrum diagram 300).

[0114] At block 806, the audio characteristic determiner 204 determines the audio characteristics of each audio signal frequency component. For example, the audio characteristic determiner 204 may determine the mean energy of each audio signal frequency component. In other examples, the audio characteristic determiner 204 may determine any other suitable audio characteristic (e.g., mean amplitude, etc.).

[0115] At block 808, the signal normalizer 206 normalizes the audio signal frequency components based on the determined audio characteristics associated with the respective audio signal frequency components. For example, the signal normalizer 206 may normalize the audio signal frequency components by the mean energy associated with the respective audio signal frequency components. In other examples, the signal normalizer 206 may normalize the audio signal frequency components using any other suitable audio characteristics. In some examples, the output of the signal normalizer 206 may be represented as a spectrogram (e.g., Figure 5 Normalized spectrum diagram 500).

[0116] At block 810, the audio characteristic determiner 204 determines if the fingerprint generation is to be weighted based on the audio category, and the process 800 proceeds to block 812. If the fingerprint generation is not to be weighted based on the audio category, the process 800 proceeds to block 816. At block 812, the audio processor 108 determines the audio category of the audio signal 106. For example, the audio processor 108 may present a prompt to the user to indicate the category of the audio (e.g., music, speech, etc.). In other examples, the audio processor 108 may determine the audio category using an audio category determination algorithm. In some examples, the audio category may be the voice of a particular person, general human speech, music, sound effects, and / or advertisements.

[0117] At block 814, the signal normalizer 206 weights the audio signal frequency components based on the determined audio category. For example, if the audio category is music, the signal normalizer 206 may weight the audio signal frequency components along each column using a different scaling value from zero to one for each frequency position from high to low associated with the average spectral envelope of the music. In some examples, if the audio category is human voice, the signal normalizer 206 may weight the audio signal frequency components in association with the spectral envelope of the human voice. In some examples, the output of the signal normalizer 206 may be represented as a spectrogram (e.g., Figure 6 Spectrum diagram 600).

[0118] At block 816, the fingerprint generator 210 generates a fingerprint of the audio signal 106 by selecting energy extremes of the normalized audio signal frequency components (eg, Figure 1 10). For example, the fingerprint generator 210 may use frequency time bins and energies associated with one or more energy extremes (e.g., twenty extremes, etc.). In some examples, the fingerprint generator 210 may select the energy maximum of the normalized audio signal. In other examples, the fingerprint generator 210 may select any other suitable feature of the normalized audio signal frequency components. In some examples, the fingerprint generator 210 may generate the fingerprint 110 representing the audio signal 106 using another suitable means (e.g., an algorithm, etc.). Once the fingerprint 110 is generated, the process 800 ends.

[0119] Fig. 9 is constructed to execute Figure 7 and / or Figure 8 Instructions to achieve Figure 2 108. For example, the processor platform 900 may be a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smart phone, an iPod, etc.), or a processor platform 900. TMtablet computer), personal digital assistant (PDA), Internet appliance, DVD player, CD player, digital video recorder, Blu-ray player, game console, personal video recorder, set-top box, head-mounted device or other wearable device, or any other type of computing device.

[0120] The processor platform 900 of the illustrated example includes a processor 912. The processor 912 of the illustrated example is hardware. For example, the processor 912 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired series or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In this example, the processor 912 implements the example frequency range separator 202, the example audio characteristic determiner 204, the example signal normalizer 206, the example point selector 208, and the example fingerprint generator 210.

[0121] The processor 912 of the illustrated example includes a local memory 913 (e.g., cache). The processor 912 of the illustrated example communicates with a main memory including a volatile memory 914 and a non-volatile memory 916 via a bus 918. The volatile memory 914 may be comprised of a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 916 may be implemented by a flash memory and / or any other desired type of memory device. Access to the main memory 914, 916 is controlled by a memory controller.

[0122] The processor platform 900 of the illustrated example also includes an interface circuit 920. The interface circuit 920 may be implemented by any type of interface standard (such as an Ethernet interface, a universal serial bus (USB), interface, near field communication (NFC) interface and / or PCI express interface).

[0123] In the example shown, one or more input devices 922 are connected to the interface circuit 920. The input devices 922 allow a user to input data and / or commands into the processor 912. For example, the input devices 922 may be implemented by an audio sensor, a microphone, a camera (still or video), and / or a voice recognition system.

[0124] One or more output devices 924 are also connected to the interface circuit 920 of the illustrated example. The output device 924 may be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-plane switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit 920 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0125] The interface circuitry 920 of the illustrated example also includes communication devices (such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces) to facilitate exchanging data with external machines (e.g., any kind of computing device) via a network 926. For example, the communication may be via an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a direct-line wireless system, a cellular telephone system, etc.

[0126] The processor platform 900 of the illustrated example also includes one or more mass storage devices 928 for storing software and / or data. Examples of such mass storage devices 928 include floppy disk drives, hard disk drives, optical disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0127] To achieve Figure 6 The machine executable instructions 932 of the method may be stored in the mass storage device 928, the volatile memory 914, the non-volatile memory 916, and / or on a removable non-transitory computer-readable storage medium (such as a CD or DVD).

[0128] From the foregoing, it will be appreciated that example methods and apparatus have been disclosed that allow for the creation of a fingerprint of an audio signal that reduces the amount of noise captured in the fingerprint. Additionally, by sampling audio from regions of the audio signal where the energy is less, a more robust audio fingerprint can be created compared to previously used audio fingerprinting methods.

[0129] Although certain example methods, apparatus, and articles of manufacture are disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus, and articles of manufacture fairly falling within the scope of the claims of this patent.

Claims

1. A method for audio fingerprint recognition, the method comprising the following steps: transforming an audio signal into a frequency domain, wherein the transformed audio signal comprises a plurality of time-frequency bins, wherein each time-frequency bin of the plurality of time-frequency bins corresponds to an intersection of a frequency bin and a time bin and comprises a portion of the audio signal; determining audio characteristics of an audio region comprising a set of a plurality of time-frequency bins, wherein the set of a plurality of time-frequency bins comprises a first time-frequency bin and a time-frequency bin within a predefined distance of the first time-frequency bin; normalizing the audio region to generate a normalized energy value, wherein normalizing the audio region comprises normalizing each portion of the audio signal for each time-frequency bin in the set of a plurality of time-frequency bins based on the determined audio characteristic associated with the audio region; selecting one of the normalized energy values ​​based on the category of the audio signal; and A fingerprint of the audio signal is generated using the selected one of the normalized energy values.

2. The method according to claim 1, wherein: The determined audio characteristics of the audio region include at least one of the following: (i) mean energy value, (ii) mode energy value, (iii) average power value, (iv) mode power value, or (v) mean amplitude of the set of multiple time-frequency bins.

3. The method according to claim 1, further comprising: determining a plurality of audio characteristics, each audio characteristic corresponding to an audio region of a plurality of audio regions, wherein each audio region of the plurality of audio regions comprises a set of time-frequency bins, the set of time-frequency bins comprising a particular time-frequency bin and a time-frequency bin within a predefined distance of the particular time-frequency bin; and Each audio region of the plurality of audio regions is normalized based on the associated determined audio characteristic of the audio region.

4. The method according to claim 3, wherein: A first audio region of the plurality of audio regions comprises a first set of time-frequency bins, wherein a second audio region of the plurality of audio regions comprises a second set of time-frequency bins, wherein at least a portion of the first set of time-frequency bins overlaps with at least a portion of the second set of time-frequency bins.

5. The method according to claim 1, wherein: The category of the audio signal includes at least one of music, human voice, sound effect, or advertisement.

6. The method according to claim 1, wherein: Selecting one of the normalized energy values ​​is based on a normalized energy extrema of the audio region.

7. The method according to claim 1, wherein: Selecting one of the normalized energy values ​​comprises: Determining a category of the audio signal; weighting each time-frequency bin in the set of the plurality of time-frequency bins based on the determined class of the audio signal; and The selection of one of the normalized energy values ​​is weighted by the category of the audio signal.

8. A tangible, non-transitory computer-readable medium comprising instructions that, when executed, cause at least one processor to perform a set of operations comprising: transforming an audio signal into a frequency domain, wherein the transformed audio signal comprises a plurality of time-frequency bins, wherein each time-frequency bin of the plurality of time-frequency bins corresponds to an intersection of a frequency bin and a time bin and contains a portion of the audio signal; determining audio characteristics of an audio region comprising a set of a plurality of time-frequency bins, wherein the set of a plurality of time-frequency bins comprises a first time-frequency bin and a time-frequency bin within a predefined distance of the first time-frequency bin; normalizing the audio region to generate a normalized energy value, wherein normalizing the audio region comprises normalizing each portion of the audio signal for each time-frequency bin in the set of a plurality of time-frequency bins based on the determined audio characteristic associated with the audio region; selecting one of the normalized energy values ​​based on the category of the audio signal; and A fingerprint of the audio signal is generated using the selected one of the normalized energy values.

9. The tangible, non-transitory computer-readable medium of claim 8, wherein: The determined audio characteristics of the audio region include at least one of: (i) a mean energy value, (ii) a mode energy value, (iii) an average power value, (iv) a mode power value, or (v) a mean amplitude of the set of multiple time-frequency bins.

10. The tangible, non-transitory computer-readable medium of claim 8, further comprising: determining a plurality of audio characteristics, each audio characteristic corresponding to an audio region of a plurality of audio regions, wherein each audio region of the plurality of audio regions comprises a set of time-frequency bins, the set of time-frequency bins comprising a particular time-frequency bin and a time-frequency bin within a predefined distance of the particular time-frequency bin; and Each audio region of the plurality of audio regions is normalized based on an associated determined audio characteristic of the audio region.

11. The tangible, non-transitory computer-readable medium of claim 10, wherein: A first audio region of the plurality of audio regions comprises a first set of time-frequency bins, wherein a second audio region of the plurality of audio regions comprises a second set of time-frequency bins, wherein at least a portion of the first set of time-frequency bins overlaps with at least a portion of the second set of time-frequency bins.

12. The tangible, non-transitory computer-readable medium of claim 8, wherein: The category of the audio signal includes at least one of music, human speech, sound effects, or advertisements.

13. The tangible, non-transitory computer-readable medium of claim 8, wherein: Selecting one of the normalized energy values ​​is based on a normalized energy extrema of the audio region.

14. The tangible, non-transitory computer-readable medium of claim 8, wherein: Selecting one of the normalized energy values ​​comprises: Determining a category of the audio signal; weighting each time-frequency bin in the set of the plurality of time-frequency bins based on the determined class of the audio signal; and The selection of one of the normalized energy values ​​is weighted by the class of the audio signal.

15. A computing device, the computing device comprising: at least one processor; as well as A tangible, non-transitory computer-readable medium comprising instructions that, when executed, cause the at least one processor to perform a set of operations comprising: transforming an audio signal into a frequency domain, wherein the transformed audio signal comprises a plurality of time-frequency bins, wherein each time-frequency bin of the plurality of time-frequency bins corresponds to an intersection of a frequency bin and a time bin and contains a portion of the audio signal; determining audio characteristics of an audio region comprising a set of a plurality of time-frequency bins, wherein the set of a plurality of time-frequency bins comprises a first time-frequency bin and a time-frequency bin within a predefined distance of the first time-frequency bin; normalizing the audio region to generate a normalized energy value, wherein normalizing the audio region comprises normalizing each portion of the audio signal for each time-frequency bin in the set of a plurality of time-frequency bins based on the determined audio characteristic associated with the audio region; selecting one of the normalized energy values ​​based on the category of the audio signal; and A fingerprint of the audio signal is generated using the selected one of the normalized energy values.

16. The computing device of claim 15, wherein: The determined audio characteristics of the audio region include at least one of: (i) mean energy value, (ii) mode energy value, (iii) average power value, (iv) mode power value, or (v) mean amplitude of the set of multiple time-frequency bins.

17. The computing device of claim 15, wherein: The set of operations also includes: determining a plurality of audio characteristics, each audio characteristic corresponding to an audio region of a plurality of audio regions, wherein each audio region of the plurality of audio regions comprises a set of time-frequency bins, the set of time-frequency bins comprising a particular time-frequency bin and a time-frequency bin within a predefined distance of the particular time-frequency bin; and Each audio region of the plurality of audio regions is normalized based on the associated determined audio characteristic of the audio region.

18. The computing device of claim 17, wherein: A first audio region of the plurality of audio regions comprises a first set of time-frequency bins, wherein a second audio region of the plurality of audio regions comprises a second set of time-frequency bins, wherein at least a portion of the first set of time-frequency bins overlaps with at least a portion of the second set of time-frequency bins.

19. The computing device of claim 15, wherein: The category of the audio signal includes at least one of music, human voice, sound effect, or advertisement.

20. The computing device of claim 15, wherein: Selecting one of the normalized energy values ​​comprises: Determining a category of the audio signal; weighting each time-frequency bin in the set of the plurality of time-frequency bins based on the determined class of the audio signal; and The selection of one of the normalized energy values ​​is weighted by the class of the audio signal.

Images