Audio processing for detecting crowd noise onset in sporting event television programming

Abstract

To extract metadata of audiovisual content highlights depicting sporting events or other events from the audiovisual content.SOLUTION: Highlights can be segments of content of particular interest, such as broadcasts of sporting events. Audio data of audiovisual content is stored, and by analyzing an audio signal in joint time and frequency domains, portions of the audio data that indicate crowd excitement (noise) are automatically identified. To detect, verify, and render occurrence of crowd noise, multiple indicators are derived and subsequently processed. Metadata is automatically generated that includes the time of occurrence, level of noise (excitement), and duration of cheer. The metadata can be stored including at least a time index indicating the time of occurrence of each of the portions in the audiovisual content. The duration of intense crowd noise can be used to identify highlights and / or to indicate the crowd excitement while viewing the highlights.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority from U.S. Provisional Application No. 62 / 680,955 (Attorney Docket No. THU007-PROV), filed June 5, 2018, for "Audio Processing for Detecting Occurrences of Crowd Noise in Sporting Event Television Programming," which is incorporated herein by reference in its entirety.

[0002] This application claims priority from U.S. Provisional Application No. 62 / 712,041, entitled "Audio Processing for Extraction of Variable Length Disjoint Segments from Television Signal," filed July 30, 2018 (Attorney Docket No. THU006-PROV), which is incorporated herein by reference in its entirety.

[0003] This application claims priority from U.S. Provisional Application No. 62 / 746,454, filed October 16, 2018, entitled "Audio Processing for Detecting Occurrences of Loud Sound Characterized by Short-Time Energy Bursts" (Attorney Docket No. THU016-PROV), which is incorporated herein by reference in its entirety.

[0004] This application claims priority from U.S. Utility Application No. 16,421,391 (Attorney Docket No. THU007), filed May 23, 2019, for "Audio Processing for Detecting Occurrences of Crowd Noise in Sporting Event Television Programming," which is incorporated herein by reference in its entirety.

[0005] This application is related to U.S. Utility Model Application No. 13 / 601,915, "Generating Excitement Levels for Live Performances," filed August 31, 2012, and issued June 16, 2015, as U.S. Patent No. 9,060,210, which is incorporated herein by reference in its entirety.

[0006] This application is related to U.S. Utility Application No. 13 / 601,927, "Generating Alerts for Live Performances," filed August 31, 2012, and issued September 23, 2014, as U.S. Patent No. 8,842,007, which is incorporated herein by reference in its entirety.

[0007] This application is related to U.S. Utility Application No. 13 / 601,933, "Generating Teasers for Live Performances," filed August 31, 2012, and issued November 26, 2013, as U.S. Patent No. 8,595,763, which is incorporated herein by reference in its entirety.

[0008] This application is related to U.S. Utility Application No. 14 / 510,481 (Attorney Docket No. THU001), filed October 9, 2014, for "Generating a Customized Highlight Sequence Depicting an Event," which is hereby incorporated by reference in its entirety.

[0009] This application is related to U.S. Utility Application No. 14 / 710,438 (Attorney Docket No. THU002), filed May 12, 2015, for "Generating a Customized Highlight Sequence Depicting Multiple Events," which is hereby incorporated by reference in its entirety.

[0010] This application is related to U.S. Utility Application No. 14 / 877,691 (Attorney Docket No. THU004), filed October 7, 2015, for "Customized Generation of Highlight Show with Narrative Component," which is hereby incorporated by reference in its entirety.

[0011] This application is related to U.S. Utility Application No. 15 / 264,928 (Attorney Docket No. THU005), filed September 14, 2016, entitled "User Interface for Interaction with Customized Highlight Shows," which is hereby incorporated by reference in its entirety.

[0012] This application is related to U.S. Utility Application No. 16 / 411,704, "Video Processing for Enabling Sports Highlights Generation," filed May 14, 2019 (Attorney Docket No. THU009), which is incorporated herein by reference in its entirety.

[0013] This application is related to U.S. Utility Application No. 16 / 411,710, filed May 14, 2019, for "Machine Learning for Recognizing and Interpreting Embedded Information Card Content" (Attorney Docket No. THU010), which is incorporated herein by reference in its entirety.

[0014] This application is related to U.S. Utility Application No. 16 / 411,713 (Attorney Docket No. THU012), filed May 14, 2019, entitled "Video Processing for Embedded Information Card Localization and Content Extraction," which is hereby incorporated by reference in its entirety.

[0015] This document relates to techniques for identifying multimedia content and associated information on television devices and video servers that deliver the multimedia content, enabling embedded software applications to utilize the multimedia content and provide content and services that synchronize with the multimedia content. Various embodiments relate to methods and systems for providing automated audio analysis that identify and extract information from television programming content depicting sporting events to create metadata associated with video highlights for in-game and post-game viewing. [Background technology]

[0016] Enhanced television applications such as interactive advertising, and enhanced program guides with pre-game, during-game, and post-game interactive applications have long been envisioned. Existing cable systems, originally designed for broadcast television, are being called upon to support new applications and services, including interactive television services and enhanced (interactive) program guides.

[0017] Several frameworks have been standardized to enable enhanced television applications. Examples include the OpenCable™ Enhanced TV Applications Messaging specification, which refers to interactive digital cable services delivered over cable video networks and includes features such as interactive program guides, interactive advertising, and games, as well as the Tru2way specification. Additionally, cable operator "OCAP" programming offers interactive services such as e-commerce shopping, online banking, electronic program guides, and digital video recording. These efforts have enabled the first generation of video synchronization applications that are synchronized with the video content delivered by the programmizer / broadcaster, providing additional data and interactivity to television programming.

[0018] Recent developments in video and audio content analysis technologies and corresponding mobile devices have opened up a range of new possibilities for the development of sophisticated applications that operate in sync with live TV programming events. These new technologies, along with advances in audio signal processing and computer vision, as well as the improved computing power of modern processors, enable the real-time generation of sophisticated program content highlights accompanied by metadata currently lacking in television and other media environments. Summary of the Invention

[0019] A system and method are presented that enables automated real-time processing of audio data, such as audio streams extracted from television program content of sporting events, to detect, select, and track prominent crowd noises (e.g., audience cheers).

[0020] In at least one embodiment, a spectrogram of the audio data is constructed, and a significant collection of spectral magnitude peaks is identified at each position of a sliding two-dimensional time-frequency area window. A spectral indicator is generated for each position of the analysis window, and a vector of spectral indicators with associated time positions is formed. In a subsequent processing step, runs of selected indicator-position pairs closely spaced in time are identified as potential events of interest. For each run, the internal indicator values ​​are sorted to obtain the maximum magnitude indicator value with the associated time position. Furthermore, the time position (start / center) and duration (count of indicator-position pairs) for each run are extracted. A preliminary event vector is formed, including a triplet of parameters (M, P, D) representing the maximum indicator value, start / center time position, and run duration for each event. This preliminary event vector is then processed to generate a final crowd noise event vector corresponding to the desired event interval, event loudness, and event duration.

[0021] In at least one embodiment, once the crowd noise event information is extracted, it is automatically added to sporting event metadata associated with highlights of the sporting event video and can then be used in connection with the automatic generation of the highlights.

[0022] In at least one embodiment, a method for extracting metadata from an audiovisual stream of an event may include storing audio data extracted from the audiovisual stream in a data store, automatically identifying, using a processor, one or more portions of the audio data that indicate crowd excitement at the event, and storing metadata in the data store that includes at least a time index indicating the time in the audiovisual stream at which each of the portions occurs. Alternatively, the audio data may be extracted from the audio stream or from previously stored audiovisual or audio content.

[0023] The audiovisual stream may be a broadcast of an event. The event may be a sporting event or some other type of event. The metadata may relate to highlights that are deemed to be of particular interest to one or more users.

[0024] The method may further include presenting metadata using an output device during viewing of the highlight by one of the one or more users to indicate a crowd excitement level associated with the highlight.

[0025] The method may further include using a time index to identify a beginning and / or an end of the highlight, which may be adjusted based on the offset, as described below.

[0026] The method may further include using an output device to present highlights to one of the one or more users during the automatic identification of the one or more portions.

[0027] The method may further include pre-processing the audio data by resampling the audio data to a desired sampling rate prior to the automatic identification of the one or more portions.

[0028] The method may further include pre-processing the audio data by filtering the audio data to reduce or remove noise prior to the automatic identification of the one or more portions.

[0029] The method may further include pre-processing the audio data to generate a spectrogram (a two-dimensional time-frequency representation) for at least a portion of the audio data prior to automatic identification of the one or more portions.

[0030] Automatically identifying the one or more portions may include identifying a spectral magnitude peak at each position of a sliding two-dimensional time-frequency analysis window of the spectrogram.

[0031] Automatically identifying the one or more portions may further include generating a spectral indicator for each position in the analysis window and using the spectral indicators to form a vector of spectral indicators with associated time portions.

[0032] The method may further include identifying runs of selected pairs of spectral indicators and analysis window positions, capturing the identified runs in a set of R vectors, and obtaining one or more maximum magnitude indicators using the set of R vectors.

[0033] The method may further include extracting a time index from each of the R vectors.

[0034] The method may further include generating a preliminary event vector by replacing each R vector with a parameter triplet representing a maximum magnitude indicator, a time index, and a run length of one of the runs.

[0035] The method may further include processing the preliminary event vector to generate crowd noise event information including a time index.

[0036] Further details and variations are described herein. [Brief explanation of the drawings]

[0037] The accompanying drawings, together with the description, illustrate several embodiments. Those skilled in the art will recognize that the specific embodiments shown in the drawings are merely exemplary and are not intended to be limiting in scope.

[0038] [Figure 1A] FIG. 2 is a block diagram illustrating the hardware architecture of a client / server embodiment in which event content is provided via networked content providers. [Figure 1B] FIG. 2 is a block diagram illustrating the hardware architecture of another client / server embodiment in which event content is stored on a client-based storage device. [Figure 1C] FIG. 2 is a block diagram illustrating a hardware architecture according to a standalone embodiment. [Figure 1D] FIG. 1 is a block diagram illustrating an overview of a system architecture, according to one embodiment. [Figure 2] 1A, 1B, and 1C are schematic block diagrams illustrating example data structures that may be incorporated into the audio data, user data, and highlight data of FIGS. 1A, 1B, and 1C, according to one embodiment. [Figure 3A]1 illustrates an example audio waveform graph showing the occurrence of crowd noise events (e.g., crowd cheering) in an audio stream extracted from television program content of a sporting event in the time domain, according to one embodiment. [Figure 3B] 3B illustrates an example spectrogram corresponding to the audio waveform graph of FIG. 3A in the time-frequency domain, according to one embodiment. [Figure 4] 1 is a flowchart illustrating a method for performing on-the-fly processing of audio data to extract metadata, according to one embodiment. [Figure 5] 1 is a flowchart illustrating a method for analyzing audio data in the time-frequency domain to detect clustering of spectral magnitude peaks associated with prolonged crowd cheers, according to one embodiment. [Figure 6] 1 is a flowchart illustrating a method for generating a crowd noise event vector according to one embodiment. [Figure 7] 10 is a flowchart illustrating a method for internal processing of each R vector, according to one embodiment. [Figure 8] 10 is a flowchart illustrating a method for further selecting a desired crowd noise event, according to one embodiment. [Figure 9] 10 is a flowchart illustrating a method for further selecting a desired crowd noise event, according to one embodiment. [Figure 10] 10 is a flowchart illustrating a method for further selecting a desired crowd noise event, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0039] definition The following definitions are offered for illustrative purposes only and are not intended to be limiting in scope. Event: In the description herein, the term “event” refers to a game, session, match, series, performance, program, concert, etc., or portions thereof (such as an act, period, quarter, half, inning, scene, or chapter). An event can be a sporting event, an entertainment event, a specific performance of a single individual or a subset of individuals within a larger group of event participants, etc. Examples of non-sporting events include television shows, breaking news, sociopolitical events, natural disasters, movies, plays, radio shows, podcasts, audiobooks, online content, musical performances, etc. An event can be of any length. For purposes of explanation, the technology is often described herein in terms of sporting events. However, those skilled in the art will recognize that the technology can be used in other contexts, including highlight shows for audiovisual, audio, visual, graphics-based, interactive, non-interactive, or text-based content. Thus, the use of the term “sporting event” and other sport-specific terminology in the description is intended to illustrate one possible embodiment but is not intended to limit the scope of the described technology to that one embodiment. Rather, such terms should be considered to extend to suitable contexts other than sports, as appropriate to this technology. For ease of explanation, the term "event" is also used to refer to an account or representation of an event, such as an audiovisual recording of the event, or other content item that includes an accounting, description, or depiction of the event. Highlights: An excerpt or portion of an event, or an excerpt or portion of content related to an event that is deemed to be of particular interest to one or more users. Highlights can be of any length. Generally, the technology described herein provides a mechanism for identifying and presenting a customized set of highlights (which may be selected based on particular characteristics and / or user preferences) for any suitable event. "Highlights" may also be used to refer to an account or representation of a highlight, such as an audiovisual recording of a highlight, or other content item that includes an accounting, description, or depiction of a highlight. Highlights need not be limited to depictions of the event itself, but can include other content related to the event. For example, in the case of a sporting event, highlights can include in-game audio / video, as well as other content such as pre-game, in-game, and post-game interviews, analysis, commentary, etc. Such content can be recorded from linear television (e.g., as part of an audiovisual stream depicting the event itself) or obtained from any number of other sources. Various types of highlights can be provided, including, for example, occurrences (plays), strings, possessions, and sequences, all of which are defined below. The highlight need not be of fixed duration, but may incorporate a start and / or end offset, as described below. Clip: A portion of an audio, visual, or audiovisual representation of an event. A clip may correspond to or represent a highlight. In many contexts herein, the term "segment" is used interchangeably with "clip." A clip may be a portion of an audio, video, or audiovisual stream, or may be a portion of stored audio, video, or audiovisual content. ● Content Delineator: One or more video frames that indicate the start or end of a highlight. Occurrence: Something that occurs during an event. Examples include a goal, a play, a down, a hit, a save, a shot on goal, a basket, a steal, a snap or snap attempt, a near miss, a fight, the start or end of a game, a quarter, half, period, or inning, a pitch, a penalty, an injury, a dramatic incident at an entertainment event, a song, a solo, etc. Occurrences may also be anomalous, such as a power outage or an incident of unruly fans. Detection of such occurrences can be used as the basis for determining whether to designate a particular portion of an audiovisual stream as a highlight. Occurrences are also referred to herein as "plays" for ease of naming, although such usage should not be construed as limiting in scope. Occurrences may be of any length, and representations of occurrences may vary in length. For example, as noted above, an expanded representation of an occurrence may include scenes depicting the time periods immediately preceding and following the occurrence, while a simple representation may include only the occurrence itself. Optional intermediate representations may also be provided. In at least one embodiment, the selection of a duration for representing an occurrence may depend on user preference, available time, a determined excitement level for the occurrence, the importance of the occurrence, and / or any other factors. Offset: An amount by which the length of a highlight is adjusted. In at least one embodiment, a start offset and / or an end offset may be provided to adjust the start time and / or end time of the highlight, respectively. For example, if a highlight depicts a goal, the highlight may be extended by a few seconds (via the end offset) to include the celebration and / or fan reaction following the goal. The offset may be configured to vary automatically or manually based, for example, on the amount of time available for the highlight, the importance and / or excitement level of the highlight, and / or other suitable factors. String: A series of occurrences that are linked or related to each other in some way. An occurrence may occur within a possession (defined below) or across multiple possessions. An occurrence may occur within a sequence (defined below) or across multiple sequences. Occurrences may be linked or related because they have some thematic or narrative connection to each other, or because one leads to another, or for other reasons. An example of a string is a goal or a set of passes that lead to a basket. This should not be confused with a "text string," which has the meaning commonly assigned to it in the computer programming art. Possession: A time-bound portion of an event. The boundaries of possession start / end time may depend on the type of event. For certain sporting events (e.g., basketball or soccer) where one team may be on offense and the other team on defense, possession can be defined as the period of time when one of the teams has the ball. In sports where possession of the puck or ball is more fluid, such as hockey or soccer, possession can be considered to extend to the period of time when one team has substantial control of the puck or ball, ignoring momentary contact by the other team (such as a blocked shot or save). For baseball, possession is defined as a half-inning. For soccer, possession can include several sequences where the same team has the ball. For other types of sporting events, as well as non-sporting events, the term "possession" may be somewhat of a misnomer, but is still used herein for purposes of explanation. Examples in non-sporting contexts can include chapters, scenes, acts, etc. For example, in the context of a music concert, a possession could correspond to the performance of a single song. A possession can include any number of occurrences. Sequence: A time-delimited portion of an event that includes one continuous action time period. For example, in a sporting event, a sequence may begin when the action begins (such as a face-off or tip-off) and end when the whistle is blown to indicate a halt to the action. In sports such as baseball or soccer, a sequence may equate to a play, which is a form of occurrence. A sequence can include any number of possessions or can be portions of possessions. ● Highlight show: A set of highlights arranged for presentation to a user. A highlight show may be presented linearly (such as an audiovisual stream) or may be presented to allow the user to select which highlights to view and in what order (e.g., by clicking links or thumbnails). The presentation of a highlight show may be non-interactive or interactive, for example, allowing the user to pause, rewind, skip, fast-forward, communicate a thumbs-up or thumbs-down preference, etc. A highlight show may be, for example, a condensed game. A highlight show may include any number of sequential or non-sequential highlights from a single event or multiple events, and may even include highlights from different types of events (e.g., different sports and / or a combination of highlights from sporting and non-sporting events). ● User / Viewer: The terms "user" or "viewer" refer interchangeably to an individual, group, or other entity that is watching, listening to, or otherwise experiencing an event, one or more highlights of the event, or a highlight show. The terms "user" or "viewer" may also refer to an individual, group, or other entity that may watch, listen to, or otherwise experience either an event, one or more highlights of the event, or a highlight show at some future time. While the term "viewer" is sometimes used for descriptive purposes, an event need not have a visual component, and therefore a "viewer" may instead be a listener or other consumer of the content. Excitement Level: A measure of how exciting or interesting an event or highlight is expected to be to a particular user or to users in general. Excitement level can also be determined with respect to a particular occurrence or player. Various techniques for measuring or assessing excitement level are described in the related applications referenced above. As described, excitement level can depend on the occurrence within the event as well as other factors, such as the overall context or importance of the event (e.g., a playoff game, pennant impact, rivalry, etc.). In at least one embodiment, an excitement level can be associated with each occurrence, string, possession, or sequence within an event. For example, the excitement level of a possession can be determined based on the occurrences that occur within that possession. Excitement level may be measured differently for different users (e.g., fans of one team vs. neutral fans) and can vary depending on each user's personal characteristics. • Metadata: Data that is related to and stored in association with other data. Primary data can be media such as sports programming or highlights. ● Video data. A length of video, which may be in digital or analog format. The video data may be stored on a local storage device or may be received in real time from a source such as a TV broadcast antenna, a cable network, or a computer server, in which case it may be called a "video stream." The video data may or may not include an audio component, in which case it may be called "audiovisual data" or an "audiovisual stream." Audio data. A length of audio, which may be in digital or analog form. The audio data may be the audio component of audiovisual data or an audiovisual stream, and may be separated by extracting the audio data from the audiovisual data. The audio data may be stored in local storage or may be received in real time from a source such as a TV broadcast antenna, a cable network, or a computer server, in which case it may also be referred to as an "audio stream." ● Stream. An audio stream, a video stream, or an audiovisual stream. • Time index: An indicator of a time within the audio, video, or audiovisual data at which an event occurs or is otherwise related to a specified segment, such as a highlight. Spectrogram: A visual representation of the frequency spectrum of a signal, such as an audio stream, as it changes over time. Analysis window: A designated subset of video data, audio data, audiovisual data, spectrogram, stream, or an otherwise processed version of the stream or data, in which one step of analysis is focused. Audio data, video data, audiovisual data, or spectrograms can be analyzed in segments, for example, using a moving analysis window and / or a series of analysis windows that cover different segments of the data or spectrogram.

[0040] overview According to various embodiments, methods and systems are provided for automatically creating time-based metadata associated with highlights of television programming, such as sporting events, where such video highlights and associated metadata are generated either synchronously with a television broadcast of the sporting event or generated after the television broadcast of the sporting event while video content of the sporting event is being streamed from a storage device via a video server.

[0041] In at least one embodiment, an automated video highlight and associated metadata generation application can receive a live broadcast audiovisual stream or a digital audiovisual stream received via a computer server. The application can then process audio data, such as an audio stream extracted from the audiovisual stream, using, for example, digital signal processing techniques, to detect crowd noise, such as the cheering of a crowd.

[0042] In alternative embodiments, the techniques described herein can be applied to other types of source content. For example, the audio data need not be extracted from an audiovisual stream, but rather may be a radio broadcast or other audio depiction of a sporting or other event. Alternatively, the techniques described herein can be applied to stored audio data depicting an event, where such data may or may not be extracted from the stored audiovisual data.

[0043] Interactive television applications enable timely and relevant presentation of highlighted television program content to users watching the television program on either a primary television display or a secondary display, such as a tablet, laptop, or smartphone. In at least one embodiment, a set of clips representing highlights of the television broadcast content are generated and / or stored in real time, along with a database including time-based metadata that more fully describes the events presented by the highlight clips. As described in more detail herein, the start and / or end times of such clips can be determined, at least in part, based on an analysis of extracted audio data.

[0044] In various embodiments, the metadata associated with a clip may be any information, such as text information, images, and / or any type of audiovisual data. One type of metadata associated with both in-game and post-game video content highlights presents events detected by real-time processing of audio data extracted from television coverage of a sporting event. In various embodiments, the systems and methods described herein enable automatic metadata generation and video highlight processing, and the start and / or end times of highlights may be detected and determined by analyzing digital audio data, such as an audio stream. For example, event information may be extracted by analyzing such audio data to detect cheering crowd noise following a particular exciting event, audio announcement, music, etc., and such information may be used to determine the start and / or end times of highlights.

[0045] In at least one embodiment, real-time processing is performed on audio data, such as an audio stream extracted from television program content of a sporting event, to detect, select, and track prominent crowd noises (such as audience cheers).

[0046] In at least one embodiment, the system and method receives compressed audio data, reads, decodes, and resamples the compressed audio data to a desired sampling rate. Pre-filtering can be performed for noise reduction, click removal, and selection of frequency bands of interest. Any of several interchangeable digital filtering stages can be used.

[0047] A spectrogram can be constructed for audio data: a significant collection of spectral magnitude peaks can be identified at each position in a sliding two-dimensional time-frequency area window.

[0048] A spectral indicator may be generated for each position of the analysis window, and a vector of spectral indicators with associated time positions may be formed.

[0049] Runs of selected indicator-position pairs with narrow time intervals are identified and a set of vectors

number

[0050] The time position (start / center) and length (duration) of the run (count of indicator-position pairs) can be extracted from each R vector.

[0051] Each R vector can be replaced with a parameter triplet (M, P, D) representing the maximum indicator value, start / median time position, and run length (duration), respectively, to form a preliminary event vector.

[0052] The preliminary event vectors can be processed to generate final crowd noise event vectors according to desired event intervals, event loudness, and event duration.

[0053] The extracted crowd noise event information may be automatically added to sporting event metadata associated with highlights of the sporting event video.

[0054] In another embodiment, a system and method performs real-time processing of an audio stream extracted from a television program of a sporting event to detect, select, and track prominent crowd noise events. The system and method may include capturing television program content, extracting and processing digital audio data, such as a digital audio stream, to detect prominent crowd noise events, generating a time-frequency audio spectrogram, performing a combined time-frequency analysis of the audio data to detect areas of high spectral activity, generating spectral indicators for overlapping spectrogram areas, forming vectors of selected indicator-location pairs, identifying runs of selected indicator-location pairs having close time intervals, forming a set of vectors with the identified runs, forming at least one preliminary event vector with a parameter triplet (M, P, D) derived from each run of selected indicator-location pairs, and modifying the at least one preliminary event vector to generate at least one final crowd noise event vector with a desired event interval, event loudness, and event duration.

[0055] Initial pre-processing of the decoded audio data may be performed for at least one of noise reduction, removal of clicks and other spurious sounds, and selection of frequency bands of interest by selection of interchangeable digital filtering stages.

[0056] A spectrogram can be constructed to analyze audio data in the spectral domain. In at least one embodiment, the size of the analysis window is selected along with the size of the analysis window overlap region. In at least one embodiment, the analysis window is slid along the spectrogram, and at each analysis window position, a normalized average magnitude of the analysis window is calculated. In at least one embodiment, the average magnitude is determined as the spectral indicator at each analysis window position. In at least one embodiment, an initial event vector is populated with the calculated pairs of analysis window indicators and associated positions. In at least one embodiment, the initial event vector indicators are subject to thresholding to retain only indicator-position pairs with indicators above a threshold.

[0057] Each run may include a variable count of unequal-sized indicators. In at least one embodiment, for each run, the indicators may be sorted internally by indicator value to obtain the largest magnitude indicator.

[0058] For each run, the start / median time position and run duration can be extracted.

[0059] The preliminary event vector can be formed using a parameter triplet (M, P, D), which in at least one embodiment represents a maximum indicator value, a start / median time position, and a run duration, respectively.

[0060] The preliminary event vectors can be modified to generate a final crowd noise event vector according to a desired event interval, event loudness, and event duration. In various embodiments, the preliminary event vectors are modified by an acceptable event distance selection, an acceptable event duration selection, and / or an acceptable event loudness selection.

[0061] The crowd noise event information may be further processed and automatically added to the metadata associated with television highlights of the sporting event.

[0062] System Architecture According to various embodiments, the system can be implemented on any electronic device or set of electronic devices equipped to receive, store, and present information. Such electronic devices can be, for example, desktop computers, laptop computers, televisions, smartphones, tablets, music players, audio devices, kiosks, set-top boxes (STBs), gaming systems, wearable devices, consumer electronic devices, etc.

[0063] Although the system is described herein with reference to implementation in a particular type of computing device, those skilled in the art will recognize that the techniques described herein can be implemented in other contexts and in any suitable device capable of receiving and / or processing user input and presenting output to a user. Accordingly, the following description is intended to illustrate various embodiments by way of example, rather than to limit the scope.

[0064] 1A, a block diagram illustrating the hardware architecture of a system 100 for automatically extracting metadata based on audio data of an event according to a client / server embodiment is shown. Event content, such as an audiovisual stream containing audio content, may be provided via a network-connected content provider 124. An example of such a client / server embodiment is a web-based implementation, in which one or more client devices 106 each run a browser or app that provides a user interface for interacting with content from the various servers 102, 114, 116, including the data provider server 122 and / or the content provider server 124, via a communications network 104. Transmission of content and / or data in response to requests from the client devices 106 may be performed using any known protocol and language, such as Hypertext Markup Language (HTML), Java, Objective C, Python, JavaScript, etc.

[0065] The client device 106 may be any electronic device, such as a desktop computer, laptop computer, television, smartphone, tablet, music player, audio device, kiosk, set-top box, gaming system, wearable device, consumer electronic device, etc. In at least one embodiment, the client device 106 has several hardware components well known to those skilled in the art. The input device 151 may be any component that receives input from the user 150, including, for example, a handheld remote control, keyboard, mouse, stylus, touch-sensitive screen (touch screen), touchpad, gesture receptor, trackball, accelerometer, five-way switch, microphone, etc. The input may be provided via any suitable mode, including, for example, one or more of pointing, tapping, typing, dragging, gesture, tilting, shaking, and / or speech. The display screen 152 may be any component that graphically displays information, video, content, etc., including event depictions, highlights, etc. Such output may also include, for example, audiovisual content, data visualization, navigational elements, graphical elements, queries requesting information and / or parameters for selection of content, etc. In at least one embodiment, if only a portion of the desired output is presented at a time, dynamic controls such as a scrolling mechanism may be available via input device 151 to select currently displayed information and / or change how the information is displayed.

[0066] Processor 157 may be a conventional microprocessor for performing operations on data under the direction of software in accordance with well-known techniques. Memory 156 may be random access memory having a structure and architecture known in the art for use by processor 157 in the course of running software to perform the operations described herein. Client device 106 may also include local storage (not shown), which may be a hard drive, flash drive, optical or magnetic storage device, web-based (cloud-based) storage, etc.

[0067] Any suitable type of communications network 104, such as the Internet, a television network, a cable network, a cellular network, etc., can be used as a mechanism for transmitting data between the client device 106 and the various servers 102, 114, 116 and / or content providers 124 and / or data providers 122 according to any suitable protocols and techniques. In addition to the Internet, other examples include cellular telephone networks, EDGE, 3G, 4G, Long Term Evolution (LTE), Session Initiation Protocol (SIP), Short Message Peer-to-Peer Protocol (SMPP), SS7, Wi-Fi, Bluetooth, ZigBee, Hypertext Transfer Protocol (HTTP), Secure Hypertext Transfer Protocol (SHTTP), Transmission Control Protocol / Internet Protocol (TCP / IP), etc., and / or any combination thereof. In at least one embodiment, the client device 106 sends requests for data and / or content over the communications network 104 and receives responses from the servers 102, 114, 116 containing the requested data and / or content.

[0068] 1A operates in connection with a sporting event. However, it should be understood that the teachings herein also apply to non-sporting events, and the techniques described herein are not limited to application to sporting events. For example, the techniques described herein can be utilized to operate in connection with television shows, movies, news events, game shows, political campaigns, business shows, dramas, and / or other episodic content, or for two or more such events.

[0069] In at least one embodiment, system 100 identifies highlights of a broadcast event by analyzing audio content representing the event. This analysis can occur in real time. In at least one embodiment, system 100 includes one or more web servers 102 coupled to one or more client devices 106 via a communications network 104. Communications network 104 can be a public network, a private network, or a combination of public and private networks, such as the Internet. Communications network 104 can be a LAN, a WAN, wired, wireless, and / or a combination of the above. Client devices 106, in at least one embodiment, can connect to communications network 104 via either a wired or wireless connection. In at least one embodiment, client devices can also include a recording device capable of receiving and recording events, such as a DVR, PVR, or other media recording device. Such a recording device can be part of client device 106 or can be external. In other embodiments, such a recording device can be omitted. While FIG. 1A shows one client device 106, system 100 can be implemented using any number of client devices 106 of a single type or multiple types.

[0070] The web server 102 may include one or more physical computing devices and / or software capable of receiving requests from client devices 106, responding to those requests with data, and sending unsolicited alerts and other messages. The web server 102 may employ various strategies for fault tolerance and scalability, such as load balancing, caching, and clustering. In at least one embodiment, the web server 102 may include caching techniques, as known in the art, for storing client requests and information related to events.

[0071] The web server 102 may maintain or otherwise designate one or more application servers 114 to respond to requests received from the client devices 106. In at least one embodiment, the application servers 114 provide access to business logic for use by client application programs in the client devices 106. The application servers 114 may be co-located with, shared with, or co-managed by the web server 102. The application servers 114 may also be remote from the web server 102. In at least one embodiment, the application servers 114 interact with one or more analytics servers 116 and one or more data servers 118 to perform one or more operations of the disclosed techniques.

[0072] One or more storage devices 153 may act as a "data store" by storing data related to the operation of system 100. This data may include, for example, but not be limited to, audio data 154 representing one or more audio signals. Audio data 154 may be derived from audiovisual streams or stored audiovisual content representing, for example, sporting and / or other events.

[0073] Audio data 154 may include any information related to audio embedded in an audiovisual stream, such as an audio stream accompanying a video image, a processed version of the audiovisual stream, and metrics and / or vectors related to audio data 154, such as an event's time index, duration, loudness, and / or other parameters. User data 155 may include any information describing one or more users 150, including, for example, demographics, purchasing behavior, audiovisual stream viewing behavior, interests, preferences, etc. Highlight data 164 may include highlights, highlight identifiers, time indicators, categories, excitement levels, and other data related to the highlights. Audio data 154, user data 155, and highlight data 164 are described in more detail below.

[0074] Notably, many components of system 100 may be or include computing devices. Each of such computing devices may have an architecture similar to that of client device 106, as shown and described above. Accordingly, any of communication network 104, web server 102, application server 114, analytics server 116, data provider 122, content provider 124, data server 118, and storage device 153 may include one or more computing devices, each of which may optionally have input device 151, display screen 152, memory 156, and / or processor 157, as described above in connection with client device 106.

[0075] In an exemplary operation of the system 100, one or more users 150 of the client devices 106 view content from the content provider 124 in the form of an audiovisual stream. The audiovisual stream may depict an event, such as a sporting event. The audiovisual stream may be a digital audiovisual stream that can be easily processed using known computer vision techniques.

[0076] Once the audiovisual stream is displayed, one or more components of system 100, such as client device 106, web server 102, application server 114, and / or analysis server 116, analyze the audiovisual stream to identify highlights within the audiovisual stream and / or extract metadata from the audiovisual stream, for example, from the audio component of the stream. This analysis may occur in response to receiving a request to identify highlights and / or metadata for the audiovisual stream. Alternatively, in another embodiment, the highlights and / or metadata may be identified without a specific request made by user 150. In yet another embodiment, analysis of the audiovisual stream may occur without the audiovisual stream being displayed.

[0077] In at least one embodiment, user 150 can specify certain parameters for the analysis of audio data 154 (e.g., which events / games / teams to include, how much time user 150 has available to view highlights, what metadata is desired, and / or other parameters, etc.) via input device 151 of client device 106. User preferences can also be extracted from storage, such as from user data 155 stored on one or more storage devices 153, to customize the analysis of audio data 154 without necessarily requiring user 150 to specify preferences. In at least one embodiment, user preferences can be determined based on observed behavior and actions of user 150, for example, by observing website visiting patterns, television viewing patterns, music listening patterns, online purchases, previous highlight identification parameters, highlights and / or metadata actually viewed by user 150, etc.

[0078] Additionally or alternatively, user preferences can be retrieved from previously stored preferences explicitly provided by user 150. Such user preferences may indicate which teams, sports, players, and / or types of events are of interest to user 150, and / or they may indicate what type of metadata or other information related to highlights would be of interest to user 150. Such preferences can thus be used to guide analysis of the audiovisual stream to identify highlights and / or extract highlight metadata.

[0079] The analytics server 116, which may include one or more computing devices as described above, can analyze live and / or recorded feeds of play-by-play statistics related to one or more events from data providers 122. Examples of data providers 122 may include, but are not limited to, providers of real-time sports information such as STATS™, Perform (available from Opta Sports, London, UK), and SportRadar, St. Gallen, Switzerland. In at least one embodiment, the analytics server 116 generates a set of different excitement levels for the event. Such excitement levels may then be stored in conjunction with highlights identified or received by the system 100 in accordance with the techniques described herein.

[0080] The application server 114 can analyze the audiovisual stream to identify highlights and / or extract metadata. Additionally or alternatively, such analysis can be performed by the client device 106. The identified highlights and / or extracted metadata can be specific to a user 150. In such cases, it can be advantageous to identify highlights within the client device 106 that are associated with a particular user 150. The client device 106 can receive, maintain, and / or retrieve applicable user preferences for highlight identification and / or metadata extraction, as described above. Additionally or alternatively, highlight generation and / or metadata extraction can be performed globally (i.e., using objective criteria applicable to the user population at large, regardless of the preferences of a particular user 150). In such cases, it can be advantageous to identify highlights and / or extract metadata within the application server 114.

[0081] The content facilitating highlight identification, audio analysis, and / or metadata extraction may come from any suitable source, including from content providers 124, which may include websites such as YouTube, MLB.com, sports data providers, television stations, client- or server-based DVRs, etc. Alternatively, the content can come from a local source, such as a DVR or other recording device associated with (or embedded in) the client device 106. In at least one embodiment, the application server 114 generates a customized highlight show with the highlights and metadata available to the user 150 for download, or as streaming content, or on-demand content, or in some other manner.

[0082] As noted above, it may be advantageous for user-specific highlight identification, audio analysis, and / or metadata extraction to occur at a particular client device 106 associated with a particular user 150. Such an embodiment may avoid the need for video content or other high-bandwidth content to be unnecessarily transmitted over the communications network 104, particularly if the content is already available at the client device 106.

[0083] 1B , an example of a system 160 according to an embodiment is shown in which at least a portion of the audio data 154 and highlight data 164 is stored on a client-based storage device 158, which may be any form of local storage device available to the client device 106. One example is a DVR on which events, such as video content of a complete sporting event, may be recorded. Alternatively, the client-based storage device 158 may be any magnetic, optical, or electronic storage device for data in digital form. Examples include flash memory, a magnetic hard drive, a CD-ROM, a DVD-ROM, or other devices integrated with or communicatively coupled to the client device 106. Based on information provided by the application server 114, the client device 106 can extract metadata from the audio data 154 stored on the client-based storage device 158 and store the metadata as highlight data 164 without having to retrieve other content from the content provider 124 or other remote sources. Such a configuration can conserve bandwidth and make effective use of existing hardware that may already be available to the client device 106.

[0084] Returning to FIG. 1A , in at least one embodiment, application server 114 can identify different highlights and / or extract different metadata for different users 150 depending on the individual user's preferences and / or other parameters. The identified highlights and / or extracted metadata can be presented to user 150 via any suitable output device, such as display screen 152 of client device 106. If desired, multiple highlights can be identified and compiled into a highlight show along with associated metadata. Such highlight shows can be assembled into a "highlight reel" or set of highlights accessed via a menu and / or played for user 150 according to a predetermined sequence. User 150, in at least one embodiment, can control highlight playback and / or delivery of associated metadata via input device 151, for example, to: Select specific highlights and / or metadata to display, ●Pause, rewind, fast forward, ● Skip to the next highlight, ● Return to the beginning of the previous highlight in the highlight show, and / or ●Perform other actions.

[0085] Additional details regarding such functionality are provided in the related US patent applications cited above.

[0086] In at least one embodiment, one or more data servers 118 are provided. The data server 118 can respond to requests for data from any of the servers 102, 114, 116, for example, to obtain or provide audio data 154, user data 155, and / or highlight data 164. In at least one embodiment, such information can be stored on any suitable storage device 153 accessible by the data server 118 and can come from any suitable source, such as from the client device 106 itself, the content provider 124, the data provider 122, etc.

[0087] 1C , an alternative embodiment of system 180 is shown in which system 180 is implemented in a standalone environment. Similar to the embodiment shown in FIG. 1B , at least a portion of audio data 154, user data 155, and highlight data 164 may be stored on a client-based storage device 158, such as a DVR. Alternatively, client-based storage device 158 may be flash memory or a hard drive, or other device integrated with or communicatively coupled to client device 106.

[0088] User data 155 may include preferences and interests of user 150. Based on such user data 155, system 180 may extract metadata within audio data 154 and present it to user 150 in the manner described herein. Additionally or alternatively, metadata may be extracted based on objective criteria that are not based on information specific to user 150.

[0089] 1D , an overview of a system 190 having an architecture according to an alternative embodiment is shown. In FIG. 1D , the system 190 includes a broadcast service, such as a content provider 124, a content receiver in the form of a client device 106, such as a television set with an STB, a video server, such as an analysis server 116, that can ingest and stream television program content, and / or other client devices 106, such as mobile devices and laptops, that can receive and process television program content, all connected via a network, such as the communications network 104. A client-based storage device 158, such as a DVR, can be connected to any of the client devices 106 and / or other components and can store audiovisual streams, highlights, highlight identifiers, and / or metadata to facilitate identification and presentation of highlights and / or extracted metadata via any of the client devices 106.

[0090] 1A, 1B, 1C, and 1D are merely exemplary. Those skilled in the art will recognize that other architectures can be used to implement the teachings described herein. Many of the components shown therein are optional and can be omitted, combined with, and / or replaced by other components.

[0091] In at least one embodiment, the system can be implemented as software written in any suitable computer programming language, whether in a stand-alone or client / server architecture, or it can be implemented and / or embedded in hardware.

[0092] Data Structure FIG. 2 is a schematic block diagram illustrating example data structures that may be incorporated into audio data 154, user data 155, and highlight data 164, according to one embodiment.

[0093] As shown, the audio data 154 may include a recording for each of multiple audio streams 200. For purposes of explanation, audio streams 200 are shown, but the techniques described herein may apply to any type of audio data 154 or content, whether streamed or stored. In addition to the audio streams 200, the recording of the audio data 154 may include other data generated according to or useful in analyzing the audio streams 200. For example, the audio data 154 may include, for each audio stream 200, a spectrogram 202, one or more analysis windows 204, a vector 206, and a time index 208.

[0094] Each audio stream 200 may exist in the time domain. Each spectrogram 202 may be calculated for the corresponding audio stream 200 in the time-frequency domain. The spectrograms 202 may be analyzed to more easily find audio events of desired frequencies, such as crowd noise.

[0095] The analysis window 204 may be a designation of a predetermined time and / or frequency interval of the spectrogram 202. Computationally, a single moving (i.e., "sliding") analysis window 204 may be used to analyze the spectrogram 202, or a series of displaced (optionally overlapping) analysis windows 204 may be used.

[0096] The vector 206 may be a data set containing intermediate and / or final results from the analysis of the audio stream 200 and / or the corresponding spectrogram 202 .

[0097] The time index 208 may indicate a time within the audio stream 200 (and / or the audiovisual stream from which the audio stream 200 is extracted) at which a significant event occurs. For example, the time index 208 may be a time within a broadcast at which crowd noise builds or dies down. Thus, in the context of a sporting event, the time index 208 may indicate the start or end of a particularly interesting portion of the audiovisual stream, such as an important or memorable play.

[0098] As further shown, user data 155 may include records related to users 150, each of which may include demographic data 212, preferences 214, browsing history 216, and purchasing history 218 for a particular user 150.

[0099] Demographic data 212 may include any type of demographic data, including, but not limited to, age, gender, location, nationality, religious affiliation, education level, and the like.

[0100] Preferences 214 may include selections made by user 150 regarding his or her preferences. Preferences 214 may relate directly to collecting and / or viewing highlights and metadata, or may be more general in nature. In either case, preferences 214 may be used to facilitate the identification and / or presentation of highlights and metadata to user 150.

[0101] The viewing history 216 may list television programs, audiovisual streams, highlights, web pages, search queries, sporting events, and / or other content searched for and / or viewed by the user 150 .

[0102] The purchase history 218 may list products or services purchased or requested by the user 150 .

[0103] As further shown, the highlight data 164 may include records for j highlights 220 , and each record may include an audiovisual stream 222 and / or metadata 224 for a particular highlight 220 .

[0104] The audiovisual stream 222 may include video depicting the highlight 220, which may be obtained from one or more audiovisual streams of one or more events (e.g., by cropping the audiovisual stream to include only the audiovisual stream 222 associated with the highlight 220). Within the metadata 224, the identifier 223 may include a time index (such as the time index 208 of the audio data 154) and / or other indicia indicating where the highlight 220 resides within the audiovisual stream of the event from which it was obtained.

[0105] In some embodiments, each recording of highlight 220 may include only one of audiovisual stream 222 and identifier 223. Highlight playback may be performed by playing audiovisual stream 222 to user 150, or by using identifier 223 to play only the highlighted portion of the audiovisual stream of the event from which highlight 220 was obtained. Storage of identifier 223 is optional. In some embodiments, identifier 223 may be used only to extract audiovisual stream 222 for highlight 220, which may then be stored in place of identifier 223. In either case, time index 208 of highlight 220 may be extracted from audio data 154 and added to highlight 220 or may be stored, at least temporarily, as metadata 224 added to audio data 154 and the audiovisual stream from which highlight 220 was obtained.

[0106] In addition to, or as an alternative to, the identifier 223, the metadata 224 may include information about the highlight 220, such as the date of the event, the season, and groups or individuals involved in the event or audiovisual stream from which the highlight 220 is obtained, e.g., teams, players, coaches, anchors, broadcasters, and fans. Among other information, the metadata 224 for each highlight 220 may include a phase 226, a clock 227, a score 228, a frame number 229, an excitement level 230, and / or a crowd excitement level 232.

[0107] Phase 226 may be a phase of the event associated with highlight 220. More specifically, phase 226 may be a stage of a sporting event within which the beginning, middle, and / or end of highlight 220 resides. For example, phase 226 may be the "third quarter," the "second inning," the "bottom inning," etc.

[0108] Clock 227 may be a game clock associated with highlight 220. More specifically, clock 227 may be the state of the game clock at the beginning, middle, and / or end of highlight 220. For example, clock 227 may read "15:47" for highlight 220, which begins, ends, or spans a period of the sporting event that is displayed on the game clock for 15 minutes and 47 seconds.

[0109] The score 228 may be a game score associated with the highlight 220. More specifically, the score 228 may be a score at the beginning, end, and / or middle of the highlight 220. For example, the score 228 may be "45-38," "7-0," "30-love," etc.

[0110] The frame number 229 may be the number of a video frame within the audiovisual stream from which the highlight 220 is obtained, or within the audiovisual stream 222 associated with the highlight 220 that relates to the beginning, middle, and / or end of the highlight 220.

[0111] The excitement level 230 may be a measure of how exciting or interesting an event or highlight is expected to be to a particular user 150 or to users in general. In at least one embodiment, the excitement level 230 may be calculated as described in the above-referenced related application. Additionally or alternatively, the excitement level 230 may be determined, at least in part, by analysis of the audio data 154, which may be components extracted from the audiovisual stream 222 and / or the audio stream 200. For example, audio data 154 containing higher levels of crowd noise, announcements, and / or up-tempo music may indicate a high excitement level 230 for the associated highlight 220. The excitement level 230 need not be static for the highlight 220, but instead may change over the course of the highlight 220. Thus, the system 100 may be able to further refine the highlight 220 to only present to the user portions that exceed a threshold excitement level 230.

[0112] The crowd excitement level 232 may be a measure of how excited a crowd attending an event appears. In at least one embodiment, the crowd excitement level 232 may be determined based on an analysis of the audio data 154. In other embodiments, visual analysis may be used to measure crowd excitement or to supplement the results of the audio data analysis.

[0113] For example, if intense crowd noise is detected by analysis of the audio stream 200 of the highlight 220, the crowd excitement level 232 for the highlight 220 may be considered relatively high. Like the excitement level 230, the crowd excitement level 232 may change over the course of the highlight 220. Thus, the crowd excitement level 232 may include, for example, multiple indicators corresponding to specific times within the highlight 220.

[0114] The data structures depicted in Figure 2 are merely exemplary. Those skilled in the art will recognize that in performing highlight identification and / or metadata extraction, some of the data in Figure 2 may be omitted or replaced with other data. Additionally or alternatively, data not specifically shown in Figure 2 or described in this application may be used in performing highlight identification and / or metadata extraction.

[0115] Audio Data 154 In at least one embodiment, the system performs several stages of analysis of audio data 154, such as an audio stream, in the time-frequency domain to detect crowd noises such as crowd cheers, chants, and fan support during a depiction of a sporting event or another event. The depiction may be a television broadcast, an audiovisual stream, an audio stream, a stored file, or the like.

[0116] First, the compressed audio data 154 is read, decoded, and resampled to the desired sampling rate. The resulting PCM stream is then pre-filtered for noise reduction, click removal, and / or desired frequency band selection using any of several interchangeable digital filtering stages. A spectrogram is then constructed for the audio data 154. A significant collection of spectral magnitude peaks is identified at each position in a sliding two-dimensional time-frequency domain window. A spectral indicator is generated for each position in the analysis window, forming a vector of spectral indicators with associated time positions.

[0117] Next, runs of selected indicator-location pairs with narrow time intervals are identified and a set of vectors

number

[0118] 3A shows an example of an audio waveform graph 300 in an audio stream 310 extracted from sporting event television program content in the time domain, according to one embodiment. The highlighted area 320 indicates an exemplary noise event, such as a crowd cheering. The amplitude of the captured audio is relatively high in the highlighted area 320, which may represent a relatively large portion of the audio stream 310.

[0119] 3B shows an example spectrogram 350 corresponding to the audio waveform graph 300 of FIG. 3A in the time-frequency domain, according to one embodiment. In at least one embodiment, detection and marking of the occurrence of events of interest is performed in the time-frequency domain, and the timing boundaries of the events are presented in real time to a video highlight and metadata generation application. This may enable the generation of corresponding metadata 224, such as identifiers 223 identifying the start and / or end of a highlight 220, crowd excitement levels occurring during the highlight 220, etc.

[0120] 4 is a flow chart illustrating a method 400 performed by an application (e.g., running on one of the client device 106 and / or analysis server 116) that receives the audiovisual stream 222 and performs on-the-fly processing of the audio data 154 to, for example, extract metadata 224 corresponding to the highlights 220, according to one embodiment. According to the method 400, the audio data 154, such as the audio stream 310, may be processed to detect crowd noise audio events, music events, announcement events, and / or other audible events related to television program content highlight generation.

[0121] In at least one embodiment, method 400 (and / or other methods described herein) is performed on audio data 154 extracted from an audiovisual stream or other audiovisual content. Alternatively, the techniques described herein can be applied to other types of source content. For example, audio data 154 need not be extracted from an audiovisual stream, but rather may be a radio broadcast or other audio depiction of a sporting event or other event.

[0122] In at least one embodiment, method 400 (and / or other methods described herein) may be performed by a system such as system 100 of FIG. 1A. However, alternative systems, including but not limited to system 160 of FIG. 1B, system 180 of FIG. 1C, and system 190 of FIG. 1D, may be used in place of system 100 of FIG. 1A. Furthermore, the following description assumes that a crowd noise event is identified. However, it will be understood that different types of audible events may be identified and used to extract metadata according to methods similar to those described herein.

[0123] The method 400 of FIG. 4 may begin at step 410, where audio data 154, such as audio stream 200, is read. If the audio data 154 is in a compressed format, it may optionally be decoded. In step 420, the audio data 154 may be resampled to a desired sampling rate. In step 430, the audio data 154 may be filtered using any of several interchangeable digital filtering stages. Next, in step 440, a spectrogram 202 may optionally be generated for the filtered audio data 154, for example, by computing a short-time Fourier transform (STFT) on one-second chunks of the filtered audio data 154. The time-frequency coefficients of the spectrogram 202 may be stored in a two-dimensional array for further processing.

[0124] Notably, in some embodiments, step 440 may be omitted. Rather than performing an analysis of spectrogram 202, further analysis may be performed directly on audio data 154. Figures 5 through 10 below assume that step 440 has been performed and that the remaining analysis steps are performed on spectrogram 202 corresponding to audio data 154 (e.g., after decoding, resampling, and / or filtering audio data 154 as described above).

[0125] FIG. 5 is a flow chart illustrating a method 500 for analyzing audio data 154, such as an audio stream 200, in the time-frequency domain, e.g., by analyzing a spectrogram 202 to detect clusters of spectral magnitude peaks associated with long-term crowd cheers (crowd noise), according to one embodiment. First, in step 510, a two-dimensional rectangular time-frequency analysis window 204 of size (F × T) is selected, where T is a value of several seconds (typically around 6 seconds) and F is the frequency range under consideration (typically 500 Hz to 3 kHz). Next, in step 520, a window overlap region N between adjacent analysis windows 204 is selected, and a window sliding step S = (TN) is calculated (typically around 1 second). The method proceeds to step 530, where the analysis window 204 slides along the spectral time axis. In step 540, a normalized magnitude is calculated at each position of the analysis window 204, followed by the average peak magnitude of the analysis window 204. The calculated average spectral peak magnitude represents the event indicator associated with each position in the analysis window 204. In step 550, a threshold is applied to each indicator value and an initial event vector in vector 206 is generated that includes indicator and position pairs as its elements.

[0126] As established above, the initial event vector may include a set of indicator-location pairs selected by thresholding in step 550. This vector may then be analyzed to identify closely spaced groups of indicators with closely spaced locations of adjacent elements. This process is illustrated in Figure 6.

[0127] 6 is a flow chart illustrating a method 600 for generating crowd noise event vectors, according to one embodiment. In step 610, an initial vector for a selected event may be read using a set of indicator / location pairs. In step 620, all selected indicator location runs with a location separation of S seconds between adjacent vector elements are added to the set of vectors.

number

[0128] FIG. 7 is a flowchart illustrating a method 700 for internal processing of each R vector, according to one embodiment. In step 710, the elements of R may be sorted in descending order by indicator value. The maximum indicator value may be extracted as the M parameters for the event. In step 720, the start / center time may be recorded for each of the vectors R as parameter P. In step 730, for each vector R, the number of elements may be counted and recorded as the duration parameter D of each vector R. For each event, triplets (M, P, D) may be formed that describe the event's intensity (loudness), start / center location, and / or duration. These triplets may replace the R vector as a new derived element that completely conveys the information sought about the crowd noise event. As shown in the flowchart of FIG. 7, subsequent processing may include, in step 740, combining the M, P, and D parameters of each R to form a new vector with (M, P, D) triplets as its elements. The event vectors are passed through a process of event interval selection, event duration selection, and event loudness (loudness indicator) selection to form a final timeline of detected crowd noise events.

[0129] FIG. 8 is a flowchart illustrating a method 800 for further selecting desired crowd noise events, according to one embodiment. According to one embodiment, method 800 can remove event vector elements spaced less than a minimum time distance between adjacent events. Method 800 can begin at step 810, where system 100 steps through the event vector elements one at a time. In query 820, the time distance to the previous event location can be tested. If, according to query 820, this time distance is below a threshold, the location can be skipped in step 830. If the time distance is not below the threshold, the location can be accepted in step 840. In either case, method 800 can proceed to query 850. If, according to query 850, the end of the event vector has been reached, a revised event vector can be generated, where vector elements are deemed too close to each other and removed. If the end of the event vector has not been reached, step 810 can continue, removing additional vector elements as needed.

[0130] FIG. 9 is a flowchart illustrating a method 900 for further selecting desired crowd noise events, according to one embodiment. Method 900 can remove event vector elements whose crowd noise duration is below a desired level. Method 900 can begin at step 910, where system 100 steps through the duration component of the event vector. In query 920, the duration component of the event vector element can be tested. If, according to query 920, the duration is below a threshold, the event vector element can be skipped in step 940. If the duration is not below the threshold, the event vector element can be accepted in step 930. In either case, method 900 can proceed to query 950. If, according to query 950, the end of the event vector has been reached, a modified event vector can be generated, and the vector element can be removed as being deemed to represent crowd noise of insufficient duration. If the end of the event vector has not been reached, step 910 can continue, where additional vector elements can be removed as needed.

[0131] FIG. 10 is a flowchart illustrating a method 1000 for further selecting a desired crowd noise event, according to one embodiment. Method 1000 can remove event vector elements whose crowd magnitude indicator is below a desired level. Method 1000 can begin at step 1010, where system 100 steps through the event vector and subsequent selection. The magnitude of the crowd noise event can be tested in query 1020. If the magnitude is below a threshold according to query 1020, the event vector element can be skipped in step 1040. If the magnitude is not below the threshold, the position can be accepted in step 1030. In either case, method 1000 can proceed to query 1050. If the end of the event vector has been reached according to query 1050, a modified event vector can be generated, and the vector element is removed for insufficient crowd noise magnitude. If the end of the event vector has not been reached, step 1010 can continue, removing additional vector elements as needed.

[0132] The event vector post-processing steps described in Figures 8, 9, and 10 can be performed in any desired order. The steps shown can be performed in any combination with each other, and some steps can be omitted. At the end of the event vector processing, a new final event vector can be generated that contains the desired event timeline of the sporting events.

[0133] In at least one embodiment, the automatic video highlight and associated metadata generation application receives a live broadcast audiovisual stream containing audio and video components, or a digital audiovisual stream received via a computer server, and processes audio data 154 extracted from the audiovisual stream using digital signal processing techniques to detect distinct crowd noises (e.g., audience cheers), as described above. These events can be sorted and selected using techniques described herein. The extracted information can then be added to sporting event metadata 224 associated with the sporting event television program video and / or video highlights 220. Such metadata 224 can be used, for example, to determine the start / end times of segments used in highlight generation. As described herein and in the related applications referenced above, the start and / or end times of a highlight can be adjusted based on an offset, which can be based on the amount of time available for the highlight, the importance and / or excitement level of the highlight, and / or any other suitable factor. Additionally or alternatively, the metadata 224 can be used to provide information to the user 150 during viewing of the audiovisual stream or highlights 220, such as a corresponding excitement level 230 or crowd excitement level 232.

[0134] The system and method have been described in particular detail with respect to possible embodiments. Those skilled in the art will appreciate that the system and method may be implemented in other embodiments. First, the specific names of components, capitalization of terms, attributes, data structures, or other programming or structural aspects are not required or important, and mechanisms and / or features may differ in name, format, or protocol. Furthermore, the system may be implemented via a combination of hardware and software, entirely with hardware elements, or entirely with software elements. Additionally, the specific division of functionality among various system components described herein is merely exemplary and not required. Functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.

[0135] References herein to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. The appearances of the phrases "in one embodiment" or "in at least one embodiment" in various places in the specification do not necessarily all refer to the same embodiment.

[0136] Various embodiments may include any number of systems and / or methods for performing the above techniques, either alone or in any combination. Another embodiment includes a computer program product comprising a non-transitory computer-readable storage medium and computer program code encoded on the medium for causing a processor within a computing device or other electronic device to perform the above techniques.

[0137] Some portions above are presented in terms of algorithms and symbolic representations of operations on data bits within a computing device's memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps (instructions) leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It is sometimes convenient, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. Further, without loss of generality, it is also convenient to refer to specific arrangements of steps requiring physical manipulations of physical quantities as modules or code devices.

[0138] It should be borne in mind, however, that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specified, as will be apparent from the description that follows, throughout the description, descriptions utilizing terms such as "processing" or "calculating" or "computing" or "displaying" or "determining" will be understood to refer to the actions and processes of a computer system or similar electronic computing module and / or device that manipulates and transforms data represented as physical (electronic) quantities in the computer system's memory or registers or other such information storage, transmission, or display devices.

[0139] Certain aspects include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions may be embodied in software, firmware and / or hardware, and if embodied in software, may be downloaded to reside on and operate from different platforms used by a variety of operating systems.

[0140] This document also relates to apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored on the computing device. Such computer programs may be stored on any type of computer-readable storage medium, including, but not limited to, a floppy disk, optical disk, CD-ROM, DVD-ROM, magneto-optical disk, read-only memory (ROM), random-access memory (RAM), EPROM, EEPROM, flash memory, solid-state drive, magnetic or optical card, application-specific integrated circuit (ASIC), or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus. The program and its associated data may also be hosted and run remotely, for example, on a server. Furthermore, the computing devices referred to herein may include a single processor or may be architectures employing multiple processor designs to increase computing power.

[0141] The algorithms and displays presented herein are not inherently related to any particular computing device, virtualization system, or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may be more convenient to construct specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description provided herein. Moreover, the systems and methods are not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​can be used to implement the teachings described herein, and that the above references to specific languages ​​are provided for the purpose of enabling and disclosing the best mode.

[0142] Accordingly, various embodiments include software, hardware, and / or other elements for controlling a computer system, computing device, or other electronic device, or any combination or plurality thereof. Such electronic devices may include, for example, a processor, input devices (such as a keyboard, mouse, touchpad, trackpad, joystick, trackball, microphone, and / or any combination thereof), output devices (such as a screen, speakers, etc.), memory, long-term storage (such as magnetic storage, optical storage, etc.), and / or network connectivity according to techniques known in the art. Such electronic devices may be portable or non-portable. Examples of electronic devices that can be used to implement the described systems and methods include desktop computers, laptop computers, televisions, smartphones, tablets, music players, audio devices, kiosks, set-top boxes, gaming systems, wearable devices, home electronic devices, server computers, etc. The electronic device may use any operating system, such as, but not limited to, Linux, Microsoft Windows available from Microsoft Corporation of Redmond, Washington, Mac OS X available from Apple Inc. of Cupertino, California, iOS available from Apple Inc. of Cupertino, California, Android available from Google, Inc. of Mountain View, California, and / or other operating systems adapted for use on the device.

[0143] While a limited number of embodiments have been described herein, those skilled in the art, having the benefit of the above description, will appreciate that other embodiments may be devised. Furthermore, it should be noted that the language used herein has been chosen primarily for ease of reading and didactic purposes, and may not have been chosen to delineate or limit the subject matter. Accordingly, the present disclosure is intended to be illustrative, but not limiting, in scope.

Claims

1. 1. A method for extracting metadata from a representation of an event, comprising: a processor receiving audio data for one or more events; the processor identifying one or more portions of the audio data that include crowd excitement data; identifying a spectral magnitude peak at each position of a time-frequency analysis window of the spectrogram of the audio data; generating a spectral indicator for each position of the time-frequency analysis window; using the spectral indicators to form a vector of spectral indicators having associated time portions, wherein the crowd excitement data corresponds to audio data that hits a threshold excitement level; and the processor extracting the crowd excitement data from the audio data; the processor determining start and end times for one or more segments of the audio data based on the crowd excitement data; the processor generating one or more highlights based on the start times and the end times of the one or more segments; the processor storing the one or more highlights in a data store.

2. The method of claim 1 , further comprising the processor adding the crowd excitement data to event metadata associated with the audio data.

3. The method of claim 1 , further comprising: the processor causing the one or more highlights to be displayed on an output device.

4. The method of claim 1 , further comprising the processor preprocessing the audio data by resampling the audio data to a desired sampling rate.

5. The method of claim 1 , further comprising the processor preprocessing the audio data by filtering the audio data to reduce or remove noise.

6. The method of claim 1 , wherein the processor identifying the one or more portions of the audio data that include the crowd excitement data comprises analyzing visual data that corresponds to the audio data.

7. A non-transitory computer-readable medium containing one or more sequences of instructions, which, when executed by a processor: receiving audio data for one or more events; the processor identifying one or more portions of the audio data that include crowd excitement data; identifying a spectral magnitude peak at each position of a time-frequency analysis window of the spectrogram of the audio data; generating a spectral indicator for each position of the time-frequency analysis window; using said spectral indicators to form a vector of spectral indicators with associated time portions; extracting, by the processor, the crowd excitement data from the audio data, the crowd excitement data corresponding to audio data hitting a threshold excitement level; the processor determining start and end times for one or more segments of the audio data based on the crowd excitement data; the processor generating one or more highlights based on the start times and the end times of the one or more segments; and the processor storing the one or more highlights in a data store.

8. The non-transitory computer-readable medium of claim 7 , wherein the operations further include the processor adding the crowd excitement data to event metadata associated with the audio data.

9. The non-transitory computer-readable medium of claim 7 , wherein the actions further include the processor causing the one or more highlights to be displayed on an output device.

10. 8. The non-transitory computer-readable medium of claim 7, wherein the operations further include the processor preprocessing the audio data by resampling the audio data to a desired sampling rate.

11. 8. The non-transitory computer-readable medium of claim 7, wherein the operations further include the processor preprocessing the audio data by filtering the audio data to reduce or remove noise.

12. 8. The non-transitory computer-readable medium of claim 7, wherein the processor identifying the one or more portions of the audio data that include the crowd excitement data comprises analyzing visual data that corresponds to the audio data.

13. 1. A computer system comprising: a processor; and a memory having stored therein program instructions that, when executed by the processor, receiving audio data for one or more events; identifying one or more portions of the audio data that include crowd excitement data; identifying a spectral magnitude peak at each position of a time-frequency analysis window of the spectrogram of the audio data; generating a spectral indicator for each position of the time-frequency analysis window; using said spectral indicators to form a vector of spectral indicators with associated time portions; extracting the crowd excitement data from the audio data; determining start and end times for one or more segments of the audio data based on the crowd excitement data, the crowd excitement data corresponding to audio data hitting a threshold excitement level; generating one or more highlights based on the start times and end times of the one or more segments; and storing the one or more highlights in a data store.

14. The computer system of claim 13 , wherein the operations further include adding the crowd excitement data to event metadata associated with the audio data.

15. The computer system of claim 13 , wherein the action further comprises causing the one or more highlights to be displayed on an output device.

16. 14. The computer system of claim 13, wherein the operations further comprise preprocessing the audio data by resampling the audio data to a desired sampling rate.

17. 14. The computer system of claim 13, wherein the operations further comprise preprocessing the audio data by filtering the audio data to reduce or remove noise.

18. 2. The method of claim 1, further comprising: the processor adjusting the start time or the end time of the one or more segments based on one or more offsets, the one or more offsets being based on an amount of time available for the one or more highlights.

19. 8. The non-transitory computer-readable medium of claim 7, wherein the operations further include the processor adjusting the start time or the end time of the one or more segments based on one or more offsets, the one or more offsets being based on an amount of time available for the one or more highlights.

20. 14. The computer system of claim 13, wherein the operations further include adjusting the start time or the end time of the one or more segments based on one or more offsets, the one or more offsets being based on an amount of time available for the one or more highlights.

Images