Audio processing for extraction of variable length disjoint segments from audiovisual content

JP2024133486A5Inactive Publication Date: 2024-11-19STATS LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024092728
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-06-13
Filing Date
2024-06-07
Publication Date
2024-11-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing television systems lack the ability to automatically and efficiently segment audiovisual content, such as sporting events, to generate high-quality video highlights in real-time, with smooth transitions and minimal audio interruptions.

Method used

An audio processing method that detects soft entry points in the audio stream, such as the end of syllables or sentences, to segment audiovisual content, using spectrogram analysis and non-maximal suppression to identify optimal transition points for video highlight generation, with optional fading functions for smoother transitions.

Benefits of technology

Enables automated real-time segmentation of audiovisual content for highlight generation with improved transition smoothness, reducing audio interruptions and facilitating the creation of high-quality video highlights.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

To provide a method, system, and non-temporary computer readable medium for adjusting boundaries of highlights of audiovisual content describing an event.SOLUTION: A method identifies temporary boundaries 2 of highlights, performs a search of a list of soft-entry points, generates the best approximation of one or more of the temporary boundaries of the highlights, adjusts the temporary boundaries according to the best approximation obtained from the list, extract the highlights using the modified boundaries, optionally process them by deploying a fading function using mirrored lead and trail curvature, and perform further smoothing of the audio transitions between segments that are mutually prime.SELECTED DRAWING: Figure 9
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 62 / 712,041, entitled “Audio Processing for Extraction of Variable Length Disjoint Segments from Television Signal,” filed July 30, 2018 (Attorney Docket No. THU006-PROV), which is incorporated by reference in its entirety.

[0002] This application claims priority from U.S. Provisional Application No. 62 / 746,454, “Audio Processing for Detecting Occurrences of Loud Sound Characterized by Short-Time Energy Bursts,” filed October 16, 2018 (Attorney Docket No. THU016-PROV), which is incorporated by reference in its entirety.

[0003] This application claims priority from U.S. Utility Application No. 16 / 440,229, entitled “Audio Processing for Extraction of Variable Length Disjoint Segments from Audiovisual Content,” filed June 13, 2019 (Attorney Docket No. THU006), which is incorporated by reference in its entirety.

[0004] This application is related to U.S. Utility Application No. 13 / 601,915, “Generating Excitement Levels for Live Performances,” filed August 31, 2012, and issued June 16, 2015 as U.S. Patent No. 9,060,210, which is hereby incorporated by reference in its entirety.

[0005] This application is related to U.S. Utility Application No. 13 / 601,927, for “Generating Alerts for Live Performances,” filed August 31, 2012, and issued September 23, 2014 as U.S. Patent No. 8,842,007, which is hereby incorporated by reference in its entirety.

[0006] This application is related to U.S. Utility Application No. 13 / 601,933, for “Generating Teasers for Live Performances,” filed August 31, 2012, and issued November 26, 2013 as U.S. Patent No. 8,595,763, which is hereby incorporated by reference in its entirety.

[0007] This application is related to U.S. Utility Application No. 14 / 510,481, entitled “Generating a Customized Highlight Sequence Depicting an Event,” filed on October 9, 2014 (Attorney Docket No. THU001), which is hereby incorporated by reference in its entirety.

[0008] This application is related to U.S. Utility Application No. 14 / 710,438, entitled “Generating a Customized Highlight Sequence Depicting Multiple Events,” filed May 12, 2015 (Attorney Docket No. THU002), which is hereby incorporated by reference in its entirety.

[0009] This application is related to U.S. Utility Application No. 14 / 877,691, entitled “Customized Generation of Highlight Show with Narrative Component,” filed on October 7, 2015 (Attorney Docket No. THU004), which is incorporated by reference in its entirety herein.

[0010] This application is related to U.S. Utility Application No. 15 / 264,928, entitled “User Interface for Interaction with Customized Highlight Shows,” filed Sep. 14, 2016 (Attorney Docket No. THU005), which is hereby incorporated by reference in its entirety.

[0011] This application is related to U.S. Utility Application No. 16 / 411,704, “Video Processing for Enabling Sports Highlights Generation,” filed May 14, 2019 (Attorney Docket No. THU009), the entire contents of which are incorporated herein by reference.

[0012] This application is related to U.S. Utility Application No. 16 / 411,710, filed May 14, 2019, for “Machine Learning for Recognizing and Interpreting Embedded Information Card Content” (Attorney Docket No. THU010), which is incorporated by reference in its entirety.

[0013] This application is related to U.S. Utility Application No. 16 / 411,713, “Video Processing for Embedded Information Card Localization and Content Extraction,” filed May 14, 2019 (Attorney Docket No. THU012), the entire contents of which are incorporated herein by reference.

[0014] This application is related to U.S. Utility Application No. 16 / 421,391, filed May 23, 2019, for “Audio Processing for Detecting Occurrences of Crowd Noise in Sporting Event Television Programming” (Attorney Docket No. THU007), which is incorporated by reference in its entirety.

[0015] This document relates to techniques for identifying multimedia content and associated information on a television device or video server that delivers the multimedia content, and enabling embedded software applications to utilize the multimedia content to provide content and services in synchronization with the delivery of the multimedia content. Various embodiments relate to methods and systems for providing automated audio analysis that segments program content depicting sporting events to create video highlights for in-game and post-game viewing. [Background technology]

[0016] Enhanced television applications such as interactive advertising, as well as enhanced program guides with pre-, during-, and post-game interactive applications have long been envisioned. Existing cable systems, originally designed for broadcast television, are being called upon to support a host of new applications and services, including interactive television services and enhanced (interactive) program guides.

[0017] Several frameworks have been standardized to enable enhanced television applications. Examples include the OpenCable™ Enhanced TV Application Messaging and Tru2way specifications, which refer to interactive digital cable services delivered over cable video networks and include features such as interactive program guides, interactive ads, and games. In addition, cable operator "OCAP" programming offers interactive services such as e-commerce shopping, online banking, electronic program guides, and digital video recording. These efforts have enabled the first generation of video synchronization applications that are synchronized with the video content delivered by the programmable / broadcaster and provide additional data and interactivity to television programming.

[0018] Recent developments in video / audio content analysis technologies and corresponding mobile devices have opened up a range of new possibilities for the development of sophisticated applications that operate in sync with live TV programming events. These new technologies, advances in audio signal processing and computer vision, and the improved computational power of modern processors, make it possible to generate sophisticated program content highlights in real time, accompanied by metadata that is currently lacking in television and other media environments. Summary of the Invention

[0019] A system and method are presented that enables automatic real-time segmentation of television programming of sporting events and / or other audiovisual content based on audio processing to detect low spectral activity and / or low volume entry points in the audio stream, such as the end of a syllable, the end of a word or group of words, and / or the end of a sentence (soft entry points). A list of detected soft entry points (with minimal spectral activity and / or low volume) can be used to extract segments of the audiovisual content according to criteria for video highlight generation.

[0020] In at least one embodiment, a spectrogram is constructed for the audio signal, allowing time-frequency analysis with a sliding 2D area window. For each position of the analysis window, a spectral modifier may be generated that represents the spectral activity within the analysis window. A vector of spectral modifiers with associated time positions may be formed, and the vector may be further divided into a set of consecutive 1 second intervals. For each 1 second interval, the internal modifiers may be sorted, and a non-maximal suppression may be performed to form a first vector of modifier / position pairs with maximized modifiers for each 1 second interval. In a subsequent step, the detected entry points within the 1 second interval may be processed to provide a desired average time spacing and / or provide further selection of modifier / position pairs.

[0021] After an initial anchor element (first element) is established in the first vector of modifier / location pairs, the next element of the first vector of modifier / location pairs can be selected at the minimum desired distance (e.g., 2 seconds). The immediate left and right neighbors of the next element can then be examined to generate a new anchor element with maximized modifiers for the local neighborhood. This process continues until all elements of the first vector of modifier / location pairs are exhausted, which can generate a new set of entry points (soft entries) with variable mutual distances of 2-3 seconds on average and with maximized spectral modifiers for each local neighborhood.

[0022] The vector of soft entries may then be converted into a list of best entry points and then applied to the video highlight generation. The boundaries of the highlighted video segments may be modified according to the best available soft entries in the vicinity of the original boundaries, and then the highlights may be extracted for further processing by the video highlight generation application.

[0023] In at least one embodiment, the extracted video highlight segments may also be processed by deploying an optional fading function with mirrored lead and trail curvatures applied to the segment boundaries, allowing further smoothing of transitions between the extracted decomposed video segments.

[0024] The methods presented herein are generalizable in the sense that they can be used for video segmentation in any application that requires smooth reassembly of segmented video with minimal unobtrusive audio transitions.

[0025] In at least one embodiment, a method for identifying boundaries of highlights in audiovisual content depicting an event may include storing audio data depicting at least a portion of the event, automatically analyzing the audio data to detect soft entry points in the audio data, and designating time indexes in the audiovisual content that correspond to the soft entry points as boundaries, the boundaries including a start or an end of the highlights. The audiovisual content may be, for example, a television broadcast of a sporting event.

[0026] The audiovisual content may include an audiovisual stream. The method may further include extracting audio data from the audiovisual stream prior to storing the audio data depicting at least a portion of the event.

[0027] The audiovisual content may include stored audiovisual content and / or real-time content. The method may further include extracting audio data from the stored audiovisual content and / or real-time content prior to storing the audio data depicting at least a portion of the event.

[0028] The event may be a sporting event. The highlights may depict portions of the sporting event that are deemed to be of particular interest to one or more users.

[0029] The method may further include playing one of the audiovisual content and the highlights at an output device during detection of the soft entry point.

[0030] The method may further include pre-processing the audio data by resampling the audio data to a desired sampling rate prior to detecting the soft entry point.

[0031] The method may further include pre-processing the audio data by filtering the audio data to reduce or remove noise prior to detecting the soft entry point.

[0032] The method may further include processing the audio data to generate a spectrogram of at least a portion of the audio data prior to detecting the soft entry point.

[0033] Detecting the soft entry point may include applying a sliding two-dimensional time-frequency analysis window to the spectrogram.

[0034] Detecting the soft entry point may include calculating an average spectral amplitude indicator for each position of a sliding two-dimensional time-frequency analysis window of the spectrogram, and using the average spectral amplitude indicator to form a vector of spectral amplitude indicator / position pairs of the spectrogram.

[0035] Detecting the soft entry point may further include converting the average spectral amplitude indicator of each vector element to an integer qualifier Q and generating an initial vector having a Q / position pair.

[0036] Detecting the soft entry point may further include stepping through elements of an initial vector having Q / position pairs and maximizing Q for each 1 second interval by performing non-maximum suppression of the Q modifier for each 1 second interval to form a first vector with a maximized Q modifier.

[0037] Detecting the soft entry point may further include stepping through the time component of each entry of the first vector with a maximized Q modifier, and for each time position, comparing the time component of the current position with the previous time component of the previous position to obtain a distance, and for each element of the first vector whose distance is greater than a threshold, finding the largest Q in an immediate neighborhood of the element's position, and inputting the Q / position pair with the largest Q in each neighborhood into a new soft entry vector.

[0038] The method may further include identifying the highlight with a tentative boundary prior to designating the time index as the boundary. Designating the time index as the boundary may include replacing the tentative boundary with a boundary obtained from a list of available soft entries.

[0039] Further details and variations are described herein. [Brief description of the drawings]

[0040] The accompanying drawings, together with the description, illustrate several embodiments. Those skilled in the art will recognize that the specific embodiments shown in the drawings are merely illustrative and are not intended to be limiting in scope.

[0041] [Figure 1A] FIG. 2 is a block diagram illustrating a hardware architecture in accordance with a client / server embodiment, where event content is provided via networked content providers. [Figure 1B] FIG. 11 is a block diagram illustrating a hardware architecture according to another client / server embodiment, where event content is stored on a client-based storage device. [Figure 1C] FIG. 2 is a block diagram illustrating a hardware architecture according to a stand-alone embodiment. [Figure 1D]FIG. 1 is a block diagram illustrating an overview of a system architecture, according to one embodiment. [Diagram 2] 1A-1C are schematic block diagrams illustrating example data structures that may be incorporated into the audio data, user data, and highlight data of FIGS. 1A-1C, according to one embodiment. [Figure 3A] 1 illustrates an example audio waveform graph showing occurrences of soft entry points in an audio stream extracted from television program content of a sporting event in the time domain, according to one embodiment. [Figure 3B] 3B illustrates an example of a spectrogram corresponding to the audio waveform graph of FIG. 3A in the time-frequency domain, according to one embodiment. [Figure 4] 1 is a flow chart illustrating a method for audio pre-processing by resampling, filtering, and spectrogram construction according to one embodiment. [Diagram 5] 4 is a flow chart illustrating a method for analyzing audio data, such as an audio stream, in the time-frequency domain and generating an initial vector of modifiers, according to one embodiment. [Figure 6] 1 is a flowchart illustrating a method for generating a vector with a modifier maximized within each one second interval, according to one embodiment. [Figure 7] 11 is a flowchart illustrating a method for further selecting soft entry points with variable spacing and maximized local neighborhood modifiers according to one embodiment. [Figure 8] 1 is a flow chart illustrating a method for optional refinement of detected entry points for sparse segmentation according to one embodiment. [Figure 9] 1 is a flow chart illustrating a method for assembling a tailored highlight based on a list of available soft entry points according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] definition The following definitions are offered for illustrative purposes only and are not intended to be limiting in scope. Event: For purposes of the description herein, the term "event" refers to a match, session, matchup, series, performance, program, concert, etc., or portions thereof (such as an act, period, quarter, half, inning, scene, chapter, etc.). An event may be a sporting event, an entertainment event, a particular performance of an individual or a subgroup within a larger group of participants of the event, etc. Examples of non-sporting events include television shows, breaking news, socio-political events, natural disasters, movies, plays, radio shows, podcasts, audiobooks, online content, performances, etc. An event may have any length. For illustrative purposes, the technology is often described herein in terms of sporting events, but those skilled in the art will recognize that the technology can be used in other contexts, including highlight shows of any audiovisual, audio, visual, graphics-based, interactive, non-interactive, or text-based content. Thus, the use of the term "sporting event" and any other sport-specific terminology in this description is intended to illustrate one contemplated embodiment, but is not intended to limit the scope of the described technology to that one embodiment. Rather, such terms should be considered to extend to any suitable non-sports context appropriate to the technology. For ease of explanation, the term "event" is also used to refer to a report or representation of an event, such as an audiovisual recording of an event, or any other content item that includes a report, description, or depiction of an event. Highlight: An excerpt or portion of an event, or an excerpt or portion of content related to an event that is deemed of particular interest to one or more users. Highlights can be of any length. In general, the technology described herein provides a mechanism for identifying and presenting a customized set of highlights (which may be selected based on specific characteristics and / or user preferences) for any suitable event. The term "highlight" may also be used to refer to a report or representation of a highlight, such as an audiovisual recording of a highlight, or any other content item that includes a report, description, or depiction of a highlight. Highlights need not be limited to depictions of the event itself, but can include other content associated with the event. For example, in the case of a sporting event, highlights can include audio / video from the match, as well as other content such as pre-match, in-match, and post-match interviews, analysis, commentary, etc. Such content may be recorded from linear television (e.g., as part of an audiovisual stream depicting the event itself) or obtained from any number of other sources. Various types of highlights may be provided, including, for example, occurrences (plays), strings, possessions, and sequences, all of which are defined below. A highlight need not be of fixed duration, but may incorporate a beginning and / or ending offset, as described below. Clip: A portion of an audio, visual, or audiovisual representation of an event. A clip may correspond to or represent a highlight. In many contexts herein, the term "segment" is used interchangeably with "clip." A clip may be a portion of an audio, video, or audiovisual stream, or may be a portion of stored audio, video, or audiovisual content. · Content Delineator: One or more video frames that indicate the start or end of a highlight. Occurrence: Something that occurs during an event. Examples include a goal, a play, a down, a hit, a save, a shot on goal, a basket scored, a stolen base, a snap or snap attempt, a near miss, a fight, the start or end of a game, quarter, half, period, or inning, a pitch, a penalty, an injury, a dramatic event at an entertainment event, a song, a solo, etc. An occurrence may also be an anomaly, such as a power outage, an unruly fan incident, etc. Detection of such occurrences may be used as the basis for determining whether to designate a particular portion of an audiovisual stream as a highlight. An occurrence is also referred to herein as a "play" for ease of terminology, although such usage should not be construed as limiting in scope. An occurrence may have any length, and a representation of an occurrence may have a variety of lengths. For example, as noted above, an expanded representation of an occurrence may include footage depicting the time periods immediately preceding and following the occurrence, while a simple representation may include only the occurrence itself. Optional intermediate representations may also be provided. In at least one embodiment, the selection of a duration for representing an occurrence may depend on user preferences, available time, a determined excitement level for the occurrence, the importance of the occurrence, and / or any other factors. Offset: the amount by which the length of the highlight is adjusted. In at least one embodiment, a start offset and / or end offset may be provided to adjust the start time and / or end time, respectively, of the highlight. For example, if the highlight depicts a goal, the highlight may be extended by a few seconds (via the end offset) to include the celebration and / or fan reaction following the goal. The offset may be configured to change automatically or manually based, for example, on the amount of time available for the highlight, the importance and / or excitement level of the highlight, and / or any other suitable factors. String: A series of occurrences that are linked or related to each other in some way. An occurrence may occur within a Possession (defined below) or across multiple possessions. An occurrence may occur within a Sequence (defined below) or across multiple Sequences. Occurrences may be linked or related because they have some thematic or narrative connection to each other, or because one leads to another, or for any other reason. An example of a string is a set of passes that lead to the scoring of a goal or basket. This should not be confused with "text string", which has the meaning typically assigned to strings in the computer programming arts. Possession: any time-delimited portion of an event. The start / end time demarcation of possession may depend on the type of event. For certain sporting events (e.g., basketball or soccer, etc.) where one team may be offensive while the other team is defensive, possession may be defined as the period of time when one of the teams has the ball. In sports where possession of the puck or ball is more fluid, such as hockey or soccer, possession may be considered to extend to the period of time when one of the teams has substantial control of the puck or ball, ignoring momentary contact by the other team (such as a blocked shot or save). For baseball, possession is defined as a half-inning. For football, possession may include several sequences in which the same team has the ball. For other types of sporting events as well as non-sporting events, the term "possession" may be somewhat of a misnomer, but is still used herein for purposes of explanation. Examples in non-sporting contexts may include chapters, scenes, acts, etc. For example, in the context of a music concert, a possession may correspond to the performance of a single song. A possession may include any number of occurrences. Sequence: A time-bound portion of an event that includes a time period of one continuous action. For example, in a sporting event, a sequence may begin at the start of the action (such as a faceoff, or a jump ball) and end when the whistle is blown to indicate a stop in the action. In sports such as baseball or football, a sequence may equate to a play, which is a form of occurrence. A sequence may include any number of possessions or may be a portion of a possession. Highlight Show: A set of highlights configured for presentation to a user. A highlight show may be presented linearly (such as an audiovisual stream) or may be presented to allow the user to select which highlights to view and in what order (e.g., by clicking links or thumbnails). Presentation of a highlight show may be non-interactive or interactive, for example allowing the user to pause, rewind, skip, fast forward, communicate thumbs up or thumbs down preferences, etc. A highlight show may be, for example, a condensed match. A highlight show may include any number of consecutive or non-consecutive highlights from a single event or from multiple events, and may even include highlights from different types of events (e.g., a combination of highlights from different sports and / or sporting and non-sporting events). · User / Viewer: The terms "user" or "viewer" refer interchangeably to an individual, group, or other entity that watches, hears, or otherwise experiences the event, one or more highlights of the event, or a highlight show. The terms "user" or "viewer" may also refer to an individual, group, or other entity that may watch, hear, or otherwise experience either the event, one or more highlights of the event, or a highlight show at some future point in time. Although the term "viewer" is sometimes used for descriptive purposes, an event need not include a visual component, so a "viewer" may instead be a listener or any other consumer of the content. · Excitement Level: A measure of how exciting or interesting an event or highlight will be to a particular user or users in general. Excitement level may also be determined with respect to a particular occurrence or player. Various techniques for measuring or assessing excitement level are described in the related applications referenced above. As described, excitement level may depend on the occurrence within the event and other factors such as the overall context or importance of the event (such as a playoff match, pennant impact, and / or rivalry). In at least one embodiment, an excitement level may be associated with each occurrence, string, possession, or sequence within the event. For example, the excitement level of a possession may be determined based on the occurrences that occur within that possession. Excitement level may be measured differently by different users (e.g., fans of a team and neutral fans) and may depend on the personal characteristics of each user. Metadata: Data that is related to and stored in association with other data. Primary data may be media such as sports programming or highlights. Video Data. A length of video, which may be in digital or analog form. The video data may be stored on a local storage device or may be received in real time from a source such as a TV broadcast antenna, a cable network, or a computer server, in which case it may be referred to as a "video stream." The video data may or may not include an audio component, in which case it may be referred to as "audiovisual data" or an "audiovisual stream." Audio data. A length of audio, which may be in digital or analog form. The audio data may be the audio component of audiovisual data or an audiovisual stream, and may be separated by extracting the audio data from the audiovisual data. The audio data may be stored on a local storage device or may be received in real time from a source such as a TV broadcast antenna, a cable network, or a computer server, in which case it may also be referred to as an "audio stream." Stream. An audio stream, a video stream or an audiovisual stream. Time Index. An indicator of a time within the audio, video, or audiovisual data at which an event occurs or otherwise relates to a specified segment, such as a highlight. Spectrogram. A visual representation of the frequency spectrum of a signal, such as an audio stream, as it varies over time. A spectrogram can be a two-dimensional time-frequency representation of an audio signal derived by applying a short-time Fourier transform (STFT) to the audio signal. Analysis window: a specified subset of video data, audio data, audiovisual data, spectrogram, stream, or an otherwise processed version of a stream or data, in which one step of analysis is focused. Audio data, video data, audiovisual data, or spectrograms can be analyzed in segments, for example, using a moving analysis window and / or a series of analysis windows covering different segments of the data or spectrogram. Boundary. A demarcation that separates one audio, video, and / or audiovisual segment from another. A boundary may be the start or end of a segment, such as a highlight of audiovisual content, such as a television broadcast. A boundary may be provisional (i.e., intended for provisional and / or subsequent replacement) or final. In some embodiments, a highlight may first be identified with a provisional boundary. An audio analysis may be performed to identify a soft entry point that is then used to locate (in time) the final boundary of the highlight. Soft entry point. A part of an audio, video, or audiovisual stream that represents a likely or preferred transition point between different scenes, types of content, plays, etc. In the context of a sporting event, a soft entry point might be a quieter part of the audio signal representing commentary during a match or a pause between plays.

[0043] overview In at least one embodiment, the systems and methods described herein perform automatic real-time variable-length segmentation of audiovisual programs, such as television programs, based on audio processing to detect low activity entry points ("soft entry points"), such as the end of syllables, sentences, and / or groups of words. These entry points can be used as guides in extracting segments of the audiovisual program, which can facilitate highlight generation by improving the transition from one highlight to the next, avoiding cut-outs in dialogue or other sounds, and avoiding abrupt transitions.

[0044] In at least one embodiment, an automated video highlight and associated metadata generation application can receive a live broadcast audiovisual stream or a digital audiovisual stream received via a computer server. The application can then process audio data, such as an audio stream extracted from the audiovisual stream, using, for example, digital signal processing techniques to detect soft entry points.

[0045] In alternative embodiments, the techniques described herein can be applied to other types of source content. For example, the audio data need not be extracted from an audiovisual stream, but rather may be a radio broadcast or other audio depiction of a sporting or other event. Alternatively, the techniques described herein can be applied to stored audio data depicting an event, such data may or may not be extracted from the stored audiovisual data.

[0046] An interactive television application enables timely and appropriate presentation of highlighted television program content to a user watching a television program on either a primary television display or a secondary display such as a tablet, laptop, or smartphone. In at least one embodiment, a set of clips representing highlights of the television broadcast content are generated and / or stored in real time along with a database including time-based metadata that more fully describes the events presented by the highlight clips. As described in more detail herein, start and / or end times of such clips may be determined, at least in part, based on an analysis of extracted audio data.

[0047] In various embodiments, the metadata associated with a clip may be any information, such as text information, images, and / or any type of audiovisual data. One type of metadata associated with both in-game and post-game video content highlights presents events detected by real-time processing of audio data extracted from television coverage or other audiovisual content of a sporting event. In various embodiments, the systems and methods described herein enable automatic metadata generation and video highlight processing, where start and / or end times of a highlight may be detected and / or refined by analyzing digital audio data, such as an audio stream.

[0048] For example, highlights can be identified by analyzing such audio data to detect cheering crowd noise following a particular exciting event, audio announcement, music, etc. Additionally or alternatively, highlights can be detected in any of the other ways presented in any of the parent applications listed above. Identification of soft entry points can be used to identify or refine start and / or end times of highlights so that they begin and / or end at natural breaks in the audiovisual content, thus avoiding unnecessarily abrupt transitions. In at least one embodiment, real-time processing is performed on the audio data, which may be an audio stream extracted from television programming content of a sporting event, to detect, select, and track such soft entry points.

[0049] In at least one embodiment, a method for automated real-time processing of audio signals extracted from audiovisual content, such as television program content of sporting events, detects soft entry points. The method may include capturing, decoding, and pre-processing the audio signal, generating a time-frequency audio spectrogram for combined time-frequency analysis to detect areas of low spectral activity, generating a spectral indicator of overlapping spectrogram areas, and forming a vector of spectral modifiers with associated time positions. The method may further include dividing the vector into consecutive one-second intervals, sorting the intra-spectral modifiers for each one-second interval, and performing non-maximum suppression to form a first vector of modifier / position pairs. Furthermore, the method may include processing the first vector of each modifier / position pair to provide a desired average time spacing and provide further selection of modifier / position pairs. Still further, the method may include forming a list of best entry points and applying the list of best entry points to video highlight generation, for example, by modifying the boundaries of the highlighted video segments according to the best available soft entry in the vicinity of the original boundaries.

[0050] In at least one embodiment, the system and method receives compressed audio data, reads, decodes, and resamples the compressed audio data to a desired sampling rate. Pre-filtering can be performed for noise reduction, click removal, and selection of frequency bands of interest. Any of a number of interchangeable digital filtering stages can be used.

[0051] The overlapping spectrogram areas can be analyzed by a 2D time-frequency window with a time span of less than 1 second. In at least one embodiment, the analysis time-frequency window is slid along the spectrogram time coordinate and the normalized average amplitude of the window is calculated at each overlapping window position.

[0052] The average amplitude may be a spectral indicator generated at each analysis window position. An initial vector of spectral indicators with associated time positions may be formed and further divided into consecutive 1 second intervals. For each 1 second interval, the internal modifiers may be sorted and non-maximum suppression may be performed to form a first vector of modifier / position pairs. The detected entry points within the 1 second interval may be processed to provide a desired average time spacing and provide further selection of modifier / position pairs. For purposes of illustration, the description herein refers to 1 second intervals, however, one skilled in the art will recognize that intervals of any suitable length may be used.

[0053] After an initial anchor element (first element) is established, the next element can be selected at the smallest desired distance with a length, such as 2 seconds. Other lengths can also be used. The elements in the immediate left and right neighborhood of the next element can be examined to generate a new anchor element with maximized modifiers for the local neighborhood. All first vectors of modifier / location pairs can be processed in successive steps to generate a new set of soft entry points with variable mutual distances, for example, 2-3 seconds on average, and with maximized spectral modifiers for each local neighborhood.

[0054] The vector of soft entries may be converted into a list of best entry points and then applied to the video highlight generation. The boundaries of the highlighted video segments may be modified according to the best available soft entries in the vicinity of the original boundaries. The highlighted video segments with modified boundaries may be extracted for further processing by the video highlight generation application. Furthermore, the extracted video highlight segments may be processed by developing an optional fading function with mirrored lead and trail curvatures applied to the segment boundaries.

[0055] System Architecture According to various embodiments, the system can be implemented in any electronic device or set of electronic devices equipped to receive, store, and present information. Such electronic devices may be, for example, desktop computers, laptop computers, televisions, smartphones, tablets, music players, audio devices, kiosks, set-top boxes (STBs), match systems, wearable devices, consumer electronic devices, etc.

[0056] Although the system is described herein with reference to implementation on a particular type of computing device, those skilled in the art will recognize that the techniques described herein may be implemented in other contexts, and indeed on any suitable device capable of receiving and / or processing user input and presenting output to a user. Accordingly, the following description is intended to illustrate various embodiments by way of example, rather than limiting in scope.

[0057] 1A, a block diagram is shown illustrating a hardware architecture of a system 100 for automatically analyzing audio data to detect soft entry points and designate highlight boundaries according to a client / server embodiment. Event content, such as an audiovisual stream including audio content, may be provided via a networked content provider 124. An example of such a client / server embodiment is a web-based implementation, where one or more client devices 106 each execute a browser or app that provides a user interface for interacting with content from various servers 102, 114, 116, including data provider(s) server(s) 122 and / or content provider(s) server(s) 124, via a communication network 104. Transmission of content and / or data in response to requests from the client devices 106 may be performed using any known protocol and language, such as Hypertext Markup Language (HTML), Java, Objective C, Python, JavaScript, etc.

[0058] The client device 106 may be any electronic device, such as a desktop computer, a laptop computer, a television, a smartphone, a tablet, a music player, an audio device, a kiosk, a set-top box, a match system, a wearable device, a consumer electronic device, or the like. In at least one embodiment, the client device 106 has several hardware components known to those skilled in the art. The input device(s) 151 may be any component(s) that receives input from the user 150, including, for example, a handheld remote control, a keyboard, a mouse, a stylus, a touch-sensitive screen (touch screen), a touch pad, a gesture receptor, a trackball, an accelerometer, a five-way switch, or a microphone. The input may be provided via any suitable mode, including, for example, one or more of pointing, tapping, typing, dragging, gestures, tilting, shaking, and / or speech. The display screen 152 may be any component that graphically displays information, video, content, or the like, including depictions of events, highlights, and the like. Such output may also include queries requesting information and / or parameters for selection of, for example, audiovisual content, data visualization, navigational elements, graphical elements, content, metadata, etc. In at least one embodiment in which only a portion of a desired output is presented at a time, dynamic control, such as a scrolling mechanism, may be available via input device(s) 151 to select which information is currently displayed and / or to change the manner in which the information is displayed.

[0059] The processor 157 may be a conventional microprocessor for performing operations on data under the direction of software in accordance with well-known techniques. The memory 156 may be a random access memory having a structure and architecture known in the art for use by the processor 157 in the course of executing software to perform the operations described herein. The client device 106 may also include local storage (not shown), which may be a hard drive, a flash drive, an optical or magnetic storage device, a web-based (cloud-based) storage device, or the like.

[0060] Any suitable type of communications network 104, such as the Internet, a television network, a cable network, a cellular network, etc., may be used as a mechanism for transmitting data between the client device 106 and the various server(s) 102, 114, 116, and / or content provider(s) 124, and / or data provider(s) 122, according to any suitable protocols and techniques. In addition to the Internet, other examples include cellular telephone networks, EDGE, 3G, 4G, Long Term Evolution (LTE), Session Initiation Protocol (SIP), Short Message Peer-to-Peer Protocol (SMPP), SS7, Wi-Fi, Bluetooth, ZigBee, Hypertext Transfer Protocol (HTTP), Secure Hypertext Transfer Protocol (SHTTP), Transmission Control Protocol / Internet Protocol (TCP / IP), or the like, and / or any combination thereof. In at least one embodiment, the client device 106 sends a request for data and / or content over the communications network 104 and receives a response from the server(s) 102, 114, 116 that includes the requested data and / or content.

[0061] In at least one embodiment, the system of FIG. 1A operates in connection with a sporting event. However, it should be understood that the teachings herein apply to events other than sporting events, and the techniques described herein are not limited to application to sporting events. For example, the techniques described herein can be utilized to operate in connection with, or for two or more such events, television shows, movies, news events, game shows, political events, business shows, dramas, and / or other episodic content.

[0062] In at least one embodiment, the system 100 identifies highlights of audiovisual content depicting an event, such as a broadcast of a sporting event, by analyzing audio content representative of the event. This analysis can be performed in real-time. In at least one embodiment, the system 100 includes one or more web server(s) 102 coupled to one or more client devices 106 via a communication network 104. The communication network 104 may be a public network, a private network, or a combination of public and private networks, such as the Internet. The communication network 104 may be a LAN, a WAN, wired, wireless, and / or a combination of the above. The client devices 106, in at least one embodiment, may connect to the communication network 104 via either a wired or wireless connection. In at least one embodiment, the client devices may also include a recording device capable of receiving and recording the event, such as a DVR, PVR, or other media recording device. Such a recording device may be part of the client device 106 or may be external. In other embodiments, such a recording device may be omitted. Although FIG. 1A shows one client device 106, the system 100 may be implemented with any number of client device(s) 106 of a single type or multiple types.

[0063] The web server(s) 102 may include one or more physical computing devices and / or software capable of receiving requests from the client device(s) 106 and responding to those requests with data, as well as sending unsolicited alerts and other messages. The web server(s) 102 may employ various strategies for fault tolerance and scalability, such as load balancing, caching, and clustering. In at least one embodiment, the web server(s) 102 may include caching techniques as known in the art for storing information related to client requests and events.

[0064] The web server(s) 102 may manage or otherwise designate one or more application server(s) 114 to respond to requests received from the client device(s) 106. In at least one embodiment, the application server(s) 114 provide access to business logic for use by client application programs in the client device(s) 106. The application server(s) 114 may be co-located, shared, or co-managed with the web server(s) 102. The application server(s) 114 may also be separate from the web server(s) 102. In at least one embodiment, the application server(s) 114 interact with one or more analytic server(s) 116 and one or more data server(s) 118 to perform one or more operations of the disclosed techniques.

[0065] One or more storage devices 153 may function as a "data store" by storing data related to the operation of system 100. This data may include, for example, but not limited to, audio data 154 representing one or more audio signals. The audio data 154 may be derived from audiovisual streams or stored audiovisual content representing, for example, sporting events and / or other events.

[0066] The audio data 154 may include any information related to the audio embedded in the audiovisual stream, such as the audio stream accompanying the video image, a processed version of the audiovisual stream, and metrics and / or vectors related to the audio data 154, such as the time index, duration, amplitude, and / or other parameters of the event. The user data 155 may include any information describing one or more users 150, including, for example, demographics, purchasing behavior, audiovisual stream viewing behavior, interests, preferences, etc. The highlight data 164 may include highlights, highlight identifiers, time indicators, categories, excitement levels, and other data related to the highlights. The audio data 154, the user data 155, and the highlight data 164 are described in more detail below.

[0067] In particular, many components of the system 100 may be or include computing devices. Each of such computing devices may have an architecture similar to that of the client device 106, as shown and described above. Thus, any of the communication network 104, the web server 102, the application server 114, the analytics server 116, the data provider 122, the content provider 124, the data server 118, and the storage device 153 may include one or more computing devices, each of which may optionally have an input device 151, a display screen 152, a memory 156, and / or a processor 157, as described above in connection with the client device 106.

[0068] In an exemplary operation of the system 100, one or more users 150 of the client devices 106 view content from the content provider 124 in the form of an audiovisual stream. The audiovisual stream may show an event, such as a sporting event. The audiovisual stream may be a digital audiovisual stream that can be readily processed using known computer vision techniques.

[0069] Once the audiovisual stream is displayed, one or more components of the system 100, such as the client device 106, the web server 102, the application server 114, and / or the analysis server 116, analyze the audiovisual stream to identify highlights within the audiovisual stream and / or extract metadata from the audiovisual stream, for example, from the audio component of the stream. This analysis may occur in response to receiving a request to identify highlights and / or metadata of the audiovisual stream. Alternatively, in another embodiment, the highlights and / or metadata may be identified without a specific request made by the user 150. In yet another embodiment, the analysis of the audiovisual stream may occur without the audiovisual stream being displayed.

[0070] In at least one embodiment, the user 150 can specify, via the input device(s) 151 of the client device 106, certain parameters for the analysis of the audio data 154 (e.g., which events / matches / teams to include, how much time is available for the user 150 to view highlights, what metadata is desired, and / or any other parameters, etc.). User preferences can also be extracted from storage, such as from user data 155 stored in one or more storage devices 153, to customize the analysis of the audio data 154 without necessarily requiring the user 150 to specify preferences. In at least one embodiment, the user preferences can be determined based on observed behavior and actions of the user 150, for example, by observing website visiting patterns, television viewing patterns, music listening patterns, online purchases, prior highlight identification parameters, highlights and / or metadata actually viewed by the user 150, etc.

[0071] Additionally or alternatively, user preferences may be retrieved from pre-stored preferences explicitly provided by user 150. Such user preferences may indicate which teams, sports, players, and / or types of events are of interest to user 150, and / or they may indicate what type of metadata or other information related to highlights would be of interest to user 150. Such preferences may then be used to guide analysis of the audiovisual stream to identify highlights and / or extract metadata for the highlights.

[0072] The analytics server(s) 116, which may include one or more computing devices described above, may analyze live and / or recorded feeds of play-by-play statistics related to one or more events from the data provider(s) 122. Examples of the data provider(s) 122 may include, but are not limited to, providers of real-time sports information such as STATS™, Perform (available from Opta Sports, London, UK), and SportRadar, St. Gallen, Switzerland. In at least one embodiment, the analytics server(s) 116 generate a set of different excitement levels for the event. Such excitement levels may then be stored in conjunction with highlights identified or received by the system 100 in accordance with the techniques described herein.

[0073] The application server(s) 114 may analyze the audiovisual stream to identify highlights and / or extract metadata. Additionally or alternatively, such analysis may be performed by the client device(s) 106. The identified highlights and / or extracted metadata may be specific to a user 150, and in such cases, it may be advantageous to identify highlights within the client device 106 that are associated with a particular user 150. The client device 106 may receive, retain, and / or obtain applicable user preferences for highlight identification and / or metadata extraction, as described above. Additionally or alternatively, highlight generation and / or metadata extraction may be performed globally (i.e., using objective criteria applicable to the user population in general, regardless of the preferences of a particular user 150). In such cases, it may be advantageous to identify highlights and / or extract metadata within the application server(s) 114.

[0074] The content facilitating highlight identification, audio analysis, and / or metadata extraction may come from any suitable source, including from content provider(s) 124, which may include websites such as YouTube, MLB.com, sports data providers, television stations, client- or server-based DVRs, etc. Alternatively, the content may come from a local source, such as a DVR or other recording device associated with (or embedded in) the client device 106. In at least one embodiment, the application server(s) 114 generates a customized highlight show with highlights and metadata available to the user 150, either as download, or streaming content, or on-demand content, or in some other manner.

[0075] As noted above, it may be advantageous for user-specific highlight identification, audio analysis, and / or metadata extraction to be performed at a particular client device 106 associated with a particular user 150. Such an embodiment may avoid the need for video content or other high bandwidth content to be unnecessarily transmitted over the communications network 104, especially if such content is already available at the client device 106.

[0076] For example, referring now to FIG. 1B, an example of a system 160 according to an embodiment is shown in which at least a portion of the audio data 154 and highlight data 164 are stored on a client-based storage device 158, which may be any form of local storage device available to the client device 106. One example is a DVR on which an event, such as, for example, video content of a complete sporting event, may be recorded. Alternatively, the client-based storage device 158 may be any magnetic, optical, or electronic storage device for data in digital form. Examples include flash memory, magnetic hard drives, CD-ROMs, DVD-ROMs, or other devices integrated with or communicatively coupled to the client device 106. Based on information provided by the application server(s) 114, the client device 106 may extract highlights and / or metadata from the video content (e.g., including the audio data 154) stored on the client-based storage device 158 and store the highlights and / or metadata as highlight data 164, without having to retrieve other content from the content provider 124 or other remote sources. Such a configuration can conserve bandwidth and make effective use of existing hardware that may already be available to the client device 106 .

[0077] Returning to FIG. 1A , in at least one embodiment, application server(s) 114 can identify different highlights and / or extract different metadata for different users 150 depending on individual user preferences and / or other parameters. The identified highlights and / or extracted metadata may be presented to the user 150 via any suitable output device, such as a display screen 152 of the client device 106. If desired, multiple highlights can be identified and compiled into a highlight show along with associated metadata. Such highlight shows may be assembled into a “highlight reel” or set of highlights that are accessed via a menu and / or played for the user 150 according to a predetermined sequence. The user 150 can, in at least one embodiment, control the highlight playback and / or delivery of associated metadata via input device(s) 151, for example, to: Selecting specific highlights and / or metadata for display; - Pause, rewind, fast forward, Skip to the next highlight, Return to the start of the previous highlight within a highlight show, and / or · Perform other actions.

[0078] Additional details regarding such functionality are provided in the related US patent applications referenced above.

[0079] In at least one embodiment, one or more data server(s) 118 are provided. The data server(s) 118 can respond to requests for data from any of the server(s) 102, 114, 116, for example, to obtain or provide audio data 154, user data 155, and / or highlight data 164. In at least one embodiment, such information can be stored in any suitable storage device 153 accessible by the data server 118 and can come from any suitable source, such as from the client device 106 itself, the content provider(s) 124, the data provider(s) 122, etc.

[0080] 1C, an alternative embodiment of system 180 is shown in which system 180 is implemented in a stand-alone environment. Similar to the embodiment shown in FIG. 1B, at least a portion of audio data 154, user data 155, and highlight data 164 may be stored on a client-based storage device 158, such as a DVR. Alternatively, client-based storage device 158 may be a flash memory or a hard drive, or other device integrated with or communicatively coupled to client device 106.

[0081] User data 155 may include preferences and interests of user 150. Based on such user data 155, system 180 may extract highlights and / or metadata to present to user 150 in the manner described herein. Additionally or alternatively, highlights and / or metadata may be extracted based on objective criteria that are not based on information specific to user 150.

[0082] 1D, an overview of a system 190 having an architecture according to an alternative embodiment is shown. In FIG. 1D, the system 190 includes broadcast services such as content provider(s) 124, content receivers in the form of client devices 106 such as television sets with STBs, video servers such as analysis server(s) 116 that can ingest and stream audiovisual content such as television program content, and / or other client devices 106 such as mobile devices and laptops that can receive and process audiovisual content such as television program content, all connected via a network such as the communications network 104. A client-based storage device 158 such as a DVR can be connected to any of the client devices 106 and / or other components and can store audiovisual streams, highlights, highlight identifiers, and / or metadata to facilitate identification and presentation of highlights and / or extracted metadata via any of the client devices 106.

[0083] The particular hardware architectures shown in Figures 1A, 1B, 1C, and 1D are merely exemplary. Those skilled in the art will recognize that the techniques described herein can be implemented using other architectures. Many of the components shown herein are optional and may be omitted, combined with, and / or replaced by other components.

[0084] In at least one embodiment, the system can be implemented as software written in any suitable computer programming language, whether in a stand-alone or client / server architecture, or it may be implemented and / or embedded in hardware.

[0085] Data Structure FIG. 2 is a schematic block diagram illustrating example data structures that may be incorporated into audio data 154, user data 155, and highlight data 164, according to one embodiment.

[0086] As shown, the audio data 154 may include a record for each of a number of audio streams 200. For purposes of illustration, an audio stream 200 is shown, but the techniques described herein may be applied to any type of audio data 154 or content, whether streamed or stored. The record of the audio data 154 may include, in addition to the audio stream 200, other data generated pursuant to or useful in analyzing the audio stream 200. For example, the audio data 154 may include, for each audio stream 200, a spectrogram 202, one or more analysis windows 204, a vector 206, and a time index 208.

[0087] Each audio stream 200 may be in the time domain. Each spectrogram 202 may be calculated for the corresponding audio stream 200 in the time-frequency domain. The spectrograms 202 may be analyzed to more easily find soft entry points.

[0088] The analysis window 204 may be a designation of a predetermined time and / or frequency interval of the spectrogram 202. Computationally, a single moving (i.e., "sliding") analysis window 204 may be used to analyze the spectrogram 202, or a series of staggered (optionally overlapping) analysis windows 204 may be used.

[0089] The vector 206 may be a data set including intermediate and / or final results from the analysis of the audio stream 200 and / or the corresponding spectrogram 202 .

[0090] The time index 208 may indicate a time in the audio stream 200 (and / or the audiovisual stream from which the audio stream 200 is extracted) at which a significant event occurs. For example, the time index 208 may be a time in the audiovisual content at which a soft entry point begins, is centered, or ends. Thus, the time index 208 may indicate the start or end of a particularly interesting portion of the audiovisual stream, such as an important or memorable play in the context of a sporting event.

[0091] As further shown, user data 155 may include records associated with users 150, each of which may include demographic data 212, preferences 214, browsing history 216, and purchasing history 218 for a particular user 150.

[0092] Demographic data 212 may include any type of demographic data, including, but not limited to, age, gender, location, nationality, religious affiliation, education level, and the like.

[0093] Preferences 214 may include selections made by user 150 regarding his or her preferences. Preferences 214 may relate directly to collecting and / or viewing highlights and metadata or may be more general in nature. In either case, preferences 214 may be used to facilitate the identification and / or presentation of highlights and metadata to user 150.

[0094] The viewing history 216 may list television programs, audiovisual streams, highlights, web pages, search queries, sporting events, and / or other content searched for and / or viewed by the user 150 .

[0095] The purchase history 218 may list products or services purchased or requested by the user 150 .

[0096] As further shown, the highlight data 164 may include records for the j highlights 220 , and each record may include the audiovisual stream 222 and / or metadata 224 for the particular highlight 220 .

[0097] The audiovisual stream 222 may include audio and / or video depicting the highlight 220, which may be obtained from one or more audiovisual streams of one or more events (e.g., by cropping the audiovisual stream to include only the audiovisual stream 222 associated with the highlight 220). Within the metadata 224, the identifier 223 may include a time index (such as the time index 208 of the audio data 154) and / or other indicia indicating where the highlight 220 resides within the audiovisual stream of the event from which it was obtained.

[0098] In some embodiments, each recording of the highlight 220 may include only one of the audiovisual stream 222 and the identifier 223. Highlight playback may be performed by playing the audiovisual stream 222 to the user 150 or by using the identifier 223 to play only the highlighted portion of the audiovisual stream of the event from which the highlight 220 was obtained. Storage of the identifier 223 is optional. In some embodiments, the identifier 223 may be used only to extract the audiovisual stream 222 for the highlight 220, which may then be stored in place of the identifier 223. In either case, the time index 208 of the highlight 220 may be extracted from the audio data 154 and stored, at least temporarily, as metadata 224, which is added to the highlight 220 or to the audio data 154 and the audiovisual stream from which the highlight 220 was obtained. In some embodiments, the time index 208 may be stored as a boundary 232 of the identifier 223.

[0099] In addition to or as an alternative to the identifier 223, the metadata 224 may include information about the highlight 220, such as the date of the event, the season, and groups or individuals involved in the event or audiovisual stream from which the highlight 220 was captured, e.g., teams, players, coaches, presenters, broadcasters, and fans. Among other information, the metadata 224 for each highlight 220 may include a phase 226, a clock 227, a score 228, a frame number 229, and / or an excitement level 230.

[0100] Phase 226 may be a phase of the event associated with highlight 220. More specifically, phase 226 may be a stage of a sporting event within which the beginning, middle, and / or end of highlight 220 resides. For example, phase 226 may be the "third quarter," the "second inning," the "bottom inning," or the like.

[0101] The clock 227 may be the game clock associated with the highlight 220. More specifically, the clock 227 may be the state of the game clock at the beginning, middle, and / or end of the highlight 220. For example, the clock 227 may be "15:47" if 15 minutes and 47 seconds are displayed on the game clock in a highlight 220 that begins, ends, or spans a period of a sporting event.

[0102] The score 228 may be a match score associated with the highlight 220. More specifically, the score 228 may be a score at the beginning, end, and / or middle of the highlight 220. For example, the score 228 may be "45-38," "7-0," "30-love," or the like.

[0103] Frame number 229 may be the number of a video frame in the audiovisual stream from which highlight 220 is obtained or in audiovisual stream 222 associated with highlight 220 that pertains to the beginning, middle, and / or end of highlight 220.

[0104] The excitement level 230 may be a measure of how exciting or interesting an event or highlight will be to a particular user 150 or to users in general. In at least one embodiment, the excitement level 230 may be calculated as set forth in the related application referenced above. Additionally or alternatively, the excitement level 230 may be determined, at least in part, by analysis of the audio data 154, which may be components extracted from the audiovisual stream 222 and / or the audio stream 200. For example, audio data 154 including higher levels of crowd noise, announcements, and / or uptempo music may indicate a high excitement level 230 for the associated highlight 220. The excitement level 230 need not be static relative to the highlight 220, but may instead change over the course of the highlight 220. Thus, the system 100 may be able to further refine the highlight 220 to present only those portions that exceed a threshold excitement level 230 to the user.

[0105] The data structures depicted in Figure 2 are merely exemplary. Those skilled in the art will recognize that some of the data in Figure 2 may be omitted or replaced with other data in performing highlight identification and / or metadata extraction. Additionally or alternatively, data not specifically shown in Figure 2 or described in this application may be used in performing highlight identification and / or metadata extraction.

[0106] Analyzing Audio Data In at least one embodiment, the system performs several stages of analysis of audio data 154, such as an audio stream, in the time-frequency domain to detect soft entry points, such as pauses or low points in speech, music, or other sounds, during a depiction of a sporting event or another event. The depiction may be a television broadcast, an audiovisual stream, an audio stream, a stored file, or the like.

[0107] First, the compressed audio data 154 is read, decoded, and resampled to a desired sampling rate. The resulting PCM stream is then pre-filtered for noise reduction, click removal, and / or selection of desired frequency bands using any of several interchangeable digital filtering stages. A spectrogram is then constructed for the audio data 154. Valleys in the spectral amplitude are identified at each position of a sliding two-dimensional time-frequency area window. Further steps can be implemented to more completely and / or reliably identify the soft entry points 320.

[0108] The time indexes 208 corresponding to the soft entry points may be used as boundaries 232 (e.g., start or end) of the highlights 220. In some embodiments, these time indexes 208 may be used to identify actual start and / or end points of already identified highlights 220 (e.g., with interim boundaries 232 being interim start and end points that may be later adjusted based on the identification of the soft entry points). The highlights 220 may be extracted and / or identified within the video stream for subsequent viewing by a user.

[0109] 3A illustrates an example of an audio waveform graph 300 in an audio stream 310 extracted from sporting event television program content in the time domain, according to one embodiment. The highlighted area illustrates an example soft entry point 320, such as a lower intensity portion of the audio stream 310. The amplitude of the captured audio may be relatively low at the soft entry point 320, representing a relatively quiet portion of the audio stream 310.

[0110] FIG. 3B illustrates an example of a spectrogram 350 corresponding to the audio waveform graph 300 of FIG. 3A in the time-frequency domain, according to one embodiment. In at least one embodiment, detection and marking of occurrences of events of interest are performed in the time-frequency domain, and event boundaries 232 (not shown in FIGS. 3A and 3B) are presented in real time to a video highlight and metadata generation application. These boundaries 232 may be used to extract one or more highlights 220 from the video stream, or to determine with greater accuracy the beginning and / or end of each highlight 220 in the video stream, thereby allowing the highlights 220 to be played without inadvertently playing other content that represents portions of the video stream that are not part of the highlight. The boundaries 232 may be used to reduce the abruptness of transitions from one highlight 220 to another by helping to determine appropriate transition points in the content, such as at the end of a sentence or during pauses in the audio. In some embodiments, the boundaries 232 may be incorporated into metadata 224, such as identifiers 223 that identify the beginning and / or end of the highlights 220, as described in the description of FIG. 2.

[0111] Audio Data Analysis and Metadata Extraction 4 is a flow chart illustrating a method 400 for audio preprocessing by resampling, filtering, and spectrogram construction, according to one embodiment. In at least one embodiment, the method 400 may be performed by an application (e.g., running on one of the client device 106 and / or the analysis server 116) that receives the audiovisual stream and performs on-the-fly processing of the audio data 154 for identification of soft entry points 320, e.g., corresponding to boundaries 232 of highlights 220, according to one embodiment. According to the method 400, the audio data 154, such as the audio stream 310, may be processed to detect soft entry points 320 in the audio data 154 by detecting pauses, breaks, or other natural divisions between segments of audio, video, and / or audiovisual program content.

[0112] In at least one embodiment, method 400 (and / or other methods described herein) is performed on audio data 154 extracted from an audiovisual stream or other audiovisual content. Alternatively, the techniques described herein may be applied to other types of source content. For example, audio data 154 need not be extracted from an audiovisual stream, but rather may be a radio broadcast or other audio depiction of a sporting event or other event.

[0113] In at least one embodiment, the method 400 (and / or other methods described herein) may be performed by a system such as the system 100 of FIG. 1A. However, alternative systems, including but not limited to the system 160 of FIG. 1B, the system 180 of FIG. 1C, and the system 190 of FIG. 1D, may be used in place of the system 100 of FIG. 1A. Furthermore, the following description assumes that audio events of lower intensity are identified. However, it will be understood that different types of audio events may be identified and used to extract metadata and / or identify the boundaries 232 of the highlights 220 according to methods similar to those explicitly described herein. In some embodiments, certain audio and / or visual cues (tones, dark screens, etc.) may be identified as soft entry points.

[0114] The method 400 of FIG. 4 may begin at step 410 where audio data 154, such as audio stream 200, is read. If the audio data 154 is in a compressed format, it may optionally be decoded. In step 420, the audio data 154 may be resampled to a desired sampling rate. In step 430, the audio data 154 may be filtered using any of a number of interchangeable digital filtering stages. Then, in step 440, a spectrogram 202 may be generated for the filtered audio data 154, for example, by computing a short-time Fourier transform (STFT) on one-second chunks of the filtered audio data 154. The time-frequency coefficients of the spectrogram 202 may be stored in a two-dimensional array for further processing.

[0115] In some embodiments, if only relatively quiet areas in the audio stream are of interest, regardless of the spectral content, step 440 can be omitted and further analysis can be simplified by performing only on the time-domain audio data 154. However, in such cases, undesirable soft entry detection may occur due to an inherently unreliable indicator based only on an audio loudness threshold, without considering the spectral content associated with a particular sound of interest, such as a commentator's voice and / or background audience noise, which may be low volume in the time domain but have rich spectral content in the time-frequency domain. In some embodiments, analysis of the audio stream may also be performed in both the time and time-frequency domains, and the detected soft entry points may then be integrated into the final result. In the further description in Figures 5 to 10 below, it is assumed that step 440 has been performed and that the audio analysis step is performed on the spectrogram 202 corresponding to the audio data 154 (e.g., after decoding, resampling, and / or filtering the audio data 154 as described above). The final vector of soft entry points in the audio stream can be formed by focusing on detecting low volume and low spectral content of segments of the audio stream that are associated with gaps in the commentator's speech, such as, but not limited to, the end of words, groups of words, and sentences.

[0116] 5 is a flow chart showing a method 500 for analyzing audio data 154, such as an audio stream 200, in the time-frequency domain, for example by analyzing a spectrogram 202 to generate an initial vector of selected modifier / position pairs, according to one embodiment. First, in step 510, a two-dimensional rectangular time-frequency analysis window 204 of size (F×T) is selected, where T is a value of several seconds (typically around 100 ms) and F is the frequency range considered (typically 100 Hz to 3 kHz). Then, in step 520, a window overlap region N between adjacent analysis windows 204 is selected and a window sliding step S=(TN) is calculated (typically around 20 ms).

[0117] The method proceeds to step 530 where the analysis window 204 slides along the spectral time axis along the time axis of the spectrogram 202 in successive steps S. At each position of the analysis window 204, a mean spectral amplitude indicator (SMI) is calculated in step 540. A maximum SMI value M of the spectrogram may also be determined in step 550. In step 560, an SMI / position pair vector may be generated where the SMI is below a threshold.

[0118] In step 570, each SMI may be converted to an integer modifier Q according to the formula Q=(M-SMI)*MFACT, where MFACT is a factor used to extract the desired number of integer digits. In step 580, an initial vector of Q / position pairs may be generated as a superset of the potential soft entry points 320. The method 500 may then proceed to maximize the modifier for each 1 second interval.

[0119] 6 is a flow chart illustrating a method 600 for generating a vector with a maximized modifier within each 1 second interval, according to one embodiment. This may include step 610, where an initial vector with Q / position elements is split on 1 second boundaries. Within each successive 1 second interval, a sort by modifier Q may be performed. In step 620, only the Q / position pair with the maximum Q per 1 second interval may be retained. In step 630, a first soft entry vector with maximized Q value may be formed.

[0120] 7 is a flow chart illustrating a method 700 for further selecting soft entry points 320 with variable spacing and maximized local neighborhood modifiers, according to one embodiment. Such processing may generate extended variable size intervals (mutual distances) of soft entries (typically 2-3 seconds on average) and simultaneously maximize the spectral modifiers in the local neighborhood of each interval.

[0121] As shown, method 700 may begin 710 where an anchor point (previous position) is set to zero. Then, in step 720, method 700 may step through the time component of the first soft entry vector to find the next time position and load it into the current position. At each step, a query 730 may be made where the time distance from the current position to the previous position is checked against a threshold. If this distance is greater than a threshold (e.g., 2 seconds), the current position may be taken for further processing in step 740. If the distance is not greater than the threshold, step 720 may be repeated for a new time position.

[0122] Step 740 may include testing the immediate neighborhood of the current position that is held and identifying the element with the largest Q. In step 750, this element may be loaded into the previous position, which now becomes the new anchor point for further testing. In step 760, the final soft entry vector may also be populated with this locally maximized Q / position pair.

[0123] Method 700 may continue with successive steps until all elements of the first soft entry vector are exhausted. Specifically, a query 770 may check whether the end of the soft entry vector has been reached. If so, a final soft entry vector may be provided in step 780. If not, method 700 may return to step 720 for further iterations.

[0124] FIG. 8 is a flow chart illustrating a method 800 for optional refinement of detected entry points for sparse segmentation, according to one embodiment. The method 800 may be used to further maximize the Q modifier when there are less stringent requirements for the boundaries of the highlighted events. The method 800 may begin at step 810, where the method 800 steps through the soft entry vector elements one at a time. In query 820, the method 800 may test the Q value against a threshold. In step 840, the Q / location pairs below the threshold may be removed. In step 830, the Q / location pairs above the threshold may be retained.

[0125] Method 800 may continue with successive steps until all elements of the final soft entry vector are exhausted. Specifically, a query 850 may check whether the end of the final soft entry vector has been reached. If so, method 800 may proceed to forming and highlighting a list of soft entry points 320. If not, method 800 may return to step 810 for further iterations.

[0126] 9 is a flow chart illustrating a method 900 for assembling an adjusted highlight 220 based on a list of available soft entry points, according to one embodiment. First, in step 910, a tentative boundary 232 of the highlight 220 may be identified. Then, in step 920, a search of the list of soft entry points 320 may be performed to generate a best approximation of one or more tentative boundaries 232 of the highlight 220. Then, in step 930, the tentative boundary 232 may be adjusted according to the best approximation obtained from the list. In step 940, the highlight 220 may be extracted with the modified boundary, and optionally processed by developing a fading function with mirrored lead and trail curvatures to allow further smoothing of audio transitions between disjoint segments (such as multiple highlights 220 played consecutively, as in a highlight reel).

[0127] The present system and method have been described in particular detail with respect to the envisioned embodiment. Those skilled in the art will appreciate that the system and method may be practiced in other embodiments. First, the specific naming of components, capitalization of terms, attributes, data structures, or any other programming or structural aspects are not required or important, and mechanisms and / or functions may differ in name, format, or protocol. Furthermore, the system may be implemented through a combination of hardware and software, or entirely within hardware elements, or entirely within software elements. Also, the specific division of functionality between various system components described herein is merely exemplary and not required. Functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.

[0128] References herein to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. The appearances of the phrases "in one embodiment" or "in at least one embodiment" in various places in the specification do not necessarily all refer to the same embodiment.

[0129] Various embodiments may include any number of systems and / or methods for performing the techniques described above, either alone or in any combination. Another embodiment includes a computer program product including a non-transitory computer-readable storage medium and computer program code encoded thereon for causing a processor within a computing device or other electronic device to perform the techniques described above.

[0130] Some portions above are presented in terms of algorithms and symbolic representations of operations on data bits within a memory of a computing device. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is herein generally conceived to be a self-consistent sequence of steps (instructions) leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It is sometimes convenient, primarily for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. Further, without loss of generality, it is sometimes convenient to refer to certain arrangements of steps requiring physical manipulations of physical quantities as modules or code devices.

[0131] It should be borne in mind, however, that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise indicated, and as will be apparent from the following description, throughout this specification, descriptions utilizing terms such as "processing" or "computing" or "calculating" or "displaying" or "determining" will be understood to refer to operations and processes of a computer system, or similar electronic computing module and / or device, which manipulate and transform data that are represented as physical (electronic) quantities in the computer system's memory or registers or other such information storage, transmission, or display devices.

[0132] Certain aspects include process steps and instructions described herein in the form of an algorithm, it being noted that the process steps and instructions may be embodied in software, firmware, and / or hardware, and if embodied in software, may be downloaded to reside on and be operated from a variety of platforms for use by a variety of operating systems.

[0133] This document also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes or may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computing device. Such computer programs may be stored on a computer-readable storage medium such as a floppy disk, an optical disk, a CD-ROM, a DVD-ROM, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a flash memory, a solid-state drive, a magnetic or optical card, an application-specific integrated circuit (ASIC), or any type of medium suitable for storing electronic instructions, including but not limited to a disk, each coupled to a computer system bus. The program and its associated data may also be provided and executed remotely, for example, on a server. Furthermore, the computing device referred to in this specification may include a single processor or may be an architecture employing a multiple processor design to increase computing power.

[0134] The algorithms and displays presented herein are not inherently related to any particular computing device, virtualization system, or other apparatus. Various general-purpose systems may also be used with programs according to the teachings herein, or it may be more convenient to build a specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description provided herein. Furthermore, the systems and methods are not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​can be used to implement the teachings described herein, and any references above to a specific language are provided for the purposes of enabling and disclosing the best mode.

[0135] Thus, various embodiments include software, hardware, and / or other elements, or any or a combination of these, for controlling a computer system, computing device, or other electronic device. Such electronic devices may include, for example, a processor, input devices (such as a keyboard, mouse, touchpad, trackpad, joystick, trackball, microphone, and / or any combination thereof), output devices (such as a screen, speaker, etc.), memory, long-term storage (such as magnetic storage, optical storage, etc.), and / or network connectivity, according to techniques known in the art. Such electronic devices may be portable or non-portable. Examples of electronic devices that can be used to implement the described systems and methods include desktop computers, laptop computers, televisions, smartphones, tablets, music players, audio devices, kiosks, set-top boxes, gaming systems, wearable devices, home electronic devices, server computers, etc. The electronic device may use any operating system, such as, but not limited to, Linux, Microsoft Windows available from Microsoft Corporation, Redmond, Washington, Mac OS X available from Apple Inc., Cupertino, California, iOS available from Apple Inc., Cupertino, California, Android available from Google Inc., Mountain View, California, and / or any other operating system adapted for use on the device.

[0136] While a limited number of embodiments have been described herein, those skilled in the art having the benefit of the above description will appreciate that other embodiments may be devised. Furthermore, it should be noted that the language used herein has been selected primarily for ease of reading and instructional purposes, and may not have been selected to delineate or limit the subject matter. Thus, the present disclosure is intended to be illustrative, but not limiting in scope.

Claims

1. A method for identifying highlights of audiovisual content, comprising: A computer system generating an audio spectrogram of a portion of the audiovisual content; the computer system moving a time-frequency analysis window along the audio spectrogram to determine a plurality of average spectral amplitudes at a plurality of positions; the computer system determining a maximum of the plurality of average spectral amplitudes; generating a plurality of spectral modifiers for the plurality of locations based on the plurality of average spectral amplitudes and the maximum value, and identifying highlights in the audiovisual content based on the plurality of spectral modifiers, at least some of the plurality of spectral modifiers determining boundaries of the highlights.

2. The method of claim 1, wherein the boundary includes a soft entry point in the audiovisual content.

3. Generating the plurality of spectral modifiers comprises: The method of claim 1 , further comprising generating a plurality of spectral modifiers, each having an integer digit based on a predetermined coefficient.

4. Identifying the highlights comprises: The method of claim 1 , further comprising the computer system selecting, from the plurality of spectral modifiers, those spectral modifiers that each exceed a predetermined threshold.

5. Identifying the highlights comprises: the computer system dividing the plurality of spectral modifiers by a predetermined time interval; The method of claim 1 , further comprising: the computer system generating the portion of the plurality of spectral modifiers by selecting a maximum spectral modifier in each division.

6. The method of claim 1, further comprising the computer system specifying a time index corresponding to the boundary of the highlight in the audiovisual content.

7. The method of claim 6, further comprising the computer system causing an output device to play the highlights using the time index.

8. A non-transitory storage medium for storing computer program instructions; a processor, the processor executing the computer program instructions to generating an audio spectrogram of a portion of the audiovisual content; moving a time-frequency analysis window along the audio spectrogram to determine a plurality of average spectral amplitudes at a plurality of positions; determining a maximum of the plurality of average spectral amplitudes; generating a plurality of spectral modifiers for the plurality of locations based on the plurality of average spectral amplitudes and the maximum value; and identifying highlights in the audiovisual content based on the plurality of spectral modifiers, wherein at least some of the plurality of spectral modifiers determine boundaries of the highlights.

9. The system of claim 8, wherein the boundary includes a soft entry point in the audiovisual content.

10. Generating the plurality of spectral modifiers comprises: The system of claim 8 , further comprising generating a plurality of spectral modifiers, each having an integer digit based on a predetermined coefficient.

11. Identifying the highlights comprises: The system of claim 8 , comprising selecting, from the plurality of spectral modifiers, those spectral modifiers that each exceed a predetermined threshold.

12. Identifying the highlights comprises: dividing the plurality of spectral modifiers by a predetermined time interval; and generating the portion of the plurality of spectral modifiers by selecting a maximum spectral modifier in each division.

13. The system of claim 8, wherein the action includes specifying a time index corresponding to the boundary of the highlight in the audiovisual content.

14. The system of claim 13, wherein the action includes causing an output device to play the highlight using the time index.

15. A non-transitory storage medium storing computer program instructions, the computer program instructions which, when executed by a processor: generating an audio spectrogram of a portion of the audiovisual content; moving a time-frequency analysis window along the audio spectrogram to determine a plurality of average spectral amplitudes at a plurality of positions; determining a maximum of the plurality of average spectral amplitudes; generating a plurality of spectral modifiers for the plurality of locations based on the plurality of average spectral amplitudes and the maximum value; and identifying highlights in the audiovisual content based on the plurality of spectral modifiers, at least some of the plurality of spectral modifiers determining boundaries of the highlights.

16. The non-transitory storage medium of claim 15, wherein the boundary includes a soft entry point in the audiovisual content.

17. The method of claim 17, further comprising:

20. The non-transitory storage medium of claim 15, comprising generating a plurality of spectral modifiers, each having an integer digit based on a predetermined coefficient.

18. Identifying the highlights comprises: The non-transitory storage medium of claim 15 , comprising selecting, from the plurality of spectral modifiers, those spectral modifiers that each exceed a predetermined threshold.

19. The method of claim 18, further comprising: dividing the plurality of spectral modifiers by a predetermined time interval; and generating the portion of the plurality of spectral modifiers by selecting a maximum spectral modifier in each division.

20. The non-transitory storage medium of claim 15, wherein the operation includes specifying a time index corresponding to the boundary of the highlight in the audiovisual content.