System and method for detecting advertising in multimedia assets

Abstract

Method comprising: obtaining an audio signature corresponding to a time period of a multimedia asset; determining a match between the received audio signature and one or more stored audio signatures, wherein the stored audio signatures correspond to time periods of a plurality of other multimedia assets; comparing program data of the multimedia assets of the obtained audio signature and the one or more matching audio signatures; and determining whether the time period of the multimedia asset contains an advertisement based on comparing the program data of the multimedia asset, the received audio signature and the one or multiple matching audio signatures.

Description

[0001] This application claims the benefit of US Provisional Application No. 62 / 204,637 filed August 13, 2015, the contents of which are hereby incorporated by reference. TECHNICAL FIELD

[0002] This invention relates to the field of multimedia identification and recognition and, more particularly, to systems and methods for identifying or recognizing advertising in audiovisual multimedia such as television programs. BACKGROUND

[0003] Typically, audiovisual multimedia content such as television programs or video streams contains embedded advertisements. For example, to remove or replace advertisements, it may be desirable to determine the position of advertisements within the multimedia content. Markers (sometimes called "bumpers") can be used to identify or mark the beginning and end of advertisements within a given piece of multimedia content. However, such markers are often not used. Because the position of advertisements and their length often vary, identifying the position of advertisements sometimes requires significant human involvement.

[0004] US 2014 / 0 282 662 A1 shows a prior art example of automatically identifying advertisements or other parts of a television stream before the stream is distributed via cable, DBS, IPTV or other medium. OVERVIEW

[0005] The present invention describes systems and methods for determining the position of advertisements in multimedia assets.

[0006] An example method includes obtaining an audio signature corresponding to a time period of a multimedia asset, determining a match between the obtained audio signature and one or more stored audio signatures, comparing program data of the multimedia assets of the obtained audio signatures and the one or more matching audio signatures, and determining whether the time period of the multimedia asset contains advertising based on the comparison of the program data of the multimedia assets of the obtained audio signature and the one or more matching audio signatures.

[0007] Another example method includes obtaining a plurality of audio signatures corresponding to consecutive time portions of the multimedia asset, determining matches between the obtained audio signatures and a plurality of stored audio signatures, and determining whether the consecutive time portions of the multimedia asset contain advertisements based on the number of consecutive matching audio signatures of the plurality of stored audio signatures.

[0008] An example system includes a receiver, computer memory, and a processor. The receiver is configured to receive audio and video data from a multimedia stream. The computer memory contains a database of one or more stored audio signatures. The stored audio signatures correspond to time periods of a plurality of multimedia assets. The processor is in communication with the receiver and the computer memory.The processor is programmed to obtain an audio signature corresponding to a time period of a multimedia asset in the multimedia stream, to determine a match between the obtained audio signature and the one or more stored audio signatures, to compare the program data of the multimedia assets of the obtained audio signature and the one or more matching audio signatures, and to determine whether the time period of the multimedia asset contains advertising based on the comparison of the program data of the multimedia asset of the obtained audio signature and the one or more matching audio signatures. In addition to the foregoing system, a non-transitory computer-readable medium may contain computer instructions that cause the computer to perform the foregoing steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Further details of the example implementations are explained below in conjunction with the attached drawings. They show: Fig. 1: a block diagram of an advertising detection system; Fig. 2: a block diagram of a data processing server of the system of Fig. 1; Fig. 3: a flowchart showing the steps of a method for detecting advertisements; Fig. 4: an example of a process for generating a database of audio signatures; Fig. 5, Fig. 6 and Fig. 7: an example of a process for determining whether a portion of a multimedia asset contains advertising; Fig. 8, Fig. 9, Fig. 10 and Fig. 11: an example of a process for identifying the beginning of an advertisement in a multimedia asset; Fig. 12: a flowchart showing steps of a method for identifying the beginning of an advertisement in a multimedia asset. DETAILED DESCRIPTION

[0010] For simplicity and illustration, the principles of the embodiments will be described primarily with reference to examples thereof. The following description contains numerous details in order to provide a thorough understanding of the embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without limitation to these specific details. In some instances, well-known methods and structures have not been described in detail in order not to unnecessarily obscure the embodiments.

[0011] The example systems and example methods described below detect the position of advertisements in multimedia assets, such as television programs. These examples offer advantages over conventional systems by requiring less, and in some cases, negligible or no human involvement to identify and catalog advertisements. Furthermore, these examples offer advantages over conventional systems by not requiring the use of markers (such as frames corresponding to a bumper segment or audio tones or signals) to identify the position or beginning or end of advertisements in multimedia assets.

[0012] Compared to conventional systems, these examples provide greater accuracy in detecting advertisements. By identifying the position of advertisements within multimedia assets, the example systems and methods herein can be used to enable automatic removal or replacement of advertisements during subsequent viewing of stored multimedia assets.

[0013] Fig. 1 shows one embodiment of a system 100 for detecting advertisements. The system 100 may be part of a communications network that may be used as a broadcast network, ATSC, IPTV, or IP video delivery network that transports or delivers files and multimedia content, including multicast. The network environment may include an Edge Serving Office (ESO) 110, which may be a head-end or central office of a multiple service provider (MSO), such as a cable, satellite, or telephone company. The ESO 110 may include various communications devices, such as one or more modulation / demodulation devices (not shown), a content server (not shown), and other communications devices (not shown) that provide video, data, and / or voice services to a user.

[0014] The communication devices on the ESO 110 are suitable for communication with a home gateway 120 using a cable network connection 112. In one embodiment, the home gateway 120 includes the function of a set-top box (STB). The combination of the STB and the home gateway may be referred to as a set-top gateway. However, an STB that may be separate from the home gateway is also contemplated. In this case, the home gateway provides the network connection, and the STB provides the media functions.

[0015] If the STB is provided separately from the home gateway 120, the STB may be connected to the network connection 112 in parallel with the home gateway or may be connected to the home gateway 120 to receive IPTV packets from the ESO 110 via the home gateway.

[0016] The communication devices on the ESO 110 are operable to communicate with one or more user devices via the home gateway 120. The user devices may be customer premises equipment (CPE) such as a device server with direct-attached storage (NAS) 122, a digital television receiver (DTV) 124, a radio receiver 126, a digital video disc player (DVD player) 128 with video streaming capability, a mobile device 130, a computer 132, a thermostat 134, or another Internet of Things (IoT) device (e.g., an appliance, a security camera, a lighting fixture, etc.) connected to the Internet via the home gateway. These CPE devices may be located on or near a user's premises.In the example system, the CPE device 130 is a mobile device that has wireless telecommunications capability, although it is referred to as a "customer premises" device. Fig. 1, only certain CPE devices 122, 124, 126, 128, 130, 132 and 134 are shown for illustration purposes, although more or fewer devices may be connected to the ESO 110 via the Home Gateway 120.

[0017] It will continue to Fig. 1. The communication devices at the ESO 110 can communicate with one or more of the CPE devices 122, 124, 126, 128, 130, 132, and 134 over a transport network. Examples of an ESO-to-premises transport network include one or more hybrid fiber coaxial networks (HFC networks) and / or RF over Glass networks (RFoG networks). An example HFC network can utilize a combination of fiber optic and coaxial cable 112 as the network connection 112 to send data to the communication devices at the ESO 110 and to receive data from the communication devices at the ESO 110. One or more RFoG networks can be deployed with existing HFC networks. RFoG networks typically include an all-fiber service from the ESO 110 to a Field Node or the Home Gateway 120, which is typically located in or near the user's premises.A coaxial cable can be used to connect the optical network units (ONUs) of an RFoG network to one or more user devices 122, 124, 126, 128, 130, 132, and 134. In addition, any wired or wireless network can be used, including Passive Optical Networks (PON), Gigabit Passive Optical Networks (GPON), Digital Subscriber Line (DSL), Wi-MAX, or Ethernet.

[0018] In this example, the NAS device 122, the DTV receiver 124, and the wireless receiver 126 may be connected to the home gateway 120 via a physical connection. The mobile device 130 may be connected to the gateway 120 via a short-range radio frequency (RF) connection, a magnetic, or optical connection (e.g., 802.11 WiFi, Bluetooth, Zigbee Radio Frequency for Consumer Electronics (RF4CE), Near Field Communication (NFC), or infrared (IR)), and the computer 132, the thermostat 134, and the DVD player 128 may be connected to the home gateway 120 via a wired connection such as a Universal Serial Bus (USB) cable, a FireWire cable, an Ethernet connection, and / or via a short-range RF connection, a magnetic, or optical wireless connection. Each of these connections may be considered a separate communication channel.

[0019] The ESO 110 includes a network gateway 114, which provides access between a network and a data processing server 116 and optionally to other networks such as the Internet, and a telecommunications gateway 118, which provides telecommunications access to CPE devices such as the mobile device 130, which is also connected to the home gateway 120 via the short-range RF link, the magnetic link, or the optical link. The ESO 110 may include data sources (not shown) that deliver content to the CPE devices via a standard cable TV connection or as an IPTV or IP video delivery network. These data sources may, for example, be servers connected to the network gateway 114 in the same manner as the data processing servers 116.

[0020] An example of a data processing server 116 is shown in Fig. 2 and is explained below. Generally, a data processing server 116 includes a receiver 150, a computer memory 170, and a processor 190. Further details concerning the server 116 are provided below.

[0021] The receiver 150 accepts input of multimedia content for the server 116. The receiver 150 is connected to receive the audio and video data from the multimedia stream from the network gateway 114. While a multimedia asset is being streamed (e.g., during a broadcast), the audio and video portions of the asset may be captured by the receiver 150 for processing by the server 116. The receiver 150 may acquire or capture certain additional data, such as program data for the multimedia asset. There are no restrictions on the form of the receiver 150. The receiver 150 may have a physical connection and may receive the multimedia stream from the network gateway 114 via wired or fiber optic communication. Alternatively, the receiver 150 may be a wireless receiver that receives the multimedia stream via wireless communication.The receiver 150 may include one or more tuners that enable tuning of the multimedia stream to different broadcast channels to receive multimedia content from a variety of different programs.

[0022] The memory 170 stores data for use in detecting advertisements. In one example, the memory 170 includes a database of audio signatures from the multimedia content of interest. As a multimedia asset is received by the receiver 150, the audio portion of the asset may be divided into time segments of a predetermined length. Audio signatures may be generated for each time segment of the multimedia asset, either as the asset is streamed or some time thereafter. These audio signatures may be stored in the database in the memory 170 for use in identifying advertisements in other multimedia assets. The database may be sufficiently large so that the storage size and storage duration are essentially unlimited (e.g., a petabyte database), or may be a rolling database of audio signatures. In one example, audio signatures are generated for each media asset (e.g.,All television programs on all channels are stored for ten days after their original streaming. Example methods for generating audio signatures are described in more detail below. Computer memories suitable for use as memory 170 include, but are not limited to, one or more of Random Access Memory (RAM), EEPROM, magnetic media, optical media, etc.

[0023] Processor 190 is in communication with receiver 150 and memory 170. Processor 190 is programmed to determine whether a multimedia asset acquired by receiver 150 contains an advertisement. This determination is made using audio data acquired by receiver 150 and audio signatures stored in memory 170. Processor 190 is not limited to the form of a single processing component, but may include a combination of hardware and software modules to perform the following operations. Similarly, processor 190 need not refer to processing elements in a single location, but may refer to multiple separate processing elements connected by one or more network connections (such as in the distributed system described below).

[0024] In the example shown in the Fig. 1 and Fig. 2, the hardware components associated with commercial detection are provided at the headend (cable company) of the system 100, i.e., in the ESO 110, and specifically in the data processing server 116. However, it should be understood that the same components could be provided at the subscriber end, e.g., in the home gateway 120 and / or an STB. Furthermore, the hardware components for detecting or recognizing commercials could be distributed between the headend and the subscriber end. For example, certain processing functions such as generating audio signatures could be performed at the home gateway 102 and / or the STB, and other processing functions such as storing and comparing the audio signatures could be performed at the ESO 110. In such an example, the network connection 112 and the network gateway 114 can facilitate the communication of audio signatures and information about advertisements between the home gateway 120 and the ESO 110.

[0025] Fig. 3 shows a method 200 for detecting advertisements. Generally, the method 200 includes obtaining an audio signature, determining a matching audio signature, comparing program data of the audio signatures, and determining the presence of an advertisement. Further details of the method 200 are provided below and are described with reference to the components of the system 100, as appropriate.

[0026] In step 210, an audio signature is obtained. The audio signature corresponds to a time segment of a multimedia asset of interest. The length of the time segment may be predetermined. In one example, during reception of a multimedia asset from receiver 150, the audio portion of the asset is divided into time segments of a predetermined length, e.g., 10-second segments. It is understood that other time segments may also be used.

[0027] The audio signature may be a complete audio file for the audio portion of the multimedia asset during the time period, or may be a processed or compressed version of the audio portion of the multimedia asset during the time period. In one example, processor 190 creates the audio signature by applying an audio fingerprinting function to an audio file representing the audio of the multimedia asset during the time period. Suitable audio fingerprinting functions for use in generating audio signatures will be apparent from the present description and include, for example, the Echoprint Music Identification System or another function that identifies the audio segment by its frequency components.To form the audio signature, compression may be applied to limit the size of the audio signature while retaining sufficient characteristics of the underlying audio data to allow unique or near-unique identification of the audio signature. Accordingly, an example fingerprint function may perform a frequency transformation operation, for example, and without limitation, a fast Fourier transform (FFT), a discrete cosine transform (DCT), or a Hadamard transform, and then compress the result to obtain a descriptor for the sampled segments. The descriptor may be hashed to identify a position in a hash table corresponding to the descriptor.Alternatively, the descriptors can be stored in a binary tree based on descriptor features, or in another type of database that can be quickly searched. The descriptor can be stored with Electronic Program Guide (EPG) information that specifies the multimedia asset and the time slot from which the underlying audio segment was extracted.

[0028] In step 220, a match between audio signatures is determined. This is done by comparing the audio signature obtained in step 210 with a database of previously obtained audio signatures stored in memory 170. If the descriptors are scattered and stored in a hash table, other similar descriptors can be scattered in the same binary file, and all descriptors in the binary file can be treated as matching descriptors. The previously obtained audio signatures correspond to the time segments of other multimedia assets, where the time segments are the same length as the time segments of the audio signature obtained in step 210.

[0029] The database of audio signatures may contain audio signatures corresponding to all parts of a multimedia stream, including all multimedia assets received within a specific time interval, for example, within the previous ten days. In an example shown in Fig. As shown in Figure 4, the database contains audio signatures for all television programs (e.g., assets A1, A2, B1, B2, C1, C2) taken from all television channels (e.g., channels A, B, C) across all hours of the day. Such a database contains audio signatures for parts of television programs and audio signatures for advertisements positioned within television programs. These signatures can be stored in the database in conjunction with the EPG data, as explained in more detail below.

[0030] The comparison of audio signatures can be performed in real time while a multimedia asset is streaming and / or while audio signatures are being obtained, or the comparison can be performed against previously obtained audio signatures stored in the database. To identify all matches, the comparison can be performed between the obtained audio signatures and any (other) stored audio signature in the database. A match in audio signatures can be determined when the time and frequency peaks of one audio signature are sufficiently similar or identical to the time and frequency peaks of other audio signatures.

[0031] In step 230, the program data of the matching audio signatures may be compared. In one example, the receiver 150 receives program data (EPG data) of the multimedia asset in addition to the audio and video data of the multimedia asset. In one implementation, this program data is stored in memory 170 in association with audio signatures corresponding to the corresponding audio segment extracted from the multimedia asset. The processor 190 may then compare the program data of the matching audio signatures to determine differences.

[0032] The program data may include metadata or descriptive data of the asset. Categories of information that may be included in the program data received by receiver 150 include, but are not limited to, the genre of the asset, the title of the asset, the episode title for the asset, a description of the asset, a channel or service on or through which the asset is broadcast or streamed, and a time (including date) at which the asset is broadcast or streamed.

[0033] In step 240, it is determined whether the multimedia asset contains an advertisement. By comparing the program data of assets with matching audio signatures, differences in the program data can be identified. In one example, processor 190 determines, based on the differences between the program data of the obtained audio signature and matching audio signatures, whether the time period to which the obtained audio signature corresponds contains an advertisement. Specific examples of this determination are explained in more detail below.

[0034] To give an example, it is understood that the same advertisement may be streamed in multiple different multimedia assets, such as the same commercial being shown during different television programs. Therefore, if matching audio signatures appear in several multimedia assets with different titles, it can be determined that the obtained audio signature corresponds to a time period containing an advertisement.

[0035] To give another example, it is understood that the same advertisement may be streamed on different channels. Therefore, if matching audio signatures occur in multimedia assets streamed on different channels, it can be determined that the obtained audio signature corresponds to a time period containing an advertisement. Such a determination can also be limited to assets with different titles or belonging to different genres to avoid confusion in situations where the same television program or segment is broadcast on different channels (such as during television newscasts).

[0036] Advertisements appearing in multimedia assets are generally limited in length, for example, to thirty or sixty seconds. Given this limitation, multiple consecutive audio signatures from respective multimedia assets can be compared as a group to determine whether they contain an advertisement. For example, a plurality of audio signatures corresponding to consecutive time periods of a multimedia asset can be obtained in the above manner. Matches can then be found between the series of consecutive audio signatures and stored audio signatures. The determination of an existing advertisement can be made based on the length of time covered by matching audio signatures.If the series of matching audio signatures is less than a predetermined number, it can be determined that the consecutive time periods to which the matching audio signatures correspond contain an advertisement. Conversely, if the series of matching audio signatures is greater than a predetermined number, it can be determined that the consecutive time periods to which the matching audio signatures correspond do not contain an advertisement, but may instead correspond to the same program broadcast from multiple sources. The predetermined number of audio signatures can be chosen based on the length of time covered by the audio signatures and based on the length of a typical advertisement.For example, for audio segments corresponding to a 10-second interval, the specified number may be three or six, with the length of the commercials corresponding to the typical length of thirty or sixty seconds.

[0037] While method 200 can be applied to identify the position of an advertisement within a multimedia asset, this method does not necessarily provide information regarding the position at which the advertisement begins or ends. Therefore, it may be appropriate to perform further steps to identify the beginning and end of an advertisement. Examples of such identification are explained below.

[0038] To identify the beginning and end of commercials, video data may be used in addition to audio data. Accordingly, in addition to capturing audio data from a multimedia asset, receiver 150 may also capture video data from the multimedia asset. In one implementation, the video data includes a plurality of images (or frames). This video data may be processed and stored in a similar manner to the audio data. The actual images comprising the video may be stored, or the images may be processed and compressed before storage. In one example, processor 190 processes the images to create a file of the spatial distribution of colors in the image (color layout breakdown), which is then stored.Such processing may be chosen to limit the size of the stored video data while retaining sufficient characteristics of the underlying video data to allow unique or near-unique identification of the video data. The processed images may be stored in a database within memory 170 (similar to the audio signatures). Storage in a separate memory is also possible. As with the audio signatures, the video data may be stored in association with program data of the multimedia asset from which the video data was acquired.

[0039] In another implementation, the system can capture image segments representing blank frames. These blank frames are typically broadcast before and after each advertisement. The system can store a time index of the blank frames for each multimedia asset. This time index can then be correlated with the time data stored with the audio signatures to delimit the matching segments by two groups of blank frames. However, the groups of blank frames delimiting the advertisements are preferably limited to those separated by a time period equal to the length of an advertisement, since advertisements can contain internal blank frames.

[0040] If it is determined that a time period of the multimedia asset contains an advertisement, the video data for the time periods of the matching audio signatures are aligned. In other words, a series of processed images (or frames) of the video data for the time period of the obtained audio signature are placed in a sequence next to a series of processed images (or frames) of the video data for the time periods of matching audio signatures.

[0041] After aligning the video data, one simply needs to go back to identify the beginning of the advertisement. For example, video data for time periods preceding the matching audio signatures are compared. Once the beginning of an advertisement is reached, the video and / or audio data begin to differ, which is due to the differences in the multimedia assets in which the advertisement is positioned. The beginning of the advertisement can thus be identified when the video and / or audio data in the time period preceding one of the matching audio signatures begins to differ sufficiently from the video and / or audio data in the time period preceding another of the matching audio signatures. The end of the advertisement can be identified by essentially applying the same procedure.It may be useful to compare uncompressed image data from two multimedia assets, since the same frame may be compressed differently in two data streams.

[0042] The Fig. Figures 5 to 7 show an example of determining the position of advertisements in a multimedia stream. As in Fig. 5 shows a program scheme for channel 1, channel 2, channel 3 and channel 4. In each channel, different assets are shown. In the Fig. 5, an audio signature is obtained for a 10-second time period of a multimedia asset on channel 2. A matching audio signature is found in a repetition of the asset (such a repetition is identified or detected, for example, using program data obtained from an Electronic Program Guide (“EPG”)). As expected, the audio signatures from within a program segment match corresponding audio signatures in repetitions of the same program on the same channel. However, a single matching audio signature may not be sufficient to identify a matching advertisement or program. An advertisement may be identified based on audio signatures covering less than the length of an advertisement (e.g., 30, 60, or 90 seconds) if a smaller number of segments are found to match (e.g.,segments covering 20 seconds) and that the EPG information associated with the segments is different.

[0043] Fig. Figure 6 shows how program segments can be determined based on the appearance pattern of an audio signature. Fig. 6 shows the same program scheme for channel 1, channel 2, channel 3 and channel 4 with different assets in each channel. In the example shown in Fig. As shown in Figure 6, five audio signatures are obtained for consecutive 10-second time periods of a multimedia asset on channel 2. A matching series of audio signatures is found in a repetition of the asset. As expected, consecutive signatures from within a program segment match corresponding consecutive signatures within matching assets. It can therefore be determined that the audio signatures do not contain advertisements. This determination can be made, for example, by comparing the EPG data of the multimedia assets, or by the number of consecutive matching segments being greater than one advertisement time, for example, 60 or 90 seconds.

[0044] Fig. Figure 7 shows how advertising segments can be determined based on the appearance pattern of a signature. Fig. 7 shows the same program scheme for channel 1, channel 2, channel 3 and channel 4 with different assets in each channel. In the example shown in Fig. As shown in Figure 5, an audio signature is obtained for a 10-second time period of a multimedia asset on channel 2. Matching audio signatures are found in multiple assets on channels 1, 3, and 4 that have different EPG descriptions than the asset on channel 2. Consequently, it can be determined that the audio signature contains an advertisement because it matches signatures in other channels in different assets. Furthermore, if a few consecutive audio signature pairs are present, it can be determined that the audio signature contains an advertisement due to the limitation on the length of the advertisement.

[0045] Based on the examples given in the Fig. 5 to 7, an audio signature may be classified as containing a segment of a television program if signature matches occur only in assets with the same title, i.e., assets corresponding to the same program, or if signature matches occur across many consecutive signatures. Conversely, an audio signature may be classified as containing an advertisement if signature matches occur across different assets with different titles or across different channels and / or if signature matches occur across a smaller number of consecutive signatures. It is understood that these examples of distinguishing a television program segment from an advertisement are for illustrative purposes only and are not limiting.On the contrary, the distinction between a television program segment and an advertisement can also be based on other comparisons between program data, as can be inferred from the present description.

[0046] The Fig. Figures 8 to 11 show an example method for identifying the beginning of an advertisement in a multimedia asset. In this example, it has already been determined that the audio signature of a media asset contains an advertisement. For this purpose, one of the above methods was applied. If the time period of the multimedia asset has been classified as containing an advertisement, video frame alignment is performed to align the frames of the segment of interest with the video frames of the segment of the asset whose signature has been matched. This alignment is followed by comparing consecutive frames before the segments with each other until a low level of similarity is determined. Similarly, consecutive frames after the segments are compared until a low level of similarity is determined.If low similarity is detected, it is assumed that the last matching groups of video frames on either side of the segment of interest constitute the beginning and end of the advertisement.

[0047] As in Fig. As shown in Figure 8, to determine whether a segment is classified as an "ad," an audio segment is compared with an audio segment of one of the matching assets. "Ad-C" represents a 10-second audio snippet with an audio signature of "XYZ." In the example shown in Fig. 8, a match is found with a signature “XYZ” in a database of audio signatures corresponding to a previously analyzed multimedia asset.

[0048] How Fig. As shown in Figure 9, a group of video frames corresponding to the 10-second audio snippet is compared with a group of video frames of the matching audio segment. Both groups of video frames are aligned. In one example, the groups of video frames are aligned by adding time shifts until a maximum similarity metric is found.

[0049] As in Fig. As shown in Figure 10, once alignment has been achieved in the groups of video frames corresponding to the matching audio signatures, the surrounding video frames are compared. In one example, a similarity is calculated between frames prior to the aligned group of video frames. Video frames in the preceding time period are compared by going back in time until the similarity of the video frames is low. At this point, the last group of similar frames is identified, or labeled, as the first frame of the advertisement.

[0050] As in Fig. As shown in Figure 11, similarity between frames is calculated based on the aligned groups of video frames. Video frames in the subsequent time period are compared by moving forward in time until the similarity of the video frames is low. At this point, the last group of similar frames is identified or labeled as the last frame of the advertisement.

[0051] Fig. 12 shows a flowchart illustrating an example method 300 for identifying the beginning of an advertisement in a multimedia asset. Generally, the method 300 includes determining an existing advertisement, aligning the video frames of the advertisement, calculating the similarity of preceding video frames, and identifying the beginning of the advertisement. Further details of the method 300 are described below.

[0052] In step 310, it is determined whether an audio signature corresponds to a duration of a multimedia asset containing an advertisement. Such a determination may be made by comparing the audio signature to one or more matching audio signatures of other multimedia assets, applying one of the above examples provided in the application.

[0053] In step 320, video frames of the audio signature of interest and the one or more matching audio signatures are aligned. The video frames may be aligned in an uncompressed (i.e., native) form or as compressed frames. In an example shown in Fig.12, the video frames may have been compressed prior to alignment. Depending on the compression techniques used, it may be desirable to decompress compressed video frames prior to alignment. To align the video frames, chrominance information, and in particular color layout information (e.g., information about the position of colors or color spectra in the frame), of a sequence of video frames associated with the audio signature of interest is compared with corresponding information from a sequence of frames corresponding to the one or more matching audio signals. It may be advantageous to compare only the chrominance information because there is typically less chrominance information than luminance information in a frame, especially when the frame has a 4:2:0 format.In one implementation, the chrominance information is further subsampled to provide the color layout information. One of the sequences of video frames may be time-shifted until the color layout information of the sequence of video frames associated with the audio signature of interest matches the corresponding information of the sequence of video frames from the one or more matching audio signals.

[0054] In step 330, preceding video frames are checked. Once the video frames of the advertisement are aligned, it can be assumed that the sequences of video frames will continue to match as long as the content remains identical, or in other words, as long as both sequences correspond to the same advertisement. Thus, to identify the beginning of the advertisement, video frames preceding the sequence of video frames corresponding to the matching audio signatures are compared, and a similarity or dissimilarity is calculated. Depending on the compression techniques used, this comparison can be performed on compressed video frames or can be performed on uncompressed video frames to avoid inaccuracies resulting from the same video frame being compressed differently in two different data streams.The similarity or dissimilarity can be calculated based on differences in the color layout information of the respective sequence of video frames.

[0055] In step 340, the beginning of the advertisement is identified. If the similarity of the sequences of video frames falls below a predetermined level, or conversely, if the dissimilarity exceeds a predetermined level, it can be determined that the content in the respective sequences of video frames is no longer sufficiently similar. Based on this, it can be further determined that the beginning of the advertisement has been identified or found. By identifying the first pair of similar frames, the beginning of the advertisement can be identified.

[0056] While method 300 refers to identifying the beginning of a commercial, it should be understood that virtually the same steps can be applied to identify the end of a commercial. Steps 310 and 320 may remain the same, and in step 330, video frames following the sequence of video frames corresponding to the matching audio signatures may be compared.

[0057] Once an advertisement has been detected, as described above, metadata may be generated and stored in association with the audio signature of the advertisement identified as containing an advertisement. Such metadata may be useful and assistive, for example, in identifying repetitions of the advertisement in other multimedia assets or in locating related advertisements in future multimedia assets. In one example, audio, video, or text data from portions of the media asset (e.g., the television program) surrounding the particular advertisement may be captured and stored to provide metadata for the advertisement. In another example, closed captions of the segment detected as an advertisement may be captured and stored to provide metadata for the advertisement.In yet another example, other keywords, including the advertiser's name, can be extracted from the advertisement and used as metadata for the advertisement.

[0058] The foregoing examples have been described in the context of particular devices, apparatus, systems, and / or methods. However, it should be understood that this description is merely exemplary and not limiting. Certain embodiments may, for example, be implemented in a non-transitory computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, system, or machine. The computer-readable storage medium contains instructions for controlling a computer system to perform a method described by the particular embodiments. When executed by one or more processors, these instructions may be operative to implement what is described in certain embodiments.

[0059] When the term "comprising" or derivatives thereof are used in the claims, they do so in a non-exclusive sense; that is, they are not intended to exclude the presence of other or further elements or steps in a claimed structure or method. When the terms "a" and "a," "an" and "the" are used in the description and the appended claims, the plural form is also included unless expressly stated otherwise. Likewise, in the description and the appended claims, "in" has the meaning, among other things, of "at," unless the context expressly dictates otherwise.

[0060] The above-described embodiments, together with examples of possible practical implementations of aspects of the present invention, are intended to illustrate the flexibility and advantages of the particular embodiments recited in the claims and should not be considered the only embodiments. As will be appreciated by those skilled in the art, other arrangements, embodiments, implementations, and equivalents may be utilized based on the present description and the appended claims without departing from the scope of the invention as defined by the appended claims. The description and figures are therefore for illustrative purposes only and do not constitute a limitation on the invention. All modifications are intended to be within the scope of the claims.To the extent that advantages, solutions to problems, or an element or elements cause an advantage, solution to problems, or element or elements to appear more pronounced, they should not be construed as critical, necessary, or essential features or elements of any or all of the claims. The invention is defined solely by the appended claims, including any amendments made during the pendency of the application and any equivalents of such claims as granted.

Claims

[1] Method comprising: obtaining an audio signature corresponding to a time period of a multimedia asset; determining a match between the received audio signature and one or more stored audio signatures, wherein the stored audio signatures correspond to time periods of a plurality of other multimedia assets; comparing program data of the multimedia assets of the obtained audio signature and the one or more matching audio signatures; and determining whether the time period of the multimedia asset contains an advertisement based on comparing the program data of the multimedia asset, the received audio signature and the one or multiple matching audio signatures. [2] The method of claim 1, wherein the step of obtaining comprises: applying an audio fingerprint function to an audio file representing audio of the multimedia asset during the time period to create the audio signature. [3] The method of claim 1, wherein the multimedia asset comprises a television program. [4] The method of claim 1, wherein the step of determining comprises: generating a database containing one or more stored audio signatures; and comparing the obtained audio signature with one or more stored audio signatures in the database to determine the match. [5] The method of claim 1, wherein the program data comprises one or more categories of data selected from a group consisting of genre data, asset title data, episode title data, asset description data, channel data, and time data. [6] The method of claim 1, wherein the step of determining comprises: determining that the time period of the multimedia asset contains an advertisement if the asset title data of the multimedia asset of the received audio signature differs from asset title data of the multimedia asset of the one or more matching audio signatures. [7] The method of claim 1, wherein the step of determining comprises: determining that the time period of the multimedia asset contains an advertisement if channel data of the multimedia asset of the obtained audio signature differs from channel data of the multimedia asset of the one or more matching audio signatures. [8] The method of claim 1, further comprising: aligning video data for the time period of the obtained audio signature with video data for the time periods of the one or more matching audio signatures; comparing video data for a time period preceding the time period of the obtained audio signature with video data for one or more time periods preceding the respective time periods of the one or more matching audio signatures; and identifying a start of the advertisement if the video data for the time period preceding the time period of the obtained audio signature differs sufficiently from the video data for the time period preceding one or more time periods preceding the respective time periods of the one or more matching audio signatures, if it is determined that the time period of the multimedia asset contains an advertisement. [9] Method comprising: obtaining a plurality of audio signatures corresponding to successive time periods of the multimedia assets; determining matches between the received audio signatures and a plurality of stored audio signatures, wherein the stored audio signatures correspond to consecutive time periods of a plurality of other multimedia assets; and determining whether the consecutive time periods of the multimedia asset contain an advertisement based on a number of consecutive matching audio signatures of the plurality of stored audio signatures. [10] The method of claim 9, wherein the step of obtaining comprises: applying an audio fingerprinting function to consecutive segments of an audio file representing audio of the multimedia asset during consecutive time periods to create the multiple audio signatures. [11] The method of claim 9, wherein the multimedia asset comprises a television program. [12] The method of claim 9, wherein the step of determining comprises: generating a database containing the multiple stored audio files; and Comparing the obtained audio signatures with the multiple stored audio signatures in the database to determine matches. [13] The method of claim 9, wherein the step of determining comprises: determining that the consecutive time periods of the multimedia asset contain an advertisement if a number of the plural matching audio signatures is greater than a first predetermined number and less than a second predetermined number. [14] The method of claim 9, further comprising: aligning video data for the time period of the obtained audio signatures with video data for the time periods of the one or more matching audio signatures; comparing video data for a time period preceding the time period of the obtained audio signature with video data for one or more time periods preceding respective time periods of the one or more matching audio signatures; and identifying a start of the advertisement if the video data for the time period preceding the time period of the obtained audio signature differs sufficiently from the video data for the time period preceding one or more time periods preceding the respective time periods of the one or more matching audio signatures, if it is determined that the time period of the multimedia asset contains an advertisement. [15] System comprising: a receiver (150) connected to receive audio and video data from a multimedia stream; a computer memory (170) storing a database of one or more a plurality of stored audio signatures, wherein the stored audio signatures correspond to time periods of a plurality of multimedia assets; and a processor (190) in communication with the receiver (150) and the computer memory (170), the processor (190) being programmed to: obtaining an audio signature corresponding to a time segment of a multimedia asset in the multimedia stream; determining a match between the received audio signature and the one or more stored audio signatures; comparing program data of the multimedia assets of the obtained audio signature and the one or more matching audio signatures; and determining whether the time period of the multimedia asset contains an advertisement based on the comparison of the program data of the multimedia assets of the received audio signature and the one or multiple matching audio signatures. [16] The system of claim 15, wherein the processor (190) is programmed to apply an audio fingerprinting function to audio data of the multimedia asset during the time period to create the audio signature. [17] The system of claim 15, wherein the multimedia asset comprises a television program. [18] The system of claim 15, wherein the program data comprises one or more categories of data selected from a group consisting of genre data, asset title data, episode title data, asset description data, channel data, and time data. [19] The system of claim 15, wherein the processor (190) is programmed to determine that the time period of the multimedia asset contains an advertisement if the asset title data of the multimedia asset of the obtained audio signature differs from asset title data of the multimedia asset of the one or more matching audio signatures. [20] The system of claim 15, wherein the processor (190) is programmed to determine that the time period of the multimedia asset contains an advertisement if the channel data of the multimedia asset of the obtained audio signature differs from channel data of the multimedia asset of the one or more matching audio signatures. [21] The system of claim 15, wherein the processor (190) is further programmed to: align the video data for the time period of the obtained audio signature with video data for the time periods of the one or more matching audio signatures; the video data for a time period preceding the time period of the received video signature with video data for one or several periods of time, the respective periods of one or preceding several matching audio signatures; and to identify a start of the advertisement if the video data for the time period preceding the time period of the obtained audio signature differs sufficiently from the video data for the time period preceding one or more time periods preceding the respective time periods of the one or more matching audio signatures, when the processor (190) determines that the time period of the multimedia asset contains an advertisement. [22] Non-transitory computer-readable medium containing computer instructions that cause a computer: to obtain an audio signature corresponding to a time period of a multimedia asset; determine a match between the obtained audio signature and one or more stored audio signatures, wherein the stored audio signatures correspond to time periods of a plurality of other multimedia assets; compare program data of the multimedia assets of the received audio signature and the one or more matching audio signatures; and determine whether the time period of the multimedia asset contains an advertisement based on comparing the program data of the multimedia assets, the received audio signature and the one or more matching audio signatures.

Images