Real-time TV commercial detection and replacement for smart TVS and internet connected entertainment devices

Abstract

A system and method for real-time detection and replacement of advertisements. Content currently being displayed on an associated device is analyzed to determine whether a commercial is playing. When a commercial is detected, a targeted advertisement is overlaid on the content, displayed in place of the commercial. Once the original commercial ends, the overlay is discontinued, and the original program content is displayed. One or more artificial intelligence components are utilized to identify commercials, non-commercial content, particular products or services, targeted advertisements for overlaying, and the like.

Description

REAL-TIME TV COMMERCIAL DETECTION AND REPLACEMENT FOR SMART TVS AND INTERNET CONNECTED ENTERTAINMENT DEVICESCROSS-REFERENCE TO RELATED APPLICATIONSThis application claims the benefit of U.S. Provisional Patent Application No. 63 / 619788 filled on January 11 , 2024, which is incorporated by reference herein in its entirety.TECHNICAL FIELD

[0001] The present disclosure relates to the field of advertising, and in particular to advertisements in real-time video transmissions.BACKGROUND

[0002] Internet connected entertainment devices and smart televisions, may receive media content from a variety of different sources, including both broadcast and streaming data sources. Streaming services may refer to any type of service that provides media over a data network, such as the Internet, including, for example and without limitation, movies, television, short videos, news, etc. A user may access and view these various media types on smart televisions, smart phones, tablets, computers, video game systems, and the like.

[0003] Some streaming services insert advertisements such as commercials for products or services into the media content. In some cases, this advertising content is not germane to the underlying entertainment content, inapplicable to the particular user, distracting or offensive, untargeted, geographically irrelevant, or the like. Accordingly, systems and methods for detection of advertisements and targeted replacement thereof is needed.SUMMARY

[0004] One aspect provides a method for real-time detection and replacement of advertisements. The method includes receiving, from an associated user device, data corresponding to a current content stream displayed on a display of the associated user device, and determining, by at least one artificial intelligence component, that an advertisement is present in the current content stream. The method further includes overlaying a targeted advertisement in response to the determination that the advertisement is present, wherein the targeted advertisement replaces theadvertisement in the current content stream, and estimating, by the at least one artificial intelligence component, a duration of the advertisement in the current content stream, which can be continuously updated as more of the current content stream is analyzed. In addition, the method includes stopping the overlaying of the targeted advertisement in response to the estimated duration or if determined, by at least one artificial intelligence component, that an advertisement has finished before the estimated duration, wherein the display of the associated user device returns to displaying of the current content being streamed.

[0005] Other aspects provide: one or more apparatuses operable, configured, or otherwise adapted to perform any portion of any method described herein (e.g., such that performance may be by only one apparatus or in a distributed fashion across multiple apparatuses); one or more non-transitory, computer-readable media comprising instructions that, when executed by one or more processors of one or more apparatuses, cause the one or more apparatuses to perform any portion of any method described herein (e.g., such that instructions may be included in only one computer-readable medium or in a distributed fashion across multiple computer- readable media, such that instructions may be executed by only one processor or by multiple processors in a distributed fashion, such that each apparatus of the one or more apparatuses may include one processor or multiple processors, and / or such that performance may be by only one apparatus or in a distributed fashion across multiple apparatuses); one or more computer program products embodied on one or more computer-readable storage media comprising code for performing any portion of any method described herein (e.g., such that code may be stored in only one computer- readable medium or across computer-readable media in a distributed fashion); and / or one or more apparatuses comprising one or more means for performing any portion of any method described herein (e.g., such that performance would be by only one apparatus or by multiple apparatuses in a distributed fashion). By way of example, an apparatus may comprise a processing system, a device with a processing system, or processing systems cooperating over one or more networks.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying figures and Appendices. It is noted that, in accordance with the standard practice in the industry, various figures arenot drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or reduced for clarity of discussion.

[0007] Fig. 1 illustrates an example system diagram for real-time detection and replacement of advertisements in accordance with some embodiments.

[0008] Fig. 2 is an illustrative block diagram of an example server for use in a system for real-time detection and replacement of advertisements in accordance with some embodiments.

[0009] Fig. 3 is an illustrative block diagram of an example user device for use in a system for real-time detection and replacement of advertisements in accordance with some embodiments.

[0010] Fig. 4 is an example method for real-time detection and replacement of advertisements.

[0011] Figs. 5A, 5B, and 5C are example methods of real-time detection and replacement of advertisements.

[0012] Figs. 6A, 6B, 6C, 6D, and 6E illustrate an example method for real-time detection and replacement of advertisements.

[0013] Figs. 7A, 7B, 7C, 7D, 7E, 7F, 7G, 7H, 7I, 7J, 7K, 7L, 7M, and 7N illustrate an example method for real-time detection and replacement of advertisements.DETAILED DESCRIPTION

[0014] The following disclosure provides many different embodiments, or examples, for implementing different features of the provided subject matter. Specific examples of components and arrangements are described below to simplify the present disclosure. These are, of course, merely examples and are not intended to be limiting. For example, the formation of a first feature over or on a second feature in the description that follows may include embodiments in which the first and second features are formed in direct contact and may also include embodiments in which additional features may be formed between the first and second features, such that the first and second features may not be in direct contact. In addition, the present disclosure may repeat reference numerals and / or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and / or configurations discussed.

[0015] Further, spatially relative terms, such as “beneath,” “below,” “lower,” “above,” “upper” and the like, may be used herein for ease of description to describe oneelement or feature's relationship to another element(s) or feature(s) as illustrated in the figures. The spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. The apparatus may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein may likewise be interpreted accordingly.

[0016] Numerical values in the specification and claims of this application should be understood to include numerical values which are the same when reduced to the same number of significant figures and numerical values which differ from the stated value by less than the experimental error of conventional measurement technique of the type described in the present application to determine the value. All ranges disclosed herein are inclusive of the recited endpoint.

[0017] The term “about” can be used to include any numerical value that can vary without changing the basic function of that value. When used with a range, “about” also discloses the range defined by the absolute values of the two endpoints, e.g. “about 2 to about 4” also discloses the range “from 2 to 4.” The term “about” may refer to plus or minus 10% of the indicated number.

[0018] In some embodiments, a method is provided that may be implemented on a smart television or other Internet connected entertainment device, e.g., internal smart television circuit boards, cable decoders, digital video recorder (DVR) components, satellite receivers, FireSticks®, Roku® devices, Apple® TV devices, Google® Chromecast® devices, and the like. In such embodiments, the method provides for the analysis of content currently being received through the currently viewed HDMI / USB / Wireless, etc., source, internally generated by the device, or the like. In some embodiments, the content may be received via a streaming service, broadcast (e.g., cable or over-the-air), or other such origination. As used herein, the streaming service, broadcaster, cable provider, etc., are commonly referenced as the “content provider”, “source”, “content source” or other similar encompassing term intended to convey the entity and / or equipment from which the video / audio is received. Variations on the source are also contemplated herein, and the examples provided herein are intended only as non-limiting examples thereof. The analysis may include the identification of when a TV commercial or other advertisement appears (or will appear) on the viewing screen or display, e.g., identifying or extracting one or more fingerprints associated with the audio and / or video of the source. An overlay of a new, highlytargeted advertisement on the TV commercial or advertisement may then be generated in real-time, with no user input. Upon completion of the original TV commercial or advertisement, the method provides for the cessation of the overlay, and a return to the previous source, i.e., the previously running television program, movie, video, etc. Stated another way, the method provides for swapping some or all the advertisements that a viewer would normally see on the TV screen or display with highly targeted advertisements in real-time. In some embodiments contemplated herein, the user may receive compensation, possibly in the form of points that can be redeemed for gift cards, other promotional offers, or other forms of compensation in exchange for viewing replacement advertisements.

[0019] In some embodiments disclosed herein, automated content recognition may be used to identify advertisements (e.g., commercials, infomercials, program interruptions) and programs (e.g., television shows, videos, movies, news broadcasts, sporting broadcasts, i.e., non-advertising media content). Such recognition may be based on fingerprinting, watermarking or other identifying technology included in the media content for identification of content being viewed on a user device. Further, such recognition may be based on the video component, the audio component, or both, the channel being viewed, or other sources of information attainable from the device(s) being used to obtain the media, e.g., the make and model of viewing device (smart television brand), etc. Fingerprinting, for example, may be implemented on a user device, such as a smart television, that analyzes individual frames or the audio spectrum in a current video stream source being displayed thereon to distill down aspects of the frame or clip that may be used to distinctly identify the content, i.e., a compact digital representation of a video that summarizes the unique characteristics of the material.

[0020] With reference to Fig. 1 , there is shown an exemplary system 100 for realtime detection and replacement of advertisements in accordance with one embodiment. It will be appreciated by the skilled artisan that the various components depicted in Fig. 1 are for purposes of illustrating aspects of the exemplary embodiment, and that other similar components, implemented via hardware, software, or a combination thereof, are capable of being substituted therein. It will be appreciated that the system 100 is capable of implementation using a distributed computing environment, such as a computer network, which is representative of any distributed communications system capable of enabling the exchange of data betweentwo or more electronic devices. It will be further appreciated that such a computer network includes, for example and without limitation, a virtual local area network, a wide area network, a personal area network, a local area network, the Internet, an intranet, or the any suitable combination thereof. Accordingly, such a computer network comprises physical layers and transport layers, as illustrated by various conventional data transport mechanisms, such as, for example and without limitation, Token-Ring, Ethernet, or other wireless or wire-based data communication mechanisms.

[0021] As illustrated in Fig. 1 , the system 100 includes at least one server 102, which is capable of implementing the exemplary method described below (at least one server 102 can be a separate device and / or within the same smart TV (i.e, locally). The exemplary server 102, is described in greater detail below with respect to Fig. 2. In some embodiments, the server 102 may be in communication with a distributed network 101 , such as, for example and without limitation, the Internet. The server 102 may use any suitable communications link 112 for sending and receiving data or signals to the network 101 , including, for example and without limitation wireless communications, such as Bluetooth, WiMax, 802.11 a, 802.11b, 802.11g, 802.11(x), 3G, 4G, 5G cellular communication systems, a proprietary communications network, infrared, optical, the public switched telephone network, or any suitable wireless data transmission system, or wired communications.

[0022] The system 100 further includes at least one user device 104, illustrated in Fig. 1 as a smart television. It will be appreciated that the use of the smart television as the user device 104 is intended solely as one example of a suitable user device, and other examples include, without limitation, smart phones, tablets, laptops, desktop computers, streaming service devices coupled to a display device, and the like. Accordingly, the subject systems and methods described herein are not intended to be limited solely to the smart television user device 104 shown in Fig. 1. Further details regarding the various components of the user device 104 are discussed below with respect to Fig. 3. It will be appreciated that the user device 104 may be in communication with the network 101 via a communications link 114. A suitable communications link 114 may include, for example and without limitation, wireless communications, such as Bluetooth, WiMax, 802.11 a, 802.11 b, 802.11g, 802.11(x), 3G, 4G, 5G cellular communication systems, a proprietary communications network,infrared, optical, the public switched telephone network, or any suitable wireless data transmission system, or wired communications.

[0023] The system 100 of Fig. 1 further includes one or more media content providers, generally represented by the servers 106, 108, and 110. Each of these providers, i.e., servers 106-110, are intended solely to illustrate different sources of media content capable of being received and / or displayed on the user device 104, as will be appreciated by the skilled artisan. In accordance with some embodiments disclosed herein, each of the media content providers 106, 108, 110 may correspond to a cable provider, a streaming service, a broadcast service, or the like. Suitable nonlimiting examples of such media content providers 106-110 may including, without limitation, NETFLIX, DISNEY+, HULU, YOUTUBE, GOOGLE, APPLE TV, MAX, PEACOCK, PARAMOUNT+, ESPN, AMAZON, NBC, ABC, FOX, CBS, CNN, local affiliates, pay services, and the like. Further, such media content providers 106-110 may correspond to subscription-based providers, broadcast-base providers, news, television, or the like, which may or may not impose advertisements during content presentation. The servers 106-110 may include a computer server, workstation, personal computer, cellular telephone, tablet computer, pager, combination thereof, or other computing device capable of executing instructions for performing the exemplary method.

[0024] According to one example embodiment, the servers 106-110 include hardware, software, and / or any suitable combination thereof, configured to interact with an associated user, a networked device, networked storage, remote devices, or the like. In other embodiments, each of the servers 106-110 may be representative of collaborative or cloud-based distributed computing platforms providing the aforementioned media content as will be understood by those skilled in the art. As illustrated in Fig. 1 , each of the servers 106, 108, and 110 may be in communication with the network 101 via corresponding communications links 116, 118, and 120, respectively. Suitable communication links 116, 118, and 120 may include, for example and without limitation, wireless communications, such as Bluetooth, WiMax, 802.11a, 802.11 b, 802.11g, 802.11 (x), 3G, 4G, 5G cellular communication systems, a proprietary communications network, infrared, optical, the public switched telephone network, or any suitable wireless data transmission system, orwired communications.

[0025] As illustrated in Fig. 1 , the system 100 may include one or more advertisers, shown in Fig. 1 as the advertiser server 122. In some embodiments, the advertiserserver 122 may store, send, receive, produce, obtain, etc., advertisements to the server 102 and / or to the user device 104. The server 122 may include hardware, software, and / or any suitable combination thereof, configured to interact with an associated user, a networked device, networked storage, remote devices, or the like. As shown in Fig. 1 , the server 122 may be in communication with the network 101 via a communications link 124. A suitable communications link 124 may include, for example and without limitation, wireless communications, such as Bluetooth, WiMax, 802.11a, 802.11 b, 802.11g, 802.11 (x), 3G, 4G, 5G cellular communication systems, a proprietary communications network, infrared, optical, the public switched telephone network, or any suitable wireless data transmission system, or wired communications. In some embodiments, one or more such advertiser servers 122 may be included, corresponding to different entities providing or requesting the transmission of advertisements to the user device 104, as discussed in greater detail below. In some embodiments, the advertiser servers 122 can be included in the content provider servers 106, 108, 110 or in association with the content provider servers 106, 108, 110. Furthermore, in an embodiment, the software, machine learning models, or any other information can be remotely updated at predetermined intervals, when a security update, or new feature update is available for both the server 122 and / or 102. Still further, the software and the machine learning model can be customized based off of the exact country, region, and / or city to account for any differences in common TV shows and programs as well as to account for different languages.

[0026] Turning now to Fig. 2, there is shown an exemplary diagram of the server 102 of Fig. 1 in accordance with some embodiments. As shown in Fig. 2, the server 102, which is capable of implementing the methods set forth herein, includes a processor 204, which performs the exemplary method by execution of processing instructions 206 that are stored in memory 208 connected to the processor 204, as well as controlling the overall operations of the server 102. The various components of the server 102 may be connected by a data / control bus 202. The processor 204 of the server 102 may be in communication with an associated database 238 via a suitable communications link 228. A suitable communications link 228 may include, for example, a switched telephone network, a wireless radio communications network, infrared, optical, or other suitable wired or wireless data communications. The database 238 is capable of implementation on components of the server 102, e.g., stored in local memory 208, i.e., on hard drives, virtual drives, or the like, or on remotememory accessible to the server 102. It will be appreciated that while depicted in Figs. 1 and 2 as a single device, the server 102 may be representative of a cloud-based computing, i.e., distributed processing system, and the illustration of the server 102 as a single device is intended solely as a non-limiting illustrative example.

[0027] The associated database 238 is representative of any organized collections of data (e.g., advertisements 240, training data 242, fingerprinting data 244, user / account information 250, user device information 252, content provider information 254, etc.) used for one or more purposes. The skilled artisan will appreciate that such information may be updated via machine learning during operations of the subject system 100. Implementation of the associated database 238 is capable of occurring on any mass storage device(s), for example, magnetic storage drives, a hard disk drive, optical storage devices, flash memory devices, or a suitable combination thereof. The associated database 238 may be implemented as a component of the server 102, e.g., resident in memory 208, or the like. In one embodiment, the associated database 238 may include, for example and without limitation, data corresponding to television commercials, television programs, movies, video games, sporting events, news videos, education videos, infomercials, static advertisements and other advertising and non-advertising content. In other embodiments, the database 238 may further include, for example and without limitation, information about a user associated with a user device 104, e.g., preferences, genres (TV, movies, video games, videos, etc.), contact information, usernames, account information, targeted advertising information, demographic information of the user(s), geographical location, behavioral information, click-through rates, externally shared posts, hobbies and activities, usage data, ad interactions, ad engagement metrics, survey and user feedback, IP addresses, time spent on various apps or genres of film, and the like, information about a user device 104, e.g., make, model, connectivity, display features, installed apps, etc., as well as content provider information, e.g., subscriptions, names, account information, account level (with or without advertisements), rate of point collection per advertisement (depending on how much data has been collected about the user), streaming capacities, etc., and other information related to the transmission and receipt of digital media content. In some embodiments, the database 238 may store a variety of fingerprinting data 244 used for identification of advertising and non-advertising content. As shown in Fig. 2, suchfingerprinting data 244 may include, for example and without limitation, TV program fingerprint data 246 and TV advertisement fingerprint data 248.

[0028] The server 102 may include one or more input / output (I / O) interface devices 234 and 236 for communicating with external devices. The I / O interface 236 may communicate, via communications link 226, with one or more of a display device 230, for displaying information, and a user input device 232, such as a keyboard or touch or writable screen, for inputting text, remote control with buttons, and / or a cursor control device, such as mouse, trackball, or the like, for communicating user input information and command selections to the processor 204. The I / O interface 234 may communicate with external devices such as the user device 104, the media content provider servers 106-110, the advertiser server 122, and the like, via a suitable communications link 224.

[0029] It will be appreciated that the server 102 illustrated in Fig. 2 is capable of implementation using a distributed computing environment, such as a computer network, which is representative of any distributed communications system capable of enabling the exchange of data between two or more electronic devices. It will be further appreciated that such a computer network includes, for example and without limitation, a virtual local area network, a wide area network, a personal area network, a local area network, the Internet, an intranet, or any suitable combination thereof. Accordingly, such a computer network comprises physical layers and transport layers, as illustrated by various conventional data transport mechanisms, such as, for example and without limitation, Token-Ring, Ethernet, or other wireless or wire-based data communication mechanisms. Furthermore, while depicted in Fig. 2 as a networked set of components, the server 102 is capable of implementation on a standalone device adapted to interact with the servers 106, 108, 110, 122, and the user device 104 described herein.

[0030] The server 102 may include one or more of a computer server, workstation, personal computer, cellular telephone, tablet computer, pager, combination thereof, or other computing device capable of executing instructions for performing the exemplary method. According to one example embodiment, the server 102 includes hardware, software, and / or any suitable combination thereof, configured to interact with an associated user, a networked device, networked storage, remote devices, or the like.

[0031] The memory 208 illustrated in Fig. 2 as a component of the server 102 may represent any type of non-transitory computer readable medium such as random ioaccess memory (RAM), read only memory (ROM), magnetic disk or tape, optical disk, flash memory, or holographic memory. In one embodiment, the memory 208 comprises a combination of random access memory and read only memory. In some embodiments, the processor 204 and memory 208 may be combined in a single chip. The network interface(s) 234, 236 allow the computer to communicate with other devices via a computer network 101 , (e.g., the Internet), and may comprise a modulator / demodulator (MODEM). Memory 208 may store data processed in the method as well as the instructions for performing the exemplary method.

[0032] The digital processor 204 can be variously embodied, such as by a single core processor, a dual core processor (or more generally by a multiple core processor), a digital processor and cooperating math coprocessor, a digital controller, or the like. The digital processor 204, in addition to controlling the operation of the server 102, executes instructions 206 stored in memory 208 for performing the method set forth hereinafter.

[0033] As shown in Fig. 2, the instructions 206 stored in memory 208 may include a comparison component 210 configured to receive fingerprinting data from the user device 104 and compare such fingerprinting data to stored fingerprinting data 244. In some embodiments, the server 102 has access to one or more distinct databases of fingerprinting data 244, illustrated in Fig. 2 as TV program fingerprinting data 246 and TV advertisement fingerprinting data 248 . The comparison component 210 may then output a result of the comparison with the database, i.e., corresponds to TV program data 246 or TV advertisement data 248 to a determination component 212. Stated another way, the comparison component 210 analyzes the fingerprinting data received from a user device 104 via the computer network 101 and compares this fingerprinting data to fingerprinting data 244 stored in the database 238. The results of this comparison are then communicated to the determination component 212 for further analysis.

[0034] The instructions stored in memory 208 may further include the determination component 212 that is configured to determine, in accordance with an output from the comparison component 210, whether the received fingerprint data matches that of an advertisement 248 (i.e., TV commercial or Ad) or a program 246 (i.e., non-advertising related news broadcast, television show, movie, sports broadcast, video, video game, etc.). When the determination component 212 determines that the received fingerprinting data matches that of a known (i.e., recognized or stored fingerprint) TV liprogram 248 with a specific degree of certainty and provides this information to the first Al component 214 (described in greater detail below), the server 102 may communicate to the user device 104 to not overlay any new targeted advertisements 240 and cease any current overlay targeted advertisements 240 (if applicable). However, in the event that the determination component 212 determines that the received fingerprinting data matches that of a stored TV advertisement fingerprinting data 248, the determination component 212 communicates such a determination to the first Al component 214, which may communicate such determination to the overlay generating component 220, as discussed in greater detail below.

[0035] In the event that the determination component 212 determines that the received fingerprinting data does not match any known (i.e., previously stored) fingerprinting data 244 or matches multiple stored fingerprinting data in more than one database, the instructions 206 stored in memory 208 include a first artificial intelligence (Al) component 214 that is configured to determine whether the received fingerprinting data corresponds to advertising or non-advertising content. Stated another way, when received fingerprinting data cannot be recognized in either database or recognized in multiple databases, i.e., programs 246 or advertisements 248, a first Al component 214 may be implemented to determine a likelihood of the current content being an advertisement or not. The first Al component 214 may also be used to identify the type of transition that is occurring, including but not limited to TV advert to TV advert, TV content to TV advert, TV content to TV content, or TV Advert to TV Advert. The first Al component 214 may be utilized in all situations before initiating or stopping an advertisement overlay to consider various aspects including, for example and without limitation, identifying changes in color, audio, object, motion path, text, and the like, in conjunction with the information provided by the determination component 212 to make an informed decision about when to start and stop advertising overlays. In some embodiments, the first Al component 214 may be a deep-learning neural network artificial intelligence component. In such embodiments, the first Al component 214 may be trained on training data 242 that includes fingerprinting data 244 as its input for identifying whether content is a commercial 248 or not. In the event that the first Al component 214 determines that the content is a commercial 248, the first Al component 214 may send an instruction to the overlay generating component 220 to overlay the existing content with targeted advertisements 240, as well as instructions to cease running the overlay in response to a determination that the existing content,e.g., the original commercial, has stopped early. The overlay feature may also be able to selectively overlay only certain portions of a display in the event that the advertising within the content only covers a specific portion of the display (e.g., a banner ad at the bottom of a display or a small commercial in the corner of the display). The exact section of the display to be overlayed may be determined by the first Al component 214 through server 102.

[0036] In some embodiments, the first Al component 214 may further be configured to determine whether or not input from a remote (or other input device) has not been received for a predetermined period of time, e.g., indicative of user not interacting with the user device 104 (It should be appreciated that the component 214 could also contain a non-AI such as a computer program conventional in the art that receives information on the last button press initiated by a user on a remote device). In such embodiments, the user device 104 and first Al component 214 together or individually may reduce the framerate (e.g., slow down the display) to enable additional time for processing and analysis by the first Al component 214. For example, and without limitation, the user device 104 may start to slow down the content slightly and create a small delay between the TV signal (e.g., the source) and what is displayed on the screen. Such a delay may enable the first Al component 214 and the comparison / determination components 210-212 to have more time and additional frames to decide whether or not the content is an advertisement or not. Then if the remote is used and a button is clicked for any purpose other than volume change, the content may slightly speed up to get back to its live feed while the user is using the remote to navigate the Ul.

[0037] The instructions 206 stored in memory 208 may further include a second Al component 216 configured to estimate the potential length of a commercial break and how likely it is for a commercial to appear. In some embodiments, the second Al component 216 may be implemented as a deep-learning neural network, however other types of artificial intelligence and / or machine learning may be used, and the deep-learning neural network implementation is intended solely as one possible embodiment.

[0038] In accordance with some embodiments, the second Al component 216 may be trained on training data 242 to identify different genres of TV content (e.g., live news, movies, TV shows, etc.) and based upon these categories, past history and frequency of TV commercials on this particular channel or TV content, and currentlyavailable information, including additional information including but not limited to if the screen has dimmed to black, a broadcaster logo has disappear, the average color, histogram, optical flow (the pattern of how objects movie in a video), or luminance of the content has rapidly changed, or if the audio quality, signature, volume, or spectrum has rapidly changed, sudden appearance of flashy and vibrant colors, large text overlays, large logos, and if closed captioning has changed, or other changes common with advertisements or interruptions are noticed will be used to determine if an advertisement is likely to start. When the second Al component 216 determines that an advertisement may be starting imminently, it may start to fade the TV screen (e.g., display, etc.) to black. The second Al component 216 may then allow a short segment of the TV content to be sent to the server 102 to compare against the databases 238 for the first Al component 214 to determine if it is a TV commercial or not. If it is, then new advertisements will overlay it. If not, then the screen will unfade from black. Additionally, the audio may be muted or lowered during the time that the screen is held at a black state and will also faded in and out at the same time as the screen fades in and out from black.

[0039] In some embodiments, the second Al component 216 may also be configured to generate an estimate as to the length of the TV commercial break so as to allocate potential timeslots for replacement TV commercials. In such embodiments, the TV commercial length estimate may be dynamically adjusted throughout the duration of the TV content. As such, the second Al component 216 may consider the platform (e.g., TV app, streaming service, etc.) or the channel being watched as well as how long past TV commercials have historically been to predict the most likely length of the TV commercials / advertisement, which may be, for example and without limitation, a range of 9 to 20 second to 25 to 45 second increments.

[0040] Additionally, the instructions 206 in memory 208 may also include a third Al component 218, as depicted in Fig. 2. In some embodiments, the third Al component 218 may be configured to build an advertising profile 240 associated with a particular user. For example, when TV content is identified, the third Al component 218 may catalogue the content based upon the genres and categories into which the film / show / documentary (e.g., TV content) fits. Such cataloguing may be based upon the fingerprinting data 244 associated with the TV content, which such fingerprinting data 244 is available. For example, particular TV content, such as a TV show, may have a genre associated with it, collected or imported from a suitable externaldatabase, e.g., movie and / or TV rating sites. In the event that the fingerprinting data 244 is not available, the third Al component 218 may use computer vision techniques to recognize / identify into which genre the TV content likely falls. For example, the third Al component 218 may use modified open-source computer vision techniques to identify objects using the fingerprinting data and use that information to add to the advertising profile 240. For example, if the third Al component 218, via computer visioning, identifies a large number of construction materials and homes, the TV content may be identified as a “home improvement” genre and that may be added to the targeted advertisements / profiles of that user and may impact which targeted advertisements they will receive in the future.

[0041] The third Al component 218 may then use the identified genre(s) to develop an advertising profile 240 forthat particular user. As such, the third Al component 218 may use the knowledge of what TV shows / films / video games, and any other information watched on the TV to identify aspects of that user as in what they like, what demographic they are in, what their age category likely is, what topics they are interested in, their political views, their online history, potential occupation or job, potential education or degree, language preferences, and other information in conjunction with data collected by the user device like its geographical location that would allow advertisements and commercials to be targeted towards topics that they will find more interesting to fit their personality and demographics, and other collected or inferred information. Thus, continuing the above example, when the third Al component 218 identifies the genre as “home improvement”, the third Al component 218 may add a home improvement category to the advertising profile 240 and an associated value as to how important that category is to their overall advertising profile.

[0042] The instructions 206 stored in memory 208 may further include a live broadcast component 222 configured to receive live television broadcasts, e g, sporting events, news, special events, award shows, etc., and generate fingerprinting data that can be read and / or used by the one or more Al components 214-216 to determine whether or not a commercial (e.g., logo-based, fade-to-black change, audio change, closed-captioning change, etc.) is present in real-time that have not previously been aired. Thus, the comparison referenced above may use the database 238 as well as the output of the broadcast component 222 for making such determinations. Additionally, such fingerprinting data can then be used across other user devices utilizing the system and method described herein.

[0043] Although described above as multiple, e.g., individual, Al components 214- 218, the skilled artisan will appreciate that a single Al component may be configured to perform the functionality and methods described above. Thus, the illustration herein of multiple Al components is intended solely as one non-limiting embodiment. The Al components 214, 216, and / or 218 referenced above may be implemented, at least in part, using for example and without limitation a machine learning (ML) or artificial neural network (ANN) model. An example ML model may include mathematical representations or define computing capabilities for making inferences from input data based on patterns or relationships identified in the input data. For example, previous viewing patterns, known fingerprinting data, logo-recognition, color-changes, audio changes, etc. As used herein, the term “inferences” can include one or more of decisions, predictions, determinations, or values, which may represent outputs of the ML model. The computing capabilities may be defined in terms of certain parameters of the ML model, such as weights and biases. Weights may indicate relationships between certain input data and certain outputs of the ML model, and biases are offsets which may indicate a starting point for outputs of the ML model. An example ML model operating on input data may start at an initial output based on the biases and then update its output based on a combination of the input data and the weights.

[0044] ML models may be characterized in terms of types of learning that generate specific types of learned models that perform specific types of tasks. For example, different types of machine learning include supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc. ML models may be used to perform different tasks such as classification or regression, where classification refers to determining one or more discrete output values from a set of predefined output values, and regression refers to determining continuous values which are not bounded by predefined output values. The ML models may also use manual user input as to whether a TV commercial has started or stopped (in the event that it does not recognize the change) to further learn and improve the ML model.

[0045] ANN and / or other ML models may be implemented in various types of processing circuits along with memory and applicable instructions therein. For example, general-purpose hardware circuits, such as one or more central processing units (CPUs), one or more graphics processing units (GPUs), or suitable combinations thereof, may be employed to implement a model. In some implementations, one or more tensor processing units (TPUs), neural processing units (NPUs), or otherspecial-purpose processors, field-programmable gate arrays (FPGAs), applicationspecific integrated circuits (ASICs), or the like may also be employed. In some implementations, the ML model may be implemented by a NPU or a TPU embedded in a system on chip (SoC) along with other components, such as one or more CPUs, GPUs, etc. A SoC includes several components manufactured on a shared semiconductor substrate. The NPU or TPU may be controlled by the one or more CPUs by configuring the ML model implemented by the NPU or TPU with weights and biases, providing certain training data to the ML model to configure the ML model, or providing input data to the ML model to obtain related inferences. The one or more CPUs may also receive the inferences and be configured to perform certain actions based on the inferences produced by the ML model. The actions performed by the one or more CPUs may include sending commands to other components of the SoC or components external to the SoC to perform certain actions. For example, the CPU may send commands to a RF transceiver based on the outputs or inferences obtained from an ML model to cause the RF transceiver to operate on a wireless network in accordance with the ML model.

[0046] In one embodiment, the Al components are trained to identify the types of transitions between content. That is, the Al components can be trained to classify a transition as advertisement to advertisement, content to advertisement, content to content, or advertisement to content. The Al components can be trained to identify the transitions along with identifying when an advertisement, content (non-advertisement), or transition is occurring. The output provided by the Al components can include a level of confidence for each type of transition and for the type of content that is being shown. The server 102 can then use the determination component 212, an algorithm, and the confidence levels to determine whether or not to start a targeted advertisement overlay. In some embodiments, the Al components trained to detect the type of transition can be used when a transition is detected, or run continuously.

[0047] In some embodiments, the Al components can be trained to determine transitions based on audio levels or black levels of the content. That is, typically transitions occur when audio levels are near zero for a predetermined amount of time and the screen is completely black. In some embodiments, the audio and video data is stored in memory for a predetermined time to determine when transitions are occurring. For example, using a 15 second buffer, the Al components can access 13 seconds of TV content, 1 second of transition time, then 1 second of theadvertisement. The Al components can use this buffer to compare audio and video attributes to determine when transitions occur. That is, if the audio level is high, and then almost zero, then high again, the Al components can determine that a transition occurred. Or if the video contains various levels of color and light, and then a brief period of near complete blackness, then continues with potentially different or same color and light readings, the Al components can determine that a transition occurred, and the transition was a content to advertisement transition.

[0048] Referring now to Fig. 3, there is shown a simplified functional block diagram of the user device 104 for use in the system 100 of Fig. 1 in accordance with some embodiments. As shown in Fig. 3, the user device 104, e.g., a smart television, mobile device, tablet, etc., includes a controller 300, having a processor 304, which performs the exemplary method by execution of processing instructions 306 that are stored in memory 308 connected to the processor 304, as well as controlling the overall operations of the user device 104. The various components of the controller 300 may be connected by a data / control bus 302.

[0049] The controller 300 of the user device 104 may include one or more input / output (I / O) interface devices 312 for communicating with external devices. The I / O interface 312 may communicate with external devices such as the server 102, the media content provider servers 106-110, the advertiser server 122, and the like, via a suitable communications link 314, as well as a user input device 316, e.g., remote control, keyboard, mouse, touchpad, trackball, voice-activation, etc. In some embodiments, the input device 316 may communicate with the I / O interface 312 via a communications link 318 including, for example and without limitation, wired communication, infrared, Bluetooth®, 802.11(x), or other wired or wireless communication links, as will be appreciated. In such embodiments, the input device 316 may communicate text, channel selections, app selections, menu options, settings, cursor control for a graphical user interface, or the like.

[0050] The user device 104 further includes a display component 310 in communication with the data / control bus 302 and configured to display content to a user. In some embodiments, e.g., wherein the user device 104 is a smart TV, the display component 310 may correspond to any type of display, including, for example and without limitation, LED, LCD, plasma, CRT, OLED, etc. In other embodiments, e.g., wherein the user device 104 is a mobile computing device such as a smart phone, tablet, personal computer / laptop, and wherein the display device 310 may comprisesimilar display types. It will be appreciated that the relative size of the display device 310 may be dependent upon the type of user device 104, e.g., smaller sized for mobile devices, larger sizes for televisions and / or monitors. Accordingly, a plurality of display devices 310 are contemplated herein.

[0051] It will be appreciated that the controller 300 illustrated in Fig. 3 is capable of implementation using a distributed computing environment, such as a computer network, which is representative of any distributed communications system capable of enabling the exchange of data between two or more electronic devices. It will be further appreciated that such a computer network includes, for example and without limitation, a virtual local area network, a wide area network, a personal area network, a local area network, the Internet, an intranet, or any suitable combination thereof. Accordingly, such a computer network comprises physical layers and transport layers, as illustrated by various conventional data transport mechanisms, such as, for example and without limitation, Token-Ring, Ethernet, or other wireless or wire-based data communication mechanisms. Furthermore, while depicted in Fig. 3 as a networked set of components, the controller 300 is capable of implementation on a stand-alone device adapted to interact with the servers 102, 106, 108, 110, and 122 described herein.

[0052] The controller 300 of the user device 104 may include one or more of a computer server, workstation, personal computer, cellular telephone, tablet computer, pager, combination thereof, or other computing device capable of executing instructions for performing the exemplary method. According to one example embodiment, the controller 300 includes hardware, software, and / or any suitable combination thereof, configured to interact with an associated user, a networked device, networked storage, remote devices, or the like.

[0053] The memory 308 illustrated in Fig. 3 as a component of the controller 300 may represent any type of non-transitory computer readable medium such as random access memory (RAM), read only memory (ROM), magnetic disk or tape, optical disk, flash memory, or holographic memory. In one embodiment, the memory 308 comprises a combination of random access memory and read only memory. In some embodiments, the processor 304 and memory 308 may be combined in a single chip. The network interface 312 allows the computer to communicate with other devices via a computer network 101 , (e.g., the Internet), and may comprise amodulator / demodulator (MODEM). Memory 308 may store data processed in the method as well as the instructions for performing the exemplary method.

[0054] The digital processor 304 can be variously embodied, such as by a single core processor, a dual core processor (or more generally by a multiple core processor), a digital processor and cooperating math coprocessor, a digital controller, or the like. The digital processor 304, in addition to controlling the operation of the user device 104, executes instructions 306 stored in memory 308 for performing the method set forth hereinafter.

[0055] The instructions 306 stored in memory 308 of the controller 300 include an extraction component 320 configured to extract fingerprinting data 244 from content being actively displayed on the display device 310. As noted above, such fingerprinting data 244 may correspond to TV programs 246 (as discussed above), TV advertisements / commercials 248, or the like. In some embodiments, the fingerprinting data 244 may include both audio and / or visual components. The extraction component 320 may extract such fingerprinting data 244 responsive to a request received from an app running on the user device 104, from the server 102, or the like. The extraction component 320 may communicate the extracted fingerprinting data 244 to the communication component 322 for further processing.

[0056] The communication component 322 included in the instructions 306 stored in the memory 308 of the controller 300 may be configured to receive and process extracted fingerprinting data 244 from the extraction component 320, transmit real time or past fingerprinting data to the network interface 312 and then to the servers 102, 106, 108, 110, 124, and the like, receive and process instructions for an internal app, receive and process instructions received via the computer network 101 from the server 102, receive and process data (e.g., content) received via the computer network 101 from the servers 106, 108, 110, and / or 122, and the like. In some embodiments, the communication component 322 may be configured to receive and process information received from the input device 316 via the communications link 318 and output various content, images, text, etc., to the display device 310.

[0057] The instructions 306 stored in memory 308 of the controller 300 may further include an overlay component 324 configured to receive, via the communication component 322, an overlay from the server 102. In some embodiments, the overlay component 324 controls the display of the overlay on the display device 310 based upon instructions from the server 102. For example, when the server 102 determines(as discussed above) that the content being displayed on the display device 310 corresponds to an advertisement, a targeted advertisement overlay may be sent to the user device 104 for display on the display device 310. The overlay component 324 may also be configured to selectively overlay only certain portions of the display in the event that the advertising within the content only covers a specific portion of the display (i.e. a banner ad at the bottom of a display or a small commercial in the corner of the display). As noted above, the particular section of the display device 310 to be overlayed may be determined by the first Al component 214 of server 102 and communicated to the user device 104 via the computer network 101 .

[0058] As noted above, the systems and methods set forth herein provide for use of audio / video fingerprinting data 244 and one or more Al components 214, 216, 218 to determine the likelihood of TV commercials playing, their likely duration and end point, and the section of the display that they occupy. In other embodiments, additional features and aspects are contemplated to identify if a TV program is interrupted with an advertisement. Such additional features and aspects, include, for example and without limitation, determining if a rapid shift in the average color or histogram of the video input has occurred. As will be appreciated, TV shows commonly use certain colors or shades of colors within the same episode or movie. Such usage of colors and / or shades may also apply to TV commercials. Accordingly, in some embodiments, a rapid change in the color or histogram (the luminance value of colors or tones), may be used by the systems and methods set forth herein to assist in the determination that there is an elevated chance / possibility / likelihood that a TV commercial has started or ended.

[0059] Another aspect that may be used in determining whether a TV commercial has started or ended by the one or more Al components 214, 216, 218 or other components associated with the server 102 may include identifying when the video input dims or fades to black. As will be appreciated, the dimming or fading to black is commonly associated with a change to and from a TV commercial. Thus, there is an elevated chance that a TV commercial started or stopped at this time, and this information will be considered in the analysis by the one or more Al components 214, 216, 218 in determining the likelihood of a TV commercial playing and ending. For example, in the event that the first Al component 214 is unable to determine with a high enough degree of likelihood if a TV commercial has started, then the TV (e.g., the user device 104) may manually hold the frame black for an extended period of time,typically around a second, after the video signal fades to new content. Such holding may allow the user device 104 to analyze or send fingerprint data to be analyzed by server 102 for additional frames once the video signal unfades to locate / identify fingerprinting data 244 or allow the one or more Al components 214, 216, 218 to determine if the current content is an advertisement or not. Upon a determination that the current content is an advertisement, then the black or other colored frame overlay can fade to new replacement advertisements supplied by the server 102 and / or the advertisement server 122. Upon a determination that the TV content is still playing then, the user device 104 can unfade the TV back to the original video input source. In this manner, if the one or more Al components 214, 216, 218 need additional time to analyze the video source (e.g., current content), the component(s) 214-218 can receive extra frames of the video without causing the TV viewer to see them, thereby enabling a seamless transition to a new advertisement without the viewer seeing the original advertisement for a small portion of time and then having the original advertisement replaced with a new advertisement.

[0060] In other embodiments contemplated herein, analysis of the audio of the current content may be used to provide additional insight as to whether the current content has switched to advertisements. Typically, advertisements will be louder than the average volume of the TV show. Further, the audio quality and signature of a TV show / movie is usually constant throughout its duration. However, TV commercials generally have different audio quality, signature, or spectrum. The systems and methods set forth herein, e.g., the one or more Al components 214, 216, 218 may be configured to recognize this sudden shift to a new audio quality or signature. As such, the first Al component 214 may consider this in increasing the likelihood of determining that a TV commercial has started.

[0061] According to still other embodiments, recognition of a change in the subtitles, whether they disappear or change font can indicate that an advertisement has started. In other embodiments, the one or more Al components 214, 216, 218 may be configured to recognize if certain objects or characteristics are on the screen, such as, for example, and without limitation, flashy and vibrant colors, large text overlays, large logos, rapidly changing scenes, and the like. The aforementioned objects and / or characteristics may be indicative of a TV commercial playing and if these are present in the current content, the first Al component 214, for example, may use thisinformation to determine the likelihood that a TV commercial has occurred and needs to be replaced.

[0062] Recognizing if a logo or title, e.g., news networks, sports networks, cable channels, etc., or some other element on the screen is constantly or commonly displayed when the TV content is playing may also be used by the one or more Al components 214, 216, 218. For example, in the event that that static element changes or disappears, the one or more Al components 214, 216, 218 may interpret such change as indicative that a TV commercial has started or will be starting very soon. Additionally, if a common brand logo or common music / jingle plays or is shown on the screen, the Al component(s) 214-218 may interpret such appearance as indicating that an advertisement has just started. In still other embodiments, additional aspects that may be used as fingerprint techniques by the user device 104 and / or Al components 214, 216, 218 involve analyzing the frame for watermarks, logos, easily identifiable items in the frames, similar motion movement of objects between frames, fingerprinting only sections of the live feed instead of the entire frame, and optical flow (the pattern of how objects movie in a video) information. Furthermore, in some embodiments, one or more of the Al components 214-218 may be configured to detect a commercial using closed-captioning services, e.g., when the closed-captioning starts / stops, the text of the closed-captioning, etc., may be used to identify when a commercial has started or stopped in the original content. Additionally, if one or more Al components identify a change in the type / category of content (e.g., sports, commentary, news, etc.) before and after a transition, that may be used to determine that an advertisement has started or the show a user is watching has changed.

[0063] According to other embodiments, the personalized advertisements referenced herein may compile all the TV channels, shows, movies, video games, and any other content shown on the stream into an algorithm that identifies products or things the user may be interested in to show more of those types of advertisements. In such embodiments, the algorithm, employed by the one or more Al components 214-218 may also log the user’s location to allow local businesses to target advertisements within a certain radius of their store.

[0064] Additionally, the systems and methods described herein may include access to apps that are open and playing (on the user device 104) when the user is loading apps through that native Ul of the smart TV or connected entertainment / streaming box, e.g., user device 104, cable boxes, TV streaming devices like FireTV Stick. Thisinformation may be communicated to the server 102, via the computer network 101 or internally if server 102 is incorporated with the same device, and provided to the one or more Al components 214, 216, 218 to adjust how the components 214-218 determine if a TV commercial is playing or not. For example, YouTube® indicates all of their commercials with a yellow timeline (e.g., yellow graphic bar / line) below the video content. Thus, when the one or more Al components 214, 216, 218 knows or can recognize from fingerprint data that the viewer is watching YouTube®, it can adjust its recognition to also look forthat yellow line below the screen. If an app is opened on an external device connected via HDMI, the one or more Al components 214-218 may be configured to recognize aspects of the user interfaces to determine which app the user has likely opened. This identification of the current app may be used by the one or more Al components 214-218 to determine if the active app has customized characteristics to be identified so as to increase the effectiveness of the one or more Al components 214-218 in determining whether or not a commercial is playing.

[0065] Thus, in accordance with some aspects described herein, the systems and methods incorporate the usage of artificial intelligence, video / audio fingerprinting techniques, and general techniques relating to color, logos, motion movement and other factors that all feed into one or more artificial intelligence components to decide in real-time whether or not a TV commercial is starting and predicting how long it is likely to last for given the type of program that is being watched.

[0066] In accordance with another embodiment, the systems and methods set forth above may include one-click functionality. When a new replacement advertisement is shown on the TV screen, a “buy now” button (e.g., graphical icon, etc.) or similar popup with different wording displayed on the user device 104 that will allow a user to purchase the product or service advertised. In such embodiments, previously entered credit card information, home address, name, email, phone number, and account number to purchase the product shown in the advertisement and send it to the viewer’s home or email address, stored on the server 102 or accessible by the server 102, may be used to effectuate the purchase. If credit card and other information is not already added by the user, then a prompt may be generated on the display 310 of the user device 104 to input such information, e.g., the ability to add it by creating an account via the user device 104 or the prompt may include a QR code on the display 310, which may then be scanned via the user’s mobile device (not shown) and redirectedto an online store where they can purchase the item or use the QR code to add their credit card and contact information to their account.

[0067] In other embodiments one or more of the Al components 214, 216, 218 (or an additional, distinct Al component) may be configured to respond to user queries regarding products or services contained in the targeted advertisement. In such an embodiment, this Al will have an existing understanding of the product or service shown in the advertisement, or access to an associated database containing product or service information. Voice-interaction with such an Al component may also be included to enable a user to use the microphone aspect of some TV remotes or smart devices to ask questions about the product or service advertisements to learn more about it. Alternatively, polls on brand recognition may be shown instead of advertisements.

[0068] In still another embodiment, the user device 104 may communicate with the various components of the system 100 using an associated app, i.e., a dedicated program installed on the user device 104, accessible to the user, and configured with an associated graphical user interface, network connectivity, storage, etc. It will be appreciated that utilizing such an app associated with the Smart TV software that runs on the user device 104 and other devices like a phone, tablet, personal computer, and the like that provides a user interface where users can further customize advertisements, add information about themselves, allow users to redeem points that can be used for gift cards or other products, and customize settings.

[0069] Turning to Fig. 4, the method for real-time detection and replacement of advertisements 400 is described. In one embodiment the server 122 receives, from an associated user device, e.g., the TV device 104, fingerprinting data corresponding to a current content stream displayed on a display of the associated user device 402. In other embodiments the TV device can send user inputs and / or other data indicating what is displayed on the screen or in the current content stream. In some embodiments, the server sends a signal to the TV device to request information from the user or the TV device. Some embodiments may also allow the user TV device 104, to store or cache a set number of advertisements or portion of advertisements within its local memory in order to ensure a seamless and delay free advertisement replacement transition to account for potential internet or communication issues.

[0070] Using the information received from the TV device and / or user, the server determines if an advertisement is present on the current content stream 404. Forexample, the server 102 can use an Al component 214, 216, 218 to determine if an advertisement is present in the current content stream. This is further described herein.

[0071] The server can then send a signal to the user’s device to overlay a targeted advertisement based on the determination if that an advertisement is present 406. The targeted advertisement can replace the advertisement in the current content stream. For example, using at least one Al component, the duration of the advertisement in the current content stream is estimated, and the server can overlay an advertisement with the same or similar duration. The server can also log advertising information corresponding to the overlayed advertisement, the existing advertisement, current content stream, and / or user inputs 408. In some embodiments the server and user device, may be the same physical device.

[0072] The server can send a signal to the user’s device to stop the overlaying of the targeted advertisement based on the estimated duration and / or based on at least one or more Al components determining that advertisements have stopped 410. After the overlay is stopped, the display of the associated user device can return to displaying of the current content stream.

[0073] It can be appreciated by one of skill in the art, that this method can be continuously repeated between the user’s device and server. For example, the user’s device and server can be in constant communication with each other, sending fingerprinting information, user inputs, commands, and the like. Alternatively, the method can be performed based on user inputs, signals from the user’s device, or other information. For example, the server can request fingerprinting data from the user’s device every minute, every thirty seconds, or the like.

[0074] Turning to Figs. 5A-5C, the method for real-time detection and replacement of advertisements 400 is descried in more detail. In one embodiment, determining if advertisement is present on the user’s device includes multiple steps and use of multiple Al components. For example, Fig. 5A describes determining if an advertisement is present on a user’s device 404A. The server can determine if the fingerprinting data 501 is an advertisement by comparing the fingerprint data 501 to monitored live TV channels 502. If the fingerprint data 501 matches the currently monitored TV channels, the server can log any available information to the user’s advertising profile 504 and can send a signal to the user device to start or end theoverlayed advertisements 406 based on whether an advertisement is displayed at the time.

[0075] If the fingerprinting data 501 does not match any currently monitored live TV channels, the server can determine if the fingerprinting data 501 is an advertisement by comparing the fingerprinting data to a database of TV commercials 506. If the fingerprinting data 501 matches data in the database of TV commercials, the server can then determine if the user device is already in “replacement commercial mode” 508, e.g., if an advertisement overlay is present. If an overlayed advertisement is present, the server can provide this information to the third Al component 218 to be used to build out an advertisement profile for the user and log the length of the advertisement in the database 512. If an overlayed advertisement is not present, the server 102 can send a signal to the user device 104 to fade out content and send a new advertisement 510. This step can also send a signal to the user device to prepare for a new advertisement.

[0076] For instance, turning to Fig. 5B, the server 102 can send a signal to the user device 104 to fade out the displayed contend and prepare the user device 104 to receive a new advertisement 511 . The server 102 can use the third Al component to determine the estimated length of the TV commercial that was displayed on the user device, divide the estimated time into potential advertisement spots based on predetermined length requirements 513. The server 102 can then send clients (e.g., advertisers) availability data corresponding with the user and the potential advertising spot 515, e.g., the user’s advertising profile, length of spots, and number of spots.

[0077] In some embodiments, external advertisers can have systems that review the availability data and send real-time bids for the available advertising spots. The server 102 can choose new advertisements based on the real-time bids from the clients. The new advertisements can be hosted on the server 102, on the client’s servers, or sent from the client to the server 102 from the client’s servers. The server 102 can than send the new advertisements to the user device with signals to unfade the screen and display the new advertisements 519. The server can also update the logs with new information, e.g., if new advertisements are being displayed to the user.

[0078] Referring back to Fig. 5A, if the fingerprinting data does not match anything in the database of TV commercials 506, the server can then compare the fingerprinting data to a database of TV shows, movies, and / or other publicly available video content 510.

[0079] If the fingerprinting data does not match any data in the database of TV shows, movies, and / or other publicly available video content, the server 102 can use the first Al component 214 to determine the likelihood of the fingerprinting data of corresponding to a TV commercial or a non-TV commercial 418. The first Al component 214 can also determine if the fingerprinting data corresponds to a new TV commercial. In some embodiments, the first Al component 214 can use information and data from the second Al component 216 and / or the third Al component 218 in the analyst.

[0080] Based on the one or more Al component’s determination, the server 102 will determine if the fingerprinting data corresponds to a commercial 520. If the server 102 determines that the fingerprinting data corresponds a TV commercial, the server than can then determine if the local TV is already in replacement commercial mode 508. In some embodiments, the Al component 214 can use thresholding in the determination, e.g., “more than likely to be a TV commercial,” numerical analysis with chances greater than 50%, or the like. Additionally, if the fingerprinting data is determined to correspond to a TV commercial, the fingerprinting data can be added as a potential new TV commercial in the database of TV commercials, with an identifier or flag indicating further review 524.

[0081] If the server 102 determines that the fingerprinting data does not correspond to a TV commercial, the server can then determine if the local TV is already in replacement commercial mode and log the user information 522. For instance, if an advertisement overlay is active, the server 102 can send a signal to the user device 104 to end the overlay and fade back to original current content stream. If an advertisement overlay is not active, the fingerprinting data can be logged and categorized to be used in a future determination. Additionally, information can be added to the user’s advertising profile. One in the art would recognize that step 508 and step 522 are similar and interchangeable based on preferred usage.

[0082] Turning to Fig. 5C, in conjunction with determining if an advertisement is present on a user’s device 404A, the server 102 can determine if the user device is already in replacement commercial mode 404B. In one embodiment, the server 102 can determine if the fingerprinting data and / or other data sent from the user device 104 corresponds with the “replacement commercial mode.” That is, the fingerprint data and other data 501 from the user device can be analyzed by the server 102, including the Al components 214, 216, 218 to determine if the user device is in replacementcommercial mode, e.g., with an active overlayed advertisement. If the server determines that the user device is in a replacement commercial mode, the server 102 will continuously analyze the TV commercials and update the estimated commercial length 532. That is, the server 102 can use at least one of the Al components to analyze the TV commercials on the current content stream to update the predicted length of the commercial using all available information and databases. This allows the server to continuously update the advertisement overlay to with new advertisements while not using the advertisement overlay when the TV show or content is available. For instance, if a new TV commercial is determined, the server 102 can send new advertisements similar to step 510, including communicating with external clients and conducting a bidding process for new advertisement spots.

[0083] If the server 102 determines that the user device is not in replacement mode 530, the server 102 can use at least one Al component to analyze the fingerprinting data and information from the user device to determine the likelihood of a TV commercial or advertisement appearing soon 534. For example, the first Al component 214 can be used to determine the likelihood of the current content being an advertisement or not and the second Al component 216 can determine if an advertisement may be starting imminently. For example, the Al analyzation 534 can determine the best fingerprinting rate based on the likelihood of an advertisement appearing and the amount of detail needed by each fingerprinting technique based on the likelihood of an advertisement appearing.

[0084] Based on the Al analyzation 534, the server 102 can determine if a TV commercial is likely to start within a few frames of the current content stream 536. If a commercial is not starting, the server 102 can send a “follow” command to the user device 104. For example, a user device 104 receiving a “follow” command can update fingerprinting collection / gathering settings based on information sent by the server 102, can wait for a request from the server 102 for further instructions, and / or continue to send fingerprinting data based on a predetermined time period.

[0085] If the server 102 determines that a TV commercial is likely to start, the server 102 can send a command to the user device to fade the user device’s screen to black and prepare for the replacement commercial mode 540. The server 102 can then continuously analyze fingerprint data 501 sent from the user device to determine if a TV commercial did start 542. The server 102 can use the methods described in Fig. 5A to determine whether a TV commercial started. Particularly, the server 102 can useat least one Al component or recognized database to determine if the fingerprinting data corresponds to a TV commercial.

[0086] If the server 102 determines that a TV commercial started, the server can send a signal to the user device to enable replacement commercial mode 540 and prepare for an advertisement served by the server 102. Alternatively, if the server 102 determines that a TV commercial is not starting, the server 102 can send a signal to the user device to fade back to the current content stream 546.

[0087] Additionally, regarding Fig. 5A and Fig. 5C, in one embodiment, when the server determines the fingerprint data 501 matches known TV commercial from a database 506 or other video database 512, the server 102 can add information regarding that TV commercial or other video content to the user’s advertisement profile. For example, the server 102 can add information about the show name, the specific episode, categories the show belongs to, and other metadata information that is relevant to advertisers. The server 102 can then analyze the fingerprint data 501 using the Al components 534.

[0088] One in the art would recognize that the method described in Fig. 5C can be simultaneously performed with the method described in Fig. 5A. Additionally, one would recognize the interchangeability of the similar steps of the methods described herein. Figs. 6A-6E and 7A-7N illustrate different embodiments of the method described herein. While Figs. 6A-6E comprise one embodiment and Figs. 7A-7N comprise another embodiment, one in the art would recognize that the portions of methods can be interchangeable.

[0089] Additionally, the fingerprinting data 501 can include information corresponding to user inputs. For example, the fingerprinting data 501 can include all remote buttons clicked, their duration, time stamps, and the like. Additionally, the remote buttons can be used to manually indicate when to start and stop the advertisement overlays as well as allowing the user to use a “Buy it Now” feature or answer questions. This information can be included in the fingerprinting data 501. Further, the fingerprinting data 501 can include metadata about the user’s time in their region, their IP address, currently open apps, the time spent viewing current content stream, or the like.

[0090] In one embodiment, the fingerprinting data 501 includes the genre, channel watch, type of content, whether the content matches footage in a database, or if an Al component has made a determination about the content. This information can also befeed back into the Al components as training materials to improve their accuracy. In other embodiments, the fingerprinting data 501 can include a button or button sequence entered on the remote by the user. This button or button sequence can indicate if the user wanted to use the microphone on the remote or device to ask a question about a product they have seen in advertisements. In this case, the server 102 can run the audio captured by the microphone through voice to text software and a large language model that is trained about products in recent advertisements. The server 102 can then send a signal to the user device 104 to display information regarding products or answers to their questions while the advertisements continue to play or are paused.

[0091] In one embodiment, the fingerprinting data 501 can include user entered information such as payment information, email address, delivery location, and the like. This information can be used with fingerprinting data 501 that corresponds to the user indicating they want to buy the product shown in an advertisement. This information can also be saved to the user’s advertisement profile.

[0092] The server 102 can also use fingerprinting data 501 to determine when to activate “delay mode.” For example, the fingerprinting data 501 can include the last time the TV input device has been used. The server 102 can use this information to determine if the input device has not been used in a predetermined amount of time, and the active “delay mode” based on such determination. Additionally, this information can be used to resume normal speed from delay mode. For instance, if the fingerprinting data 501 contains information indicating that the user is using the remote controller. Additionally, the fingerprinting data 501 can indicate that the user or broadcaster has indicated that the TV content is resuming. The server 102 can use this information to send a signal to the user device to fade back to the current content stream.

[0093] In some embodiments, the fingerprinting data 501 can include an indication from the user that the replacement TV advertisements ended late or cut off part of the TV content. This information can be used by the server 102 to adjust or modify the Al component thresholds and add parameters to the user advertisement profile indicating that the user experiences such issues. In some embodiments, the Al components can analyze the fingerprinting data 501 with a higher level of detail to see if the commercial overlay was improperly timed. This information can be stored in the database or user advertisement profile for future improvement of the user experience.

[0094] Information about replacement commercials can also be included in the fingerprinting data 501. This data can include user experiences with the advertisements, such as ratings or survey results. This information can be added to the user advertisement profile or provided to external advertisers to improve user experience and engagement.

[0095] In some embodiments, the server 102 can send a signal to the user device 104 to help improve user experience. For example, if a user is indicating that the advertisement overlay is appearing too early, the server 102 can send a signal to the user device 104 to fade to black, lower audio for to short period of time, and increase the level of fingerprinting data 501 for a short period of time. The server 102 can then use the increased level of fingerprinting data 501 to analyze and determine if the current content stream is an advertisement. This increased fingerprinting can be a higher rate of data transfer, more details in the fingerprinting data, and the like.

[0096] Using the above systems and methods, it is possible to accurately and automatically detect and replace advertisements on a user device. Using the abovedescribed Al components, the systems and methods herein can rapidly utilize input fingerprinted data using computer vision, object recognition, differences in color, visual, and audio signals to determine the likelihood that an advertisement has started / ended, its estimated length, and the likelihood of an advertisement transition occurring based upon the type of content being watched.

[0097] Some portions of the detailed description herein are presented in terms of algorithms and symbolic representations of operations on data bits performed by conventional computer components, including a central processing unit (CPU), memory storage devices for the CPU, and connected display devices. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is generally perceived as a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0098] It should be understood, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, as apparent from the discussion herein, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0099] The exemplary embodiment also relates to an apparatus for performing the operations discussed herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.

[0100] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the methods described herein. The structure for a variety of these systems is apparent from the description above. In addition, the exemplary embodiment is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the exemplary embodiment as described herein.

[0101] A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For instance, a machine-readable medium includes read only memory (“ROM”); random access memory (“RAM”); magnetic disk storage media; optical storage media; flashmemory devices; and electrical, optical, acoustical or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), just to mention a few examples.

[0102] The methods illustrated throughout the specification, may be implemented in a computer program product that may be executed on a computer. The computer program product may comprise a non-transitory computer-readable recording medium on which a control program is recorded, such as a disk, hard drive, orthe like. Common forms of non-transitory computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, or any other magnetic storage medium, CD- ROM, DVD, or any other optical medium, a RAM, a PROM, an EPROM, a FLASH- EPROM, or other memory chip or cartridge, or any other tangible medium from which a computer can read and use.

[0103] Alternatively, the method may be implemented in transitory media, such as a transmittable carrier wave in which the control program is embodied as a data signal using transmission media, such as acoustic or light waves, such as those generated during radio wave and infrared data communications, and the like.

[0104] The foregoing outlines features of several embodiments so that those skilled in the art may better understand the aspects of the present disclosure. Those skilled in the art should appreciate that they may readily use the present disclosure as a basis for designing or modifying other processes and structures for carrying out the same purposes and / or achieving the same advantages of the embodiments introduced herein. Those skilled in the art should also realize that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that they may make various changes, substitutions, and alterations herein without departing from the spirit and scope of the present disclosure

Claims

CLAIMS1. A method for real-time detection and replacement of advertisements, comprising: receiving, from an associated user device, data corresponding to a current content stream displayed on a display of the associated user device; determining, by at least one artificial intelligence component, that an advertisement is present in the current content stream; overlaying a targeted advertisement in response to the determination that the advertisement is present, wherein the targeted advertisement replaces the advertisement in the current content stream; estimating, by the at least one artificial intelligence component, a duration of the advertisement in the current content stream; and stopping the overlaying of the targeted advertisement in response to the estimated duration, wherein the display of the associated user device returns to displaying of the current content stream.

2. The method for real-time detection and replacement of advertisements of claim 1, wherein the determining further comprises: receiving, from the associated user device, fingerprinting data corresponding to the current content stream displayed on the display of the associated user device; comparing, by the at least one artificial intelligence component, the received fingerprinting data to fingerprinting data stored in an associated database; and determining that an advertisement is present in the current content stream in accordance with a result of the comparison.

3. The method for real-time detection and replacement of advertisements of claim 1, wherein the targeted advertisement is selected in accordance with at least one of the content stream, historical data associated with a user account of the associated user device, an external bidding entity, or an advertisement based upon the calculated duration.

4. The method for real-time detection and replacement of advertisements of claim 1, wherein the at least one artificial intelligence component comprises a deep-learning neural network trained to identify advertisements using fingerprinting data.

5. The method for real-time detection and replacement of advertisements of claim 1, wherein the at least one artificial intelligence component comprises a deep-learning neural network configured to estimate the duration of an advertisement, wherein the deep-learning neural network is trained to identify a genre of content using at least one of a past history associated with the genre, a frequency of advertisements associated with the genre or a content provider, or a current display of the content including at least one of a black screen or a logo.

6. The method for real-time detection and replacement of advertisements of claim 1, wherein the at least one artificial intelligence component is trained on computervision to identify objects in the current stream and build an advertising profile associated with the user device.

7. The method for real-time detection and replacement of advertisements of claim 1, wherein the fingerprinting data comprises at least one of a video fingerprint or an audio fingerprint.

8. An apparatus configured for real-time detection and replacement of advertisements, comprising: one or more memories comprising processor-executable instructions; and one or more processors configured to execute the processor-executable instructions and cause the apparatus to: receive, from an associated user device or the same user device, data corresponding to a current content stream displayed on a display of the associated user device, determine, by at least one artificial intelligence component, that an advertisement is present in the current content stream, transmit an overlay of a targeted advertisement in response to the determination that the advertisement is present, wherein the targeted advertisement replaces the advertisement in the current content stream,estimate, by the at least one artificial intelligence component, a duration of the advertisement in the current content stream, and stop the overlay of the targeted advertisement in response to the estimated duration or based on the analysis of the underlying advertisement (advertisement playing underneath the replacement advertisement), wherein the display of the associated user device returns to displaying of the current content stream.

9. The apparatus of claim 8, wherein the one or more processors are configured to execute the processor-executable instructions and cause the apparatus to: receive, from the associated user device, fingerprinting data corresponding to the current content stream displayed on the display of the associated user device; compare, by the at least one artificial intelligence component, the received fingerprinting data to fingerprinting data stored in an associated database; and determine that an advertisement is present in the current content stream in accordance with a result of the comparison.

10. The apparatus of claim 8, wherein the targeted advertisement is selected in accordance with at least one of the content stream, historical data associated with a user account of the associated user device, an external bidding entity, or an advertisement based upon the calculated duration.

11. The apparatus of claim 8, wherein the at least one artificial intelligence component comprises a deep-learning neural network trained to identify advertisements using fingerprinting data.

12. The apparatus of claim 8, wherein the at least one artificial intelligence component comprises a deep-learning neural network configured to estimate the duration of an advertisement, wherein the deep-learning neural network is trained to identify a genre of content using at least one of a past history associated with the genre, a frequency of advertisements associated with the genre or a content provider, or a current display of the content including at least one of a black screen or a logo.

13. The apparatus of claim 8, wherein the at least one artificial intelligence component is trained on computer-vision to identify objects in the current stream and build an advertising profile associated with the user device.

14. The apparatus of claim 8, wherein the fingerprinting data comprises at least one of a video fingerprint or an audio fingerprint.

Images