Real-time television advertisement detection and replacement for smart televisions and networked entertainment devices

Abstract

A system and method for real-time detection and replacement of advertisements. The content currently being displayed on an associated device is analyzed to determine if a commercial advertisement is being played. When a commercial advertisement is detected, a targeted advertisement is overlaid on the content to be displayed in place of the commercial advertisement. Once the original commercial advertisement ends, the overlay stops and the original program content is displayed. One or more artificial intelligence components are utilized to identify commercial advertisements, non-commercial content, specific products or services, targeted advertisements for overlay, etc.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 619788, filed January 11, 2024, which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to the field of advertising, and more particularly to advertising in real-time video transmission. Background Technology

[0003] Internet-connected entertainment devices and smart TVs can receive media content from a variety of sources, including both broadcast and streaming media sources. Streaming services can refer to any type of service that provides media via a data network (such as the internet), including, but not limited to, movies, television programs, short videos, news, etc. Users can access and watch these various media types on smart TVs, smartphones, tablets, computers, video game systems, and more.

[0004] Some streaming services insert advertisements (such as commercials for products or services) into their media content. In some cases, this advertising content is irrelevant to the underlying entertainment content, unsuitable for a specific user, distracting or offensive, non-targeted, and geographically irrelevant, among other things. Therefore, systems and methods are needed to detect and target these advertisements for replacement. Summary of the Invention

[0005] One approach provides a method for real-time detection and replacement of advertisements. The method includes receiving data from an associated user device corresponding to a current content stream displayed on the display of the associated user device, and determining, by at least one artificial intelligence component, that an advertisement exists in the current content stream. The method further includes: overlaying a target advertisement in response to determining the presence of the advertisement, wherein the target advertisement replaces the advertisement in the current content stream; and estimating, by the at least one artificial intelligence component, the duration of the advertisement in the current content stream, the duration being continuously updated as more current content streams are analyzed. Furthermore, the method includes stopping the overlay of the target advertisement in response to the estimated duration or if the at least one artificial intelligence component determines that the advertisement has ended before the estimated duration, wherein the display of the associated user device resumes displaying the currently streaming content.

[0006] Other aspects provide one or more devices operable, configurable, or otherwise adapted to perform any portion of any of the methods described herein (e.g., such that performance can be by only one device or in a distributed manner across multiple devices); one or more non-transitory computer readable media comprising instructions that, when executed by one or more processors of one or more devices, cause the one or more devices to perform any portion of any of the methods described herein (e.g., such that the instructions can be contained on only one computer readable medium or in a distributed manner across multiple computer readable media; such that the instructions can be executed by only one processor or in a distributed manner by multiple processors; such that each of the one or more devices can include one processor or multiple processors; and / or such that performance can be by only one device or in a distributed manner across multiple devices); one or more computer program products instantiated on one or more computer readable storage media and comprising code for performing any portion of any of the methods described herein (e.g., such that the code can be stored on only one computer readable medium or in a distributed manner across computer readable media); and / or one or more devices comprising one or more means for performing any portion of any of the methods described herein (e.g., such that performance can be by only one device or by multiple devices in a distributed manner). As examples, a device can comprise a processing system, a device with a processing system, or processing systems cooperating on one or more networks. BRIEF DESCRIPTION OF DRAWINGS

[0007] Aspects of the disclosure can best be understood from the following detailed description when read with the accompanying drawings and annexed appendix. It is noted that the various drawings are not drawn to scale, according to standard industry conventions. Indeed, the dimensions of the various features can be arbitrarily increased or decreased for clarity of discussion.

[0008] Figure 1 An example system diagram for real-time detection and replacement of advertisements is shown in accordance with some embodiments.

[0009] Figure 2 is an illustrative block diagram of an example server in a system for real-time detection and replacement of advertisements in accordance with some embodiments.

[0010] Figure 3 is an illustrative block diagram of an example user device in a system for real-time detection and replacement of advertisements in accordance with some embodiments.

[0011] Figure 4 is an example method for real-time detection and replacement of advertisements.

[0012] Figure 5A , Figure 5B and Figure 5C are example methods for real-time detection and replacement of advertisements.

[0013] Figure 6A , Figure 6B , Figure 6C , Figure 6D and Figure 6E illustrate example methods for real-time detection and replacement of advertisements.

[0014] Figure 7A , Figure 7B , Figure 7C , Figure 7D , Figure 7E , Figure 7F , Figure 7G , Figure 7H , Figure 7I , Figure 7J , Figure 7K , Figure 7L , Figure 7M and Figure 7N illustrate example methods for real-time detection and replacement of advertisements. DETAILED DESCRIPTION

[0015] The following disclosure provides many different embodiments, or examples, for implementing different features of the provided subject matter. Specific examples of components and arrangements are described below to simplify the present disclosure. These are, of course, merely examples and are not intended to be limiting. For example, in the following description, forming a first feature over or on a second feature can include embodiments where the first feature and the second feature are formed in direct contact with each other, and can also include embodiments where additional features can be formed between the first feature and the second feature such that the first feature and the second feature can not be in direct contact. Furthermore, the present disclosure can repeat certain

[0016] Furthermore, spatially relative terms (e.g., “beneath,” “below,” “lower,” “above,” “upper,” and the like) can be used herein for ease of description to describe one element or feature’s relationship to another element(s) or feature(s) as illustrated in the figures. The spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. The devices can be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

[0017] Numerical values in the specification and claims of this application should be understood to be inclusive of the same value when reduced to the same degree of precision, and when the value is expressed as a range, the range is inclusive of the endpoints thereof. All ranges disclosed herein are inclusive of the endpoints.

[0018] The term "about" can be used to include any numerical value that, when modified by the term "about", can vary from the recited value by a margin of ten percent (10%) or less. When used with a range, "about" also discloses the range by the absolute values of both endpoints. The term "about" can refer to plus or minus 10% of the indicated number.

[0019] In some embodiments, a method is provided that can be implemented on a smart television or other internet-connected entertainment device (e.g., internal smart television circuit board, cable television decoder, digital video recorder (DVR) component, satellite receiver, FireSticks®, Roku® device, Apple® TV device, Google® Chromecast® device, etc.). In such embodiments, the method provides for analysis of content that is currently being received through a currently watched HDMI / USB / wireless, etc. source, generated internally by the device, or the like. In some embodiments, the content can be received via a streaming service, broadcast (e.g., cable or over-the-air television), or other similar source. As used herein, a streaming service, broadcaster, cable provider, etc. are generally referred to as a "content provider," "source," "content source," or other similar encompassing term that is intended to convey the entity and / or device from which video / audio is received. Variations of the source are also contemplated herein, and the examples provided herein are merely intended as non-limiting examples of the source. The analysis can include identifying when (or will) a television commercial or other advertisement appear on a viewing screen or display, e.g., identifying or extracting one or more fingerprints associated with the audio and / or video of the source. A new, highly targeted advertisement can then be generated in real-time to be overlaid on the television commercial or advertisement, without user input. Upon completion of the original television commercial or advertisement, the method provides for a step of ceasing the overlay and returning to the previous source (i.e., the television program, movie, video, etc. that was previously playing). In other words, the method provides for a step of replacing some or all of the advertisement that the viewer would normally see on the television screen or display with a real-time, highly targeted advertisement. In some embodiments contemplated herein, the user can be compensated, which can take the form of points that can be redeemed for gift cards, other promotional offers, or other forms of compensation, in exchange for viewing the replacement advertisement.

[0020] In some embodiments disclosed herein, automatic content recognition can be used to identify advertisements (e.g., commercials, infomercials, program interruptions) and programs (e.g., television programs, videos, movies, news broadcasts, sports broadcasts, i.e., non-advertising media content). This recognition can be based on fingerprints, watermarks, or other identification technologies incorporated into the media content to identify the content being viewed on a user device. Furthermore, this recognition can be based on video components, audio components, or both simultaneously, the channel being viewed, or other sources of information available from the device(s)(s) being used to acquire the media, such as the model and manufacturer of the viewing device (smart TV brand). For example, fingerprint recognition can be implemented on a user device such as a smart TV, which analyzes the individual frames or audio spectra in the current video stream source being displayed on it to extract aspects of the frame or segment that can be used to uniquely identify the content (i.e., a compact digital representation of the video that summarizes the unique characteristics of the material).

[0021] refer to Figure 1 An exemplary system 100 for real-time detection and replacement of advertisements is shown according to one embodiment. Those skilled in the art will understand that... Figure 1 The various components depicted are for illustrative purposes of exemplary embodiments, and other similar components implemented in hardware, software, or a combination thereof can be substituted. It is understood that system 100 can be implemented using a distributed computing environment (e.g., a computer network), which represents any distributed communication system capable of enabling data exchange between two or more electronic devices. It should also be understood that such a computer network includes, for example, but not limited to, virtual local area networks (VLANs), wide area networks (WANs), personal area networks (PANs), local area networks (LANs), the Internet, intranets, or any suitable combination thereof. Therefore, such a computer network includes a physical layer and a transport layer, as illustrated by various conventional data transmission mechanisms, such as, but not limited to, token ring, Ethernet, or other wireless or wire-based data communication mechanisms.

[0022] like Figure 1 As shown, system 100 includes at least one server 102 capable of implementing the exemplary methods described below (the at least one server 102 may be a separate device and / or within the same smart TV (i.e., locally)). Reference will be made below. Figure 2The exemplary server 102 is described in more detail. In some embodiments, server 102 may communicate with a distributed network 101 (e.g., but not limited to the Internet). Server 102 may use any suitable communication link 112 to send or receive data or signals to or from network 101, including, but not limited to, wireless communication (such as Bluetooth, WiMax, 802.11a, 802.11b, 802.11g, 802.11(x), 3G, 4G, 5G cellular communication systems, proprietary communication networks, infrared, optical, public switched telephone networks, or any suitable wireless data transmission system) or wired communication.

[0023] System 100 also includes at least one user equipment 104, in Figure 1 The device shown here is a smart TV. It should be understood that using a smart TV as a user device 104 is intended only as one example of a suitable user device; other examples include, but are not limited to, smartphones, tablets, laptops, desktop computers, streaming media service devices connected to a display device, etc. Therefore, the subject matter systems and methods described herein are not intended to be limited to... Figure 1 The smart TV user device 104 is shown below. (Refer to the following...) Figure 3 Further details regarding the various components of user equipment 104 are discussed below. It should be understood that user equipment 104 can communicate with network 101 via communication link 114. Suitable communication link 114 may include, for example, but not limited to, wireless communication (such as Bluetooth, WiMax, 802.11a, 802.11b, 802.11g, 802.11(x), 3G, 4G, 5G cellular communication systems, proprietary communication networks, infrared, optical, public switched telephone networks, or any suitable wireless data transmission system) or wired communication.

[0024] Figure 1System 100 also includes one or more media content providers, typically represented by servers 106, 108, and 110. These providers (i.e., each of servers 106 to 110) are intended merely to illustrate different sources of media content that can be received and / or displayed by user equipment 104, as will be understood by those skilled in the art. According to some embodiments disclosed herein, each of media content providers 106, 108, and 110 may correspond to a cable television provider, streaming service, broadcast service, etc. Suitable non-limiting examples of such media content providers 106 to 110 may include, but are not limited to, Netflix, Disney+, Hulu, YouTube, Google, Apple TV, Max, PEACOCK, Paramount+, ESPN, Amazon, NBC, ABC, FOX, CBS, CNN, local affiliates, pay services, etc. Furthermore, media content providers 106 to 110 may correspond to subscription-based providers, broadcast-based providers, news, television, etc., which may or may not apply advertising during content presentation. Servers 106 to 110 may include computer servers, workstations, personal computers, cellular phones, tablet computers, pagers, combinations thereof, or other computing devices capable of executing instructions for performing exemplary methods.

[0025] According to one example embodiment, servers 106 to 110 include hardware, software, and / or any suitable combination thereof, and are configured to interact with associated users, networked devices, networked storage devices, remote devices, etc. In other embodiments, each of servers 106 to 110 may represent a collaborative or cloud-based distributed computing platform providing the aforementioned media content, as those skilled in the art will understand. Figure 1 As shown, each of servers 106, 108, and 110 can communicate with network 101 via corresponding communication links 116, 118, and 120. Suitable communication links 116, 118, and 120 may include, for example, but not limited to, wireless communication (such as Bluetooth, WiMax, 802.11a, 802.11b, 802.11g, 802.11(x), 3G, 4G, 5G cellular communication systems, proprietary communication networks, infrared, optical, public switched telephone networks, or any suitable wireless data transmission system) or wired communication.

[0026] like Figure 1 As shown, system 100 may include one or more advertisers, in Figure 1An advertiser server 122 is shown. In some embodiments, the advertiser server 122 can store, send, receive, generate, obtain (or otherwise perform other operations on) advertisements to the server 102 and / or the user device 104. The server 122 can include hardware, software, and / or any suitable combination thereof, and be configured to interact with associated users, networked devices, networked storage, remote devices, etc. As Figure 1 shown, the server 122 can communicate with the network 101 via a communication link 124. Suitable communication links 124 can include, for example and without limitation, wireless communication (such as Bluetooth, WiMax, 802.11a, 802.11b, 802.11g, 802.11(x), 3G, 4G, 5G cellular communication systems, proprietary communication networks, infrared, optical, public switched telephone network, or any suitable wireless data transmission system), or wired communication. In some embodiments, one or more such advertiser servers 122 can be included, corresponding to different entities providing or requesting advertisement transmissions to the user device 104, as discussed in more detail below. In some embodiments, the advertiser server 122 can be included in or associated with the content provider servers 106, 108, 110. Further, in an embodiment, software, machine learning models, or any other information can be remotely updated at predetermined time intervals, or when a security update or new feature update is available, for both the server 122 and / or 102. Still further, the software and machine learning models can be customized based on the exact country, region, and / or city to account for any differences in common television programs and programming and to account for different languages.

[0027] Turning now to Figure 2 , an exemplary diagram of the server 102 of Figure 1 is shown in accordance with some embodiments. As shown in Figure 2 , the server 102 capable of implementing the methods set forth herein includes a processor 204 that executes instructions that are stored in a memory 208 coupled to the processor 204 to implement the exemplary methods of this disclosure and to operate the server 102. Various components of the server 102 can be connected by a data / control bus 202. The processor 204 of the server 102 can communicate with an associated database 238 via a suitable communication link 228. Suitable communication link 228 can include, for example, a switched telephone network, a wireless radio communication network, infrared, optical, or other suitable wired or wireless data communication. The database 238 can be implemented on a component of the server 102, for example, stored in local memory 208, i.e., on a hard drive, virtual drive, etc., or on a remote memory accessible to the server 102. It will be appreciated that while the database 238 is shown as being separate from the server 102, the database 238 can be implemented on the server 102, for example, as a component of the memory 208. Figure 1and Figure 2 While depicted as a single device, server 102 may represent a cloud-based computing system, i.e., a distributed processing system, and the representation of server 102 as a single device is intended only as a non-limiting illustrative example.

[0028] The associated database 238 represents any organized collection of data (e.g., advertisements 240, training data 242, fingerprint data 244, user / account information 250, user device information 252, content provider information 254, etc.) used for one or more purposes. Those skilled in the art will understand that this information can be updated via machine learning during the operation of the subject system 100. Implementation of the associated database 238 can occur on any high-capacity storage device (e.g., magnetic storage drives, hard disk drives, optical storage devices, flash memory devices, or suitable combinations thereof). The associated database 238 can be implemented as a component of server 102, for example, residing in memory 208 or a similar arrangement. In one embodiment, the associated database 238 may include, for example, but not limited to, data corresponding to television commercials, television programs, movies, video games, sporting events, news videos, educational videos, TV shopping advertisements, static advertisements, and other advertising and non-advertising content. In other embodiments, database 238 may further include, for example, but not limited to: information about the user associated with user device 104, such as preferences, types (TV, movies, video games, videos, etc.), contact information, username, account information, targeted advertising information, user (one or more) statistics, geolocation, behavioral information, click-through rate, externally shared posts, hobbies and activities, usage data, ad interactions, ad engagement metrics, surveys and user feedback, IP address, time spent on various applications or movie types, etc.; information about user device 104, such as brand, model, connectivity, display capabilities, installed applications, etc.; and content provider information, such as subscription, name, account information, account level (with or without ads), points collection rate per ad (depending on how much data about the user is collected), streaming capabilities, etc.; and other information related to the transmission and reception of digital media content. In some embodiments, database 238 may store various fingerprint data 244 for identifying advertising and non-advertising content. Figure 2 As shown, the fingerprint data 244 may include, for example, but not limited to, television program fingerprint data 246 and television advertisement fingerprint data 248.

[0029] Server 102 may include one or more input / output (I / O) interface devices 234 and 236 for communicating with external devices. I / O interface 236 may communicate via communication link 226 with one or more of the following: a display device 230 for displaying information, and a user input device 232. The user input device 232 may be, for example, a keyboard, touchscreen, or writable screen for inputting text, a remote control with buttons, and / or a cursor control device (e.g., mouse, trackball, etc.) for transmitting user input information and command selections to processor 204. I / O interface 234 may communicate via suitable communication link 224 with external devices (e.g., user device 104, media content provider servers 106 to 110, advertiser server 122, etc.).

[0030] It should be understood that Figure 2 The server 102 shown can be implemented using a distributed computing environment (e.g., a computer network), which represents any distributed communication system capable of enabling data exchange between two or more electronic devices. It should be further understood that this computer network includes, for example, but not limited to, virtual LANs, wide area networks, personal area networks, local area networks, the Internet, intranets, or any suitable combination thereof. Therefore, this computer network includes a physical layer and a transport layer, as illustrated by various conventional data transmission mechanisms, such as, but not limited to, token ring, Ethernet, or other wireless or wire-based data communication mechanisms. Furthermore, although in Figure 2 The system is described as a collection of networked components, but server 102 can be implemented on a separate device adapted to interact with servers 106, 108, 110, 122 and user equipment 104 as described herein.

[0031] Server 102 may include one or more of a computer server, workstation, personal computer, cellular phone, tablet computer, pager, combination thereof, or other computing device capable of executing instructions for performing exemplary methods. According to one example embodiment, server 102 includes hardware, software, and / or any suitable combination thereof configured to interact with associated users, networked devices, networked storage, remote devices, etc.

[0032] Figure 2The memory 208 of the component shown as server 102 can represent any type of non-transitory computer-readable medium, such as random access memory (RAM), read-only memory (ROM), magnetic disk or tape, optical disk, flash memory, or holographic memory. In one embodiment, memory 208 includes a combination of random access memory and read-only memory. In some embodiments, processor 204 and memory 208 may be combined in a single chip. Network interfaces (one or more) 234, 236 allow the computer to communicate with other devices via computer network 101 (e.g., the Internet) and may include a modem. Memory 208 may store data processed in the method as well as instructions for performing the exemplary method.

[0033] The digital processor 204 can be implemented in various ways, such as by a single-core processor, a dual-core processor (or more generally by a multi-core processor), a digital processor and a cooperative mathematical coprocessor, a digital controller, etc. In addition to controlling the operation of the server 102, the digital processor 204 also executes instructions 206 stored in memory 208 for performing the methods described below.

[0034] like Figure 2 As shown, the instructions 206 stored in memory 208 may include a comparison unit 210 configured to receive fingerprint data from user equipment 104 and compare the fingerprint data with stored fingerprint data 244. In some embodiments, server 102 may access information about fingerprint data 244 (in... Figure 2 The comparison unit 210 then uses one or more different databases (shown as TV program fingerprint data 246 and TV commercial fingerprint data 248). The comparison unit 210 can then output the comparison result with the database (i.e., corresponding to TV program data 246 or TV commercial data 248) to the determination unit 212. In other words, the comparison unit 210 analyzes the fingerprint data received from the user equipment 104 via the computer network 101 and compares it with the fingerprint data 244 stored in the database 238. The result of this comparison is then transmitted to the determination unit 212 for further analysis.

[0035] The instructions stored in memory 208 may further include a determining component 212 configured to determine, based on the output from the comparing component 210, whether the received fingerprint data matches advertisement 248 (i.e., a television commercial or ad) or program 246 (i.e., non-advertisement related news broadcasts, television programs, movies, sports broadcasts, videos, video games, etc.). When the determining component 212 determines with a specific degree of certainty that the received fingerprint data matches a known (i.e., identified or stored fingerprint) television program 248 and provides this information to the first AI component 214 (described in more detail below), the server 102 may communicate to the user device 104 not to overlay any new target advertisements 240 and to stop any currently overlaid target advertisements 240 (if applicable). However, if the determining component 212 determines that the received fingerprint data matches stored television advertisement fingerprint data 248, the determining component 212 communicates this determination information to the first AI component 214, which may then communicate this determination information to the overlay generation component 220, as discussed in more detail below.

[0036] If the determining component 212 determines that the received fingerprint data does not match any known (i.e., previously stored) fingerprint data 244 or matches multiple fingerprint data stored in more than one database, the instructions 206 stored in memory 208 include a first artificial intelligence (AI) component 214 configured to determine whether the received fingerprint data corresponds to advertising or non-advertising content. In other words, when the received fingerprint data cannot be identified in any database or in multiple databases (i.e., program 246 or advertisement 248), the first AI component 214 can be implemented to determine the probability that the current content is an advertisement or not. The first AI component 214 can also be used to identify the type of transition that is taking place (including, but not limited to, TV ad to TV ad, TV content to TV ad, TV content to TV content, or TV ad to TV ad). In all cases, before initiating or stopping ad overlay, the first AI component 214 can be utilized to combine the information provided by the determining component 212, considering various aspects (including, but not limited to, recognizing changes in color, audio, objects, motion paths, text, etc.), to make an informed decision about when to initiate and stop ad overlay. In some embodiments, the first AI component 214 may be a deep learning neural network artificial intelligence component. In this embodiment, the first AI component 214 may be trained on training data 242, which includes fingerprint data 244 as its input, for identifying whether the content is a commercial advertisement 248. If the first AI component 214 determines that the content is a commercial advertisement 248, the first AI component 214 may send instructions to the overlay generation component 220 to overlay the existing content onto the target advertisement 240, and to stop the overlay operation in response to the determination that the existing content (e.g., the original commercial advertisement) has previously stopped. The overlay feature is also capable of selectively overlaying only certain portions of the display to address situations where advertisements within the content only cover specific portions of the display (e.g., a banner ad at the bottom of the display or a small commercial ad in a corner of the display). The exact portion of the display to be overlaid may be determined by the first AI component 214 via server 102.

[0037] In some embodiments, the first AI component 214 may be further configured to determine whether no input has been received from a remote control (or other input device) within a predetermined time period (e.g., indicating that the user has not interacted with user device 104). (It should be understood that component 214 may also include non-AI elements, such as computer programs conventional in the art that receive information about the last button pressed by the user on the remote control device). In this embodiment, user device 104 and the first AI component 214 (together or separately) may reduce the frame rate (e.g., slow down the display) to allow additional time for processing and analysis by the first AI component 214. For example, and not limitingly, user device 104 may begin to slightly slow down the content and introduce a small delay between the television signal (e.g., the source) and the content displayed on the screen. This delay allows the first AI component 214 and comparison / determination components 210 to 212 more time and additional frames to determine whether the content is an advertisement. Then, if the remote control is used and a button is clicked for any purpose other than volume adjustment, the content may be slightly sped up to return to its live feed while the user is navigating the UI using the remote control.

[0038] The instructions 206 stored in memory 208 may further include a second AI component 216 configured to estimate the possible length of a commercial ad interruption and the likelihood of the commercial ad appearing. In some embodiments, the second AI component 216 may be implemented as a deep learning neural network; however, other types of artificial intelligence and / or machine learning may be used, and the implementation of the deep learning neural network is intended only as one possible embodiment.

[0039] According to some embodiments, the second AI component 216 can be trained on training data 242 to identify different types of television content (e.g., live news, movies, TV programs, etc.), and based on these categories, the past history and frequency of television commercials on that particular channel or television content, as well as currently available information, will be used to determine whether an advertisement is likely to begin. This currently available information includes, but is not limited to, whether the screen dims to black, whether the broadcaster logo disappears, whether the average color, histogram, optical flow (the pattern of object movement in a video), or content brightness changes rapidly, or whether the audio quality, signature, volume, or spectrum changes rapidly, whether bright colors suddenly appear, large text overlays, large logos appear, and whether closed captions change, or whether other common changes associated with an advertisement or interruption are noticed. When the second AI component 216 determines that an advertisement may be about to begin, it can begin to fade the television screen (e.g., the monitor, etc.) to black. The second AI component 216 can then allow short clips of television content to be sent to server 102 for comparison with database 238 for the first AI component 214 to determine whether it is a television commercial. If so, the new ad will replace the TV commercial. If not, the screen will return to black. Additionally, while the screen remains black, audio can be muted or reduced, and this audio will fade in and out of the screen simultaneously with the fade-in and fade-out of black.

[0040] In some embodiments, the second AI component 216 can also be configured to generate an estimate of the length of a television commercial interruption in order to allocate potential time slots for replacing the television commercial. In this embodiment, the television commercial length estimate can be dynamically adjusted throughout the entire duration of the television content. Thus, the second AI component 216 can take into account the platform (e.g., a television app, streaming service, etc.) or the channel being watched, and how long past television commercials have historically been, to predict the most likely length of a television commercial / advertisement, which can be (e.g., rather than limited) an increment of 9 to 20 seconds to 25 to 45 seconds.

[0041] Furthermore, the instructions 206 in memory 208 may also include a third AI component 218, such as Figure 2As shown. In some embodiments, the third AI component 218 can be configured to create an advertising profile 240 associated with a specific user. For example, when television content is identified, the third AI component 218 can catalog the content based on the type and category to which the movie / program / documentary (e.g., television content) belongs. This cataloging can be based on fingerprint data 244 associated with the television content (if the fingerprint data 244 is available). For example, a particular television content (e.g., a television program) may have an associated type, which is collected or imported from a suitable external database (e.g., a movie and / or television rating website). If the fingerprint data 244 is not available, the third AI component 218 can use computer vision technology to identify / determine what type the television content may belong to. For example, the third AI component 218 can identify objects by utilizing fingerprint data and using modified open-source computer vision technology, and use that information to add them to the advertising profile 240. For example, if the third AI component 218 identifies a large number of building materials and houses via computer vision, the TV content can be identified as "home improvement" type, and this type can be added to the user's targeted ads / profile, potentially affecting the targeted ads the user will receive in the future.

[0042] The third AI component 218 can then use the identified type (one or more) to build an advertising profile 240 for that specific user. In this way, the third AI component 218 can utilize knowledge of television programs / movies / video games watched on television and any other information to identify aspects of the user, such as what the user likes, which statistical group the user belongs to, what the user's age category might be, topics the user is interested in, the user's political views, the user's online history, potential occupation or job, potential education or degree, language preferences, and other information combined with data collected from the user's device (such as the user's device's geolocation) that will allow advertisements and commercials to target topics that the user would be more interested in, adapting to the user's personality and statistical information, as well as other collected or inferred information. Therefore, continuing the example above, when the third AI component 218 identifies the type as "home improvement," the third AI component 218 can add the home improvement category to the advertising profile 240 and add a correlation value to indicate the importance of that category to its overall advertising profile.

[0043] The instructions 206 stored in memory 208 may further include a live broadcast component 222 configured to receive live television broadcasts (e.g., sporting events, news, special events, award ceremonies, etc.) and generate fingerprint data that can be read and / or used by one or more AI components 214 to 216 to determine whether commercials (e.g., logo-based, fade-in-black, audio changes, closed caption changes, etc.) are present in real time within previously unplayed content. Therefore, the comparison described above can be made using database 238 and the output of broadcast component 222. Furthermore, this fingerprint data can subsequently be used among other user devices utilizing the systems and methods described herein.

[0044] Although described above as multiple (e.g., individual) AI components 214 to 218, those skilled in the art will understand that a single AI component can be configured to perform all of the functions and methods described above. Therefore, the illustrations of multiple AI components herein are intended only as a non-limiting embodiment. The AI ​​components 214, 216, and / or 218 described above can be implemented at least in part using, for example, but not limited to, machine learning (ML) or artificial neural network (ANN) models. Example ML models may include mathematical representations or definitions of computational capabilities for inferring from input data based on patterns or relationships identified in the input data. Examples include previous viewing patterns, known fingerprint data, logo recognition, color variations, audio variations, etc. As used herein, the term "inference" can include one or more of decision, prediction, determination, or value, which may represent the output of the ML model. Computational capabilities can be defined based on some parameters of the ML model, such as weights and biases. Weights can indicate the relationship between some input data and some outputs of the ML model. Bias is an offset that can indicate the starting point of the ML model's output. An example ML model that operates on input data can start with an initial output based on a bias, and then update its output based on a combination of input data and weights.

[0045] ML models can be characterized by a learning type that generates a specific type of learning model to perform a specific type of task. For example, different types of machine learning include supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc. ML models can be used to perform different tasks (such as classification or regression), where classification refers to determining one or more discrete output values ​​from a set of predefined output values, while regression refers to determining continuous values ​​that are not restricted by predefined output values. ML models can also be further learned and improved using manual user input regarding whether a television commercial has started or stopped (if the ML model has not identified a change).

[0046] ANN and / or other ML models can be implemented in various types of processing circuitry, along with their memory and applicable instructions. For example, general-purpose hardware circuitry, such as one or more central processing units (CPUs), one or more graphics processing units (GPUs), or suitable combinations thereof, can be used to implement the model. In some implementations, one or more tensor processing units (TPUs), neural processing units (NPUs), or other dedicated processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., can also be used. In some implementations, the ML model can be implemented by an NPU or TPU embedded in a system-on-a-chip (SoC) and other components such as one or more CPUs, GPUs, etc. An SoC comprises several components fabricated on a shared semiconductor substrate. An NPU or TPU can be controlled by one or more CPUs to: configure the ML model implemented by the NPU or TPU using weights and biases; provide specific training data to the ML model to configure it; or provide input data to the ML model to obtain relevant inferences. One or more CPUs can also receive inferences and be configured to perform actions based on the inferences generated by the ML model. Actions performed by one or more CPUs can include sending commands to other components of the SoC or components outside the SoC to perform actions. For example, a CPU can send commands to an RF transceiver based on the output or inference obtained from an ML model, so that the RF transceiver can operate on a wireless network according to the ML model.

[0047] In one embodiment, the AI ​​component is trained to identify the type of transition between content. That is, the AI ​​component can be trained to classify transitions as ad-to-ad, content-to-ad, content-to-content, or ad-to-content. The AI ​​component can be trained to recognize transitions and when an ad, content (non-ad), or transition occurs. The output provided by the AI ​​component can include a confidence level for each type of transition and the type of content being displayed. Server 102 can then use the determining component 212, the algorithm, and the confidence level to determine whether to initiate targeted ad overlay. In some embodiments, the AI ​​component trained to detect transition types can be used when a transition is detected, or it can run continuously.

[0048] In some embodiments, the AI ​​component can be trained to determine transitions based on the audio or black levels of the content. That is, a transition typically occurs when the audio level is close to zero for a predetermined time period and the screen is completely black. In some embodiments, audio and video data are stored in memory for a predetermined period to determine when a transition occurs. For example, using a 15-second buffer, the AI ​​component can access 13 seconds of television content, a 1-second transition time, and then a 1-second commercial. The AI ​​component can use this buffer to compare audio and video attributes to determine when a transition occurs. That is, if the audio level is high, then almost zero, and then high again, the AI ​​component can determine that a transition has occurred. Or, if the video contains various color and brightness levels, then briefly approaches complete black, and then continues with potentially different or the same color and brightness readings, the AI ​​component can determine that a transition has occurred, and that the transition is from content to commercial.

[0049] Now for reference Figure 3 This illustrates a method for using some embodiments. Figure 1 A simplified functional block diagram of user equipment 104 in system 100. (See diagram below.) Figure 3 As shown, user equipment 104 (e.g., smart TV, mobile device, tablet, etc.) includes a controller 300, which has a processor 304. The processor 304 executes exemplary methods and controls the overall operation of user equipment 104 by executing processing instructions 306 stored in a memory 308 connected to the processor 304. Various components of the controller 300 can be connected via a data / control bus 302.

[0050] The controller 300 of user equipment 104 may include one or more input / output (I / O) interface devices 312 for communicating with external devices. The I / O interfaces 312 may communicate with external devices (such as server 102, media content provider servers 106-110, advertiser server 122, etc.) and with user input devices 316 (e.g., remote control, keyboard, mouse, touchpad, trackball, voice activation, etc.) via suitable communication links 314. In some embodiments, the input devices 316 may communicate with the I / O interfaces 312 via communication links 318, which include, but are not limited to, wired communication, infrared, Bluetooth®, 802.11(x), or other wired or wireless communication links, as should be understood. In this embodiment, the input devices 316 may transmit text, channel selection, application selection, menu options, settings, cursor control for graphical user interfaces, etc.

[0051] User equipment 104 also includes a display component 310, which communicates with a data / control bus 302 and is configured to display content to a user. In some embodiments, such as when user equipment 104 is a smart TV, display component 310 may correspond to any type of display, including, but not limited to, LED, LCD, plasma, CRT, OLED, etc. In other embodiments, such as when user equipment 104 is a mobile computing device such as a smartphone, tablet, or personal computer / laptop, display device 310 may include a similar display type. It should be understood that the relative size of display device 310 may depend on the type of user equipment 104, for example, smaller for mobile devices and larger for TVs and / or monitors. Therefore, multiple display devices 310 are contemplated herein.

[0052] It should be understood that Figure 3 The controller 300 shown can be implemented using a distributed computing environment (e.g., a computer network), which represents any distributed communication system capable of enabling data exchange between two or more electronic devices. It should also be understood that this computer network includes, for example, but not limited to, virtual LANs, wide area networks, personal area networks, local area networks, the Internet, intranets, or any suitable combination thereof. Therefore, this computer network includes a physical layer and a transport layer, as illustrated by various conventional data transmission mechanisms, such as, but not limited to, token ring, Ethernet, or other wireless or wire-based data communication mechanisms. Furthermore, although in Figure 3 The controller 300 is described as a collection of networked components, but it can be implemented on a standalone device adapted to interact with the servers 102, 106, 108, 110 and 122 described herein.

[0053] The controller 300 of user equipment 104 may include one or more of the following: a computer server, workstation, personal computer, cellular phone, tablet computer, pager, combination thereof, or other computing device capable of executing instructions for performing exemplary methods. According to one example embodiment, the controller 300 includes hardware, software, and / or any suitable combination thereof, and is configured to interact with associated users, networked devices, networked storage, remote devices, etc.

[0054] Figure 3The memory 308, a component shown as controller 300, can represent any type of non-transitory computer-readable medium, such as random access memory (RAM), read-only memory (ROM), magnetic disk or magnetic tape, optical disk, flash memory, or holographic memory. In one embodiment, memory 308 includes a combination of random access memory and read-only memory. In some embodiments, processor 304 and memory 308 may be combined in a single chip. Network interface 312 allows the computer to communicate with other devices via computer network 101 (e.g., the Internet) and may include a modem. Memory 308 may store data processed in the method as well as instructions for performing the exemplary method.

[0055] The digital processor 304 can be implemented in various ways, such as by a single-core processor, a dual-core processor (or more generally by a multi-core processor), a digital processor and a cooperative math coprocessor, a digital controller, etc. In addition to controlling the operation of the user equipment 104, the digital processor 304 also executes instructions 306 stored in memory 308 for performing the methods described below.

[0056] The instructions 306 stored in the memory 308 of the controller 300 include an extraction component 320 configured to extract fingerprint data 244 from content actively displayed on the display device 310. As described above, the fingerprint data 244 may correspond to a television program 246 (as described above), a television commercial / advertisement 248, etc. In some embodiments, the fingerprint data 244 may include both audio and / or visual components. The extraction component 320 may extract the fingerprint data 244 in response to requests received from an application running on the user device 104, requests received from the server 102, etc. The extraction component 320 may communicate the extracted fingerprint data 244 to a communication component 322 for further processing.

[0057] The communication component 322, which includes instructions 306 stored in the memory 308 of the controller 300, can be configured to: receive and process extracted fingerprint data 244 from the extraction component 320; transmit real-time fingerprint data or past fingerprint data to the network interface 312, and then to servers 102, 106, 108, 110, 124, etc.; receive and process instructions for internal applications; receive and process instructions received from server 102 via computer network 101; receive and process data (e.g., content) received from servers 106, 108, 110, and / or 122 via computer network 101, etc. In some embodiments, the communication component 322 can be configured to receive and process information received from input device 316 via communication link 318, and output various content, images, text, etc. to display device 310.

[0058] The instructions 306 stored in the memory 308 of the controller 300 may further include an overlay component 324 configured to receive overlay content from the server 102 via a communication component 322. In some embodiments, the overlay component 324 controls the display of the overlay content on the display device 310 based on instructions from the server 102. For example, when the server 102 determines (as described above) that the content displayed on the display device 310 corresponds to an advertisement, the target advertisement overlay content may be sent to the user device 104 for display on the display device 310. The overlay component 324 may also be configured to selectively overlay only certain portions of the display to address situations where advertisements within the content only cover specific portions of the display (i.e., banner ads at the bottom of the display or small commercial ads in the corners). As described above, the specific portion of the display device 310 to be overlaid may be determined by the first AI component 214 of the server 102 and communicated to the user device 104 via the computer network 101.

[0059] As described above, the systems and methods described herein provide the use of audio / video fingerprint data 244 and one or more AI components 214, 216, 218 to determine the likelihood of a television commercial being played, the possible duration and end point of the television commercial, and the portion of the display occupied by the television commercial. In other embodiments, additional features and aspects are envisioned to identify whether a television program has been interrupted by a commercial. Such additional features and aspects include, for example, but not limited to, determining whether a rapid change has occurred in the average color or histogram of the video input. As should be understood, television programs typically use certain colors or hues within the same episode or film. This use of color and / or hue can also be applied to television commercials. Therefore, in some embodiments, rapid changes in color or histogram (the brightness values ​​of a color or hue) can be used by the systems and methods described herein to assist in determining an increased likelihood that a television commercial has started or ended.

[0060] Another aspect that can be used by one or more AI components 214, 216, 218, or by other components associated with server 102 to determine whether a television commercial has started or ended may include recognizing when the video input dims or fades to black. As should be understood, dimming or fading to black is generally associated with the transition of a television commercial. Therefore, the probability that a television commercial has started or stopped at this time is increased, and this information will be considered by one or more AI components 214, 216, 218 when determining the probability of a television commercial playing and ending. For example, if the first AI component 214 cannot determine with sufficient certainty whether a television commercial has started, the television (e.g., user device 104) may manually hold the frame black for a period of time, typically about one second, after the video signal fades to new content. This holding may allow user device 104 to analyze or send fingerprint data to server 102 for analysis after the video signal fades in to obtain additional frames to locate / identify fingerprint data 244 or allow one or more AI components 214, 216, 218 to determine whether the current content is an advertisement. Once it is determined that the current content is an advertisement, the overlay of black or other colored frames can fade out to a new replacement advertisement provided by server 102 and / or ad server 122. If it is determined that the television content is still playing, the user device 104 can fade the television back to the original video input source. In this way, if one or more AI components 214, 216, 218 require additional time to analyze the video source (e.g., the current content), components 214 to 218 can receive additional frames of video without causing television viewers to see them, thus achieving a seamless transition to the new advertisement, where viewers neither see the original advertisement (even for a short time) nor experience the process of the original advertisement being replaced by the new advertisement.

[0061] In other embodiments envisioned herein, the audio of the current content can be analyzed to provide additional insights into whether the current content has switched to an advertisement. Typically, advertisements will be louder than the average volume of a television program. Furthermore, the audio quality and signature of a television program / movie are generally constant throughout its duration. However, television commercials often have different audio qualities, signatures, or spectra. The systems and methods described herein (e.g., one or more AI components 214, 216, 218) can be configured to recognize such abrupt shifts to new audio quality or signatures. Thus, the first AI component 214 can take this into account when increasing the likelihood that a television commercial has begun.

[0062] According to other embodiments, recognizing changes in subtitles (whether the subtitles disappear or the font changes) can indicate that an advertisement has started. In other embodiments, one or more AI components 214, 216, 218 can be configured to recognize whether certain objects or features are on the screen, such as, but not limited to, bright colors, large text overlays, large logos, rapidly changing scenes, etc. The aforementioned objects and / or features can indicate that a television commercial is playing, and if the object and / or feature is present in the current content, then, for example, the first AI component 214 can use this information to determine the likelihood that a television commercial has occurred and needs to be replaced.

[0063] One or more AI components 214, 216, 218 may also use the recognition of whether a logo or title (e.g., news network, sports network, cable channel, etc.) or another element on the screen is continuously or normally displayed during television content playback. For example, in the event of a change or disappearance of a static element, one or more AI components 214, 216, 218 may interpret such a change as indicating that a television commercial has started or is about to start. Furthermore, if a generic brand logo or generic music / brochure is played or displayed on the screen, AI components 214 to 218 may interpret such appearance as indicating that an advertisement has just begun. In other embodiments, additional aspects of the fingerprinting technology that may be used as user device 104 and / or AI components 214, 216, 218 involve: analyzing watermarks, logos, easily identifiable items in frames, and similar motion of objects between frames; fingerprinting only a portion of the live feed rather than the entire frame; and optical flow (patterns of object movement in video). Furthermore, in some embodiments, one or more of the AI ​​components 214 to 218 can be configured to use a closed captioning service to detect commercials. For example, the start / stop of closed captions, the text of the closed captions, etc., can be used to identify when a commercial in the original content begins or stops. Additionally, if one or more AI components identify a change in the content type / category (e.g., sports, commentary, news, etc.) before and after the conversion, this can be used to determine that an advertisement has started or that the program the user is watching has changed.

[0064] According to other embodiments, the personalized advertising mentioned herein can compile all television channels, programs, movies, video games, and any other content displayed in the stream into an algorithm that identifies products or things the user may be interested in, in order to display more ads of those types. In this embodiment, the algorithm employed by one or more AI components 214 to 218 can also record the user's location to allow local businesses to target advertising within a specific radius of their stores.

[0065] Furthermore, the systems and methods described herein may include access to an application that is opened and played when a user loads an application via the local UI of a smart TV or a connected entertainment / streaming box (e.g., user device 104, cable TV box, FireTV Stick, etc.). This information may be communicated to server 102 via computer network 101 or internally (if server 102 is associated with the same device) and provided to one or more AI components 214, 216, 218 to adjust how components 214-218 determine whether a television commercial is playing. For example, YouTube® indicates all its commercials below the video content with a yellow timeline (e.g., a yellow graphic bar / line). Therefore, when one or more AI components 214, 216, 218 know or can identify from fingerprint data that a viewer is watching YouTube®, the one or more AI components 214, 216, 218 may adjust their recognition to similarly look for the yellow line below the screen. If an application is opened on an external device connected via HDMI, one or more AI components 214-218 can be configured to identify aspects of the user interface to determine which application the user may have opened. This identification of the current application can be used by one or more AI components 214-218 to determine whether the active application possesses custom characteristics to be identified, thereby improving the effectiveness of the one or more AI components 214-218 in determining whether a commercial advertisement is playing.

[0066] Therefore, based on some aspects described herein, the system and method combine the use of artificial intelligence, video / audio fingerprinting technology, and general technologies related to color, logos, motion, and other factors, all of which are fed into one or more AI components to determine in real time whether a television commercial is starting and to predict how long the commercial may last, given the type of program being watched.

[0067] According to another embodiment, the above system and method may include a one-click function. When a new replacement advertisement is displayed on a television screen, a “Buy Now” button (e.g., a graphic icon, etc.) or a similar pop-up with different wording displayed on user device 104 will allow the user to purchase the advertised product or service. In this embodiment, previously entered credit card information, home address, name, email, phone number, and account number (used to purchase the product displayed in the advertisement and send it to the viewer's home or email address) are stored on or accessible by server 102 and can be used to make the purchase. If the user has not yet added credit cards and other information, a prompt can be generated on display 310 of user device 104 to enter such information. For example, the user can add this information by creating an account on user device 104, or the prompt may include a QR code on display 310, which can then be scanned by the user's mobile device (not shown) and redirected to an online store where the user can purchase the item or add the user's credit card and contact information to their account using the QR code.

[0068] In other embodiments, one or more of the AI ​​components 214, 216, 218 (or additional, different AI components) may be configured to respond to user queries about products or services included in a targeted advertisement. In this embodiment, the AI ​​will have an existing understanding of the products or services shown in the advertisement, or access to an associated database containing information about the products or services. Voice interaction with the AI ​​component may also be included, enabling users to use the microphone of certain TV remote controls or smart devices to ask questions about the product or service advertisement to learn more. Alternatively, a poll on brand awareness may be displayed instead of an advertisement.

[0069] In another embodiment, user equipment 104 can communicate with various components of system 100 using an associated application. That is, the associated application is a dedicated program installed on user equipment 104, accessible to the user, and configured with an associated graphical user interface, network connectivity, memory, etc. It should be understood that this application, associated with smart TV software, runs on user equipment 104 and other devices (such as telephones, tablets, personal computers, etc.) and provides a user interface in which the user can further customize advertisements, add information about themselves, allow users to redeem points for gift cards or other products, and customize settings.

[0070] Turn Figure 4The document describes a method 400 for real-time detection and replacement of advertisements. In one embodiment, server 122 receives fingerprint data 402 corresponding to the current content stream displayed on the display of the associated user device (e.g., television device 104). In other embodiments, the television device may send user input and / or other data indicating content displayed on the screen or in the current content stream. In some embodiments, the server sends a signal to the television device to request information from the user or the television device. Some embodiments may also allow the user television device 104 to store or cache a set number of advertisements or portions of advertisements in its local memory to ensure seamless and lag-free advertisement replacement transitions to address potential internet or communication problems.

[0071] Using information received from television devices and / or users, the server determines whether an advertisement exists in the current content stream 404. For example, server 102 may use AI components 214, 216, 218 to determine whether an advertisement exists in the current content stream. This is described further herein.

[0072] The server can then signal the user device to overlay a target advertisement 406 based on the determination of the advertisement's presence. The target advertisement can replace an advertisement in the current content stream. For example, by using at least one AI component to estimate the duration of an advertisement in the current content stream, the server can overlay an advertisement with the same or similar duration. The server can also record advertisement information 408 corresponding to the overlaid advertisement, existing advertisements, the current content stream, and / or user input. In some embodiments, the server and the user device can be the same physical device.

[0073] The server can determine that the advertisement has stopped based on the estimated duration and / or based on at least one or more AI components, and send a signal to the user device to stop overlaying the target advertisement 410. After the overlay stops, the display of the associated user device can return to displaying the current content stream.

[0074] Those skilled in the art will understand that this method can be continuously repeated between the user device and the server. For example, the user device and the server can continuously communicate with each other to send fingerprint information, user input, commands, etc. Alternatively, the method can be executed based on user input, signals from the user device, or other information. For example, the server can request fingerprint data from the user device every minute, every thirty seconds, etc.

[0075] Turn Figures 5A to 5C The method for real-time detection and replacement of advertisement 400 is described in more detail. In one embodiment, determining the presence of an advertisement on a user's device includes multiple steps and utilizes multiple AI components. For example, Figure 5AThe description covers determining whether an advertisement exists on a user's device 404A. The server can determine whether fingerprint data 501 is an advertisement by comparing it with a monitored live TV channel 502. If fingerprint data 501 matches the currently monitored TV channel, the server can record any available information in the user's advertisement profile 504 and can send a signal to the user's device to start or stop the overlay advertisement 406 based on whether an advertisement is currently displayed.

[0076] If fingerprint data 501 does not match any currently monitored live TV channels, the server can determine whether fingerprint data 501 is an advertisement by comparing it with the TV commercial database 506. If fingerprint data 501 matches data in the TV commercial database, the server can then determine whether the user device is in "replacement commercial mode" 508 (e.g., whether an ad overlay exists). If an overlay ad exists, the server can provide this information to the third AI component 218 for creating an ad profile for the user and recording the ad length in database 512. If an overlay ad does not exist, the server 102 can signal the user device 104 to fade out the content and send a new ad 510. This step can also signal the user device to prepare to receive a new ad.

[0077] For example, turning Figure 5B Server 102 can send a signal to user device 104 to fade out the displayed content and prepare user device 104 to receive new advertisements 511. Server 102 can use a third AI component to determine the estimated length of the TV commercials displayed on the user device, and divide the estimated time into potential ad slots 513 based on predetermined length requirements. Server 102 can then send availability data 515 corresponding to users and potential ad slots (e.g., user's ad profile, slot length, and number of slots) to clients (e.g., advertisers).

[0078] In some embodiments, an external advertiser may have a system that reviews availability data and sends real-time bids for available ad slots. Server 102 may select new ads based on real-time bids from clients. New ads may be hosted on server 102, on a client's server, or sent from a client to server 102. Server 102 may then send the new ad along with a signal indicating that it is fading into the screen and being displayed to the user device 519. The server may also update its logs with new information (e.g., whether a new ad is being shown to the user).

[0079] Return to reference Figure 5AIf the fingerprint data does not match any content in the TV commercials database 506, the server can then compare the fingerprint data with the database 510 of TV programs, movies and / or other publicly available video content.

[0080] If the fingerprint data does not match any data in a database of television programs, movies, and / or other publicly available video content, server 102 can use first AI component 214 to determine the likelihood 418 that the fingerprint data corresponds to a television commercial or a non-television commercial. First AI component 214 can also determine whether the fingerprint data corresponds to a new television commercial. In some embodiments, first AI component 214 may use information and data from second AI component 216 and / or third AI component 218 in the analysis.

[0081] Based on the determination made by one or more AI components, server 102 will determine whether the fingerprint data corresponds to a commercial 520. If server 102 determines that the fingerprint data corresponds to a television commercial, the server can then determine whether the local television is already in a replacement commercial mode 508. In some embodiments, AI component 214 may use thresholding processing in the determination (e.g., “very likely to be a television commercial,” numerical analysis of a greater than 50% chance, etc.). Furthermore, if the fingerprint data is determined to correspond to a television commercial, the fingerprint data may be added as a potential new television commercial to the television commercial database with an identifier or flag 524 indicating that further review is required.

[0082] If server 102 determines that the fingerprint data does not correspond to a television commercial, the server can then determine whether the local television is already in commercial overlay mode and record user information 522. For example, if the commercial overlay is active, server 102 can send a signal to user device 104 to end the overlay and fade back to the original current content stream. If the commercial overlay is inactive, the fingerprint data can be recorded and categorized for future determination. Furthermore, the information can be added to the user's advertising profile. Those skilled in the art will recognize that steps 508 and 522 are similar and interchangeable based on preferred uses.

[0083] Turn Figure 5CBased on the determination of whether an advertisement exists on the user device 404A, server 102 can determine whether the user device is in a replacement commercial advertising mode 404B. In one embodiment, server 102 can determine whether fingerprint data and / or other data sent from user device 104 corresponds to a "replacement commercial advertising mode". That is, fingerprint data and other data 501 from the user device can be analyzed by server 102 (including AI components 214, 216, 218) to determine whether the user device is in a replacement commercial advertising mode (e.g., with active overlay advertisements). If the server determines that the user device is in a replacement commercial advertising mode, server 102 will continuously analyze television commercials and update the estimated commercial length 532. That is, server 102 can use at least one of the AI ​​components to analyze television commercials on the current content stream to update the predicted length of the commercials using all available information and databases. This allows the server to continuously update ad overlay content with new advertisements while not using ad overlays when television programs or content are available. For example, if a new television commercial is identified, server 102 can send a new advertisement similar to step 510, which includes communicating with external clients and a bidding process for the new ad slot.

[0084] If server 102 determines that the user device is not in replacement mode 530, server 102 can use at least one AI component to analyze fingerprint data and information from the user device to determine the probability 534 that a television commercial or an advertisement is about to appear. For example, a first AI component 214 can be used to determine the probability that the current content is an advertisement or not, and a second AI component 216 can determine whether an advertisement is likely to begin soon. For example, AI analysis 534 can determine the optimal fingerprint recognition rate based on the probability of an advertisement appearing and the amount of detail required for each fingerprint recognition technology, the amount of detail being based on the probability of an advertisement appearing.

[0085] Based on AI analysis 534, server 102 can determine whether a television commercial is likely to begin within a few frames of the current content stream 536. If the commercial does not begin, server 102 can send a "follow" command to user device 104. For example, user device 104 receiving the "follow" command can update its fingerprint acquisition / collection settings based on the information sent by server 102, wait for requests from server 102 for further instructions, and / or continue sending fingerprint data based on a predetermined time period.

[0086] If server 102 determines that a television commercial may begin, server 102 can send a command to the user device to fade the user device's screen to black and prepare to switch to commercial mode 540. Then, server 102 can continuously analyze the fingerprint data 501 sent from the user device to determine whether the television commercial has indeed begun 542. Server 102 can use... Figure 5A The method described herein determines whether a television commercial has commenced. In particular, server 102 may use at least one AI component or an identified database to determine whether fingerprint data corresponds to a television commercial.

[0087] If server 102 determines that a television commercial has started, the server may send a signal to the user device to enable the replacement commercial mode 540 and prepare for the advertisement to be provided by server 102. Alternatively, if server 102 determines that a television commercial has not started, server 102 may send a signal to the user device to fade back to the current content stream 546.

[0088] In addition, regarding Figure 5A and Figure 5C In one embodiment, when the server determines that fingerprint data 501 matches a known TV commercial or other video database 512 from database 506, server 102 can add information about the TV commercial or other video content to the user's advertising profile. For example, server 102 can add metadata information about the program name, specific episode, program category, and other metadata related to the advertiser. Server 102 can then use AI component 534 to analyze fingerprint data 501.

[0089] Those skilled in the art should recognize that, Figure 5C The method described in the text can be compared with... Figure 5A The methods described herein are performed simultaneously. Furthermore, those skilled in the art should recognize the interchangeability of similar steps in the methods described herein. Figures 6A to 6E and Figures 7A to 7N Different embodiments of the method described herein are shown. Although Figures 6A to 6E Including one embodiment, Figures 7A to 7N Another embodiment is included, but those skilled in the art will recognize that parts of these methods may be interchangeable.

[0090] In addition, fingerprint data 501 may include information corresponding to user input. For example, fingerprint data 501 may include all clicked remote control buttons, their duration, timestamps, etc. Furthermore, remote control buttons can be used to manually indicate when to start and stop ad overlays, and to allow users to use the "Buy Now" feature or answer questions. This information can be included in fingerprint data 501. Additionally, fingerprint data 501 may include metadata related to the user's time in their region, their IP address, currently open applications, and the time spent viewing the current content stream.

[0091] In one embodiment, fingerprint data 501 includes type, channel viewed, content type, whether the content matches footage in a database, or whether the AI ​​component has determined the content. This information can also be fed back into the AI ​​component as training material to improve its accuracy. In other embodiments, fingerprint data 501 may include a button or button sequence entered by the user on a remote control. This button or button sequence may indicate whether the user wants to use the microphone on the remote control or device to ask questions about the product seen in the advertisement. In this case, server 102 can run the audio captured by the microphone through speech-to-text software and a large language model trained on products from recent advertisements. Server 102 can then send a signal to user device 104 to display information about the product or answers to questions about the product when the advertisement continues or is paused.

[0092] In one embodiment, fingerprint data 501 may include information entered by the user, such as payment information, email address, delivery location, etc. This information can be used in conjunction with fingerprint data 501, which corresponds to the user indicating a desire to purchase the product shown in the advertisement. This information may also be saved to the user's advertising profile.

[0093] Server 102 can also use fingerprint data 501 to determine when the "delay mode" is activated. For example, fingerprint data 501 may include the time when the television input device was last used. Server 102 can use this information to determine whether the input device has not been used for a predetermined time and activate the "delay mode" based on this determination. Furthermore, this information can be used to restore normal speed from delay mode. For example, if fingerprint data 501 contains information indicating that the user is using a remote control. Additionally, fingerprint data 501 may indicate that the user or broadcaster has instructed television content to be resumed. Server 102 can use this information to send a signal to the user device to fade back to the current content stream.

[0094] In some embodiments, fingerprint data 501 may include indications from the user (such as a TV commercial ending too late or partial TV content being cut off). Server 102 can use this information to adjust or modify AI component thresholds and add parameters to the user's ad profile to indicate that the user is experiencing such issues. In some embodiments, the AI ​​component may analyze fingerprint data 501 at a higher level of detail to see if commercial overlays are being improperly timed. This information may be stored in a database or user ad profile for future user experience improvements.

[0095] Information regarding the replacement of commercial advertisements can also be included in fingerprint data 501. This data may include user experiences with the advertisements (such as ratings or survey results). This information can be added to the user's advertising profile or provided to external advertisers to improve user experience and engagement.

[0096] In some embodiments, server 102 may send signals to user device 104 to help improve the user experience. For example, if a user indicates that an ad overlay appears too early, server 102 may send signals to user device 104 to fade it to black, lower the audio for a short time, and briefly increase the level of fingerprint data 501. Server 102 can then use the increased level of fingerprint data 501 to analyze and determine whether the current content stream is an ad. This increased fingerprint recognition could be a higher data transfer rate, more detail in the fingerprint data, etc.

[0097] Using the system and method described above, advertisements on user devices can be accurately and automatically detected and replaced. By using the AI ​​components described above, the system and method of this paper can leverage input fingerprint data and utilize computer vision, object recognition, color, and differences in visual and audio signals to quickly determine the likelihood that an advertisement has started / ended, its estimated length, and the probability of an advertisement conversion based on the type of content being viewed.

[0098] Some parts of this article are described in detail in the form of algorithms and symbolic representations of operations on data bits performed by conventional computer components, including the central processing unit (CPU), memory storage devices for the CPU, and connected display devices. These algorithmic descriptions and representations are the most effective means for those skilled in the field of data processing to communicate the substance of their work to others. Algorithms are generally considered to be a self-consistent sequence of steps that leads to a desired result. These steps are steps that require physical operations on physical quantities. Typically (though not always), these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated. It is sometimes convenient, primarily for general purposes, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.

[0099] However, it should be understood that all these and similar terms are associated with appropriate physical quantities and are merely convenient labels for application to these quantities. Unless otherwise specifically stated, as is apparent from the discussion herein, it should be understood that throughout the description, the use of terms such as “processing” or “computing” or “operation” or “determining” or “displaying” refers to the actions and processes of a computer system or similar electronic computing device that manipulate and convert data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memory or registers or other such information storage, transmission, or display devices.

[0100] Exemplary embodiments also relate to means for performing the operations discussed herein. The means may be specifically constructed for the desired purpose, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a computer-readable storage medium such as, but not limited to, any type of disk (including floppy disks, optical disks, CD-ROMs, and magneto-optical disks), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards or optical cards, or any type of medium suitable for storing electronic instructions, each storage medium being coupled to a computer system bus.

[0101] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general-purpose systems can be used with the programs taught herein, or constructing more specialized devices to perform the methods described herein may prove convenient. The structure of these various systems is apparent from the description above. Furthermore, exemplary embodiments are not described with reference to any particular programming language. It should be understood that the teachings of the exemplary embodiments described herein can be implemented using various programming languages.

[0102] Machine-readable media include any mechanism for storing or transmitting information in a machine-readable (e.g., computer) form. Examples of machine-readable media include: read-only memory (“ROM”); random access memory (“RAM”); magnetic disk storage media; optical storage media; flash memory devices; and electrical, optical, acoustic, or other forms of propagation signals (e.g., carrier waves, infrared signals, digital signals, etc.), to name just a few.

[0103] The methods illustrated throughout the specification can be implemented in a computer program product that can be executed on a computer. The computer program product may include a non-transitory computer-readable recording medium (e.g., a disk, hard disk drive, etc.) on which control programs are recorded. Common forms of non-transitory computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape or any other magnetic storage media, CD-ROMs, DVDs or any other optical media, RAM, PROMs, EPROMs, FLASH-EPROMs or other memory chips or cassette tapes, or any other tangible media that a computer can read and use.

[0104] Alternatively, the method can be implemented in a temporary medium (e.g., a transmissible carrier wave), wherein the control program is implemented using data signals from a transmission medium (e.g., sound waves or light waves, such as those generated during radio wave and infrared data communication).

[0105] The foregoing has outlined features of several embodiments to enable those skilled in the art to better understand aspects of this disclosure. Those skilled in the art will understand that this disclosure can be readily used as the basis for designing or modifying other processes and structures to achieve the same purposes and / or advantages of the embodiments described herein. Those skilled in the art will also recognize that such equivalent constructions do not depart from the scheme and scope of this disclosure, and that various changes, substitutions, and modifications can be made to the embodiments disclosed herein without departing from the spirit and scope of this disclosure.

Claims

1. A method for real-time detection and replacement of advertisements, comprising: Receive data from the associated user device corresponding to the current content stream displayed on the display of the associated user device; The presence of the advertisement in the current content stream is determined by at least one artificial intelligence component; In response to determining that the advertisement exists, a target advertisement is overlaid, wherein the target advertisement replaces the advertisement in the current content stream; The duration of the advertisement in the current content stream is estimated by the at least one artificial intelligence component; and In response to the estimated duration, the overlay of the targeted advertisement is stopped, wherein the display of the associated user device resumes displaying the current content stream.

2. The method for real-time detection and replacement of advertisements according to claim 1, wherein, The determination also includes: Receive fingerprint data corresponding to the current content stream displayed on the display of the associated user device from the associated user device; The received fingerprint data is compared by the at least one artificial intelligence component with fingerprint data stored in an associated database; and Based on the results of the comparison, it is determined that the advertisement exists in the current content stream.

3. The method for real-time detection and replacement of advertisements according to claim 1, wherein, The targeted advertisement is selected based on at least one of the following: the content stream, historical data associated with the user account of the associated user device, an external bidding entity, or an advertisement based on a calculated duration.

4. The method for real-time detection and replacement of advertisements according to claim 1, wherein, The at least one artificial intelligence component includes a deep learning neural network trained to use fingerprint data to identify advertisements.

5. The method for real-time detection and replacement of advertisements according to claim 1, wherein, The at least one artificial intelligence component includes a deep learning neural network configured to estimate the duration of an advertisement, wherein the deep learning neural network is trained to identify the type of content using at least one of the following: past history associated with the type, advertising frequency associated with the type or content provider, or the current display of the content, the current display of the content including at least one of a black screen or a logo.

6. The method for real-time detection and replacement of advertisements according to claim 1, wherein, The at least one artificial intelligence component is trained based on computer vision to identify objects in the current stream and build an advertising profile associated with the user device.

7. The method for real-time detection and replacement of advertisements according to claim 1, wherein, The fingerprint data includes at least one of video fingerprints or audio fingerprints.

8. An apparatus configured for real-time detection and replacement of advertisements, comprising: One or more memories, the one or more memories including processor-executable instructions; as well as One or more processors, the one or more processors being configured to execute processor-executable instructions and cause the device to perform the following operations: Receive data from the associated user device or the same user device corresponding to the current content stream displayed on the display of the associated user device. The presence of the advertisement in the current content stream is determined by at least one artificial intelligence component. In response to determining the existence of the advertisement, overlay content of the target advertisement is transmitted, wherein the target advertisement replaces the advertisement in the current content stream. The duration of the advertisement in the current content stream is estimated by the at least one artificial intelligence component, and In response to the estimated duration or based on analysis of the underlying advertisement, the overlay of the target advertisement is stopped, wherein the underlying advertisement is an advertisement played below the replacement advertisement, and the display of the associated user device resumes displaying the current content stream.

9. The apparatus according to claim 8, wherein, The one or more processors are configured to execute processor-executable instructions and cause the device to perform the following operations: Receive fingerprint data corresponding to the current content stream displayed on the display of the associated user device from the associated user device; The received fingerprint data is compared with fingerprint data stored in an associated database by the at least one artificial intelligence component; as well as Based on the results of the comparison, it is determined that the advertisement exists in the current content stream.

10. The apparatus according to claim 8, wherein, The targeted advertisement is selected based on at least one of the following: the content stream, historical data associated with the user account of the associated user device, an external bidding entity, or an advertisement based on a calculated duration.

11. The apparatus according to claim 8, wherein, The at least one artificial intelligence component includes a deep learning neural network trained to use fingerprint data to identify advertisements.

12. The apparatus according to claim 8, wherein, The at least one artificial intelligence component includes a deep learning neural network configured to estimate the duration of an advertisement, wherein the deep learning neural network is trained to identify the type of content using at least one of the following: past history associated with the type, advertising frequency associated with the type or content provider, or the current display of the content, the current display of the content including at least one of a black screen or a logo.

13. The apparatus according to claim 8, wherein, The at least one artificial intelligence component is trained based on computer vision to identify objects in the current stream and build an advertising profile associated with the user device.

14. The apparatus according to claim 8, wherein, The fingerprint data includes at least one of video fingerprints or audio fingerprints.

Images