Systems and methods for media boundary detection

Abstract

Systems and methods are provided for detecting media boundaries for linear media programming. A display device may receive a first cue from a first video frame of media being presented by the display device and generate a first prediction of a first content type represented by the first video frame using the trained machine learning model and the first cue. The display device may then receive a second cue from a second video frame of media being presented by the display device and generate a second prediction of a second content type represented by the second video frame. The display device may determine a probability that the first content type does not match the second content type based on the first prediction and the second prediction, thereby identifying a boundary between different media being presented by the display device.

Description

[Technical field]

[0001]

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 294,319, filed December 28, 2021, which is incorporated by reference in its entirety for all purposes.

[0002]

[0002] This disclosure relates generally to media boundary detection, and more particularly to detecting media boundaries for linear media programming. [Background technology]

[0003]

[0003] Analysis of home television viewing has been ongoing since the original Nielsen TV ratings became the foundational statistic for audience measurement, influencing advertising rates and television series renewals. Progress in improving audience measurement has followed both improvements in technology in the home and the growth of the Internet. Modern audience measurement includes media consumption that extends well beyond traditional broadcast television and now includes mobile devices and laptops. A significant challenge in identifying video programming is the large number of consumer devices that are now associated or connected to TVs, including video game consoles where displayed video from a game being played can corrupt audience measurement systems. Video game attribution has become increasingly relevant for retargeting content delivery based on specific game sessions, linking specific commercial message viewing to video game play, creating audience segments for gamers, and the like. Summary of the Invention

[0004]

[0004] Methods and systems for media boundary detection are described herein. The method may include receiving a first cue from one or more frames of a video of media being presented by a display device, where the first cue includes a set of features derived from the one or more frames of the video, generating a first prediction of a first content type represented by the one or more frames of the video using the trained machine learning model and the first cue, receiving a second cue from one or more subsequent frames of the video being presented by the display device, where the second cue is received after the first cue, generating a second prediction of a second content type represented by the one or more frames of the video using the trained machine learning model and the second cue, determining a probability that the first content type is inconsistent with the second content type based on the first prediction and the second prediction, and performing a function of the display device based on the probability that the first content type is inconsistent with the second content type.

[0005]

[0005] A system described herein is used for media boundary detection. The system may include one or more processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform any of the methods described above.

[0006]

[0006] The non-transitory computer-readable media described in this specification may store instructions that, when executed by one or more processors, cause the one or more processors to perform any of the methods described above.

[0007]

[0007] These illustrative examples are mentioned not to limit or define this disclosure, but to provide examples to aid in its understanding. Further embodiments are discussed in the detailed description, and further description is provided therein.

[0008]

[0008] The features, embodiments, and advantages of the present disclosure will be better understood by reading the following detailed description in conjunction with the accompanying drawings. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates a block diagram of example components connected to a display device for a system in which boundary detection may be performed in accordance with aspects of the present disclosure. [Diagram 2]

[0010] FIG. 2 illustrates a block diagram of an example computing device configured to detect boundaries associated with content being presented by a display device according to aspects of the disclosure. [Figure 3A]

[0011] FIG. 3A is a graph of the Random Forest ROC for Test 1, according to an embodiment of the present disclosure. [Figure 3B]

[0012] FIG. 3B is a graph of the logistic regression ROC for Test 2, in accordance with an embodiment of the present disclosure. [Figure 3C]

[0013] FIG. 3C is a graph of the ensemble ROC for Test 1 in accordance with an embodiment of the present disclosure. [Figure 4A]

[0014] FIG. 4A illustrates a histogram graph of the distribution of distance from the mean for different video games, as illustrated in accordance with an embodiment of the present disclosure. [Figure 4B] FIG. 4B illustrates a histogram graph of the distribution of distances from the mean for different video games, as illustrated in accordance with an embodiment of the present disclosure. [Figure 4C] FIG. 4C illustrates a histogram graph of the distribution of distances from the mean for different video games, as illustrated in accordance with an embodiment of the present disclosure. [Figure 5A]

[0015] FIG. 5A illustrates an example ensemble model (smoothed) according to an embodiment of the present disclosure. [Figure 5B]FIG. 5B illustrates an exemplary rolling average model (smoothed) according to an embodiment of the present disclosure. [Figure 5C]

[0016] FIG. 5C illustrates an ensemble model ROC curve according to an embodiment of the present disclosure. [Figure 5D] FIG. 5D illustrates a rolling average model ROC curve according to an embodiment of the present disclosure. [Figure 6]

[0017] FIG. 6 illustrates a block diagram of a boundary detection system configured to detect a boundary between a first type of media and a second type of media, according to an embodiment of the disclosure. [Figure 7]

[0018] FIG. 7 illustrates a flowchart of an exemplary process for finding boundaries for linear programming, according to an embodiment of the present disclosure. [Figure 8]

[0019] FIG. 8 illustrates an example computing device architecture of an example computing device capable of implementing various techniques described herein according to aspects of the disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010]

[0020] A display device may receive media from a variety of sources, such as broadcast media (e.g., television programming over the air (OTA) or via a cable box), local or remote memory (e.g., media stored in memory, streaming media, etc.), one or more connected devices (e.g., video game consoles, devices configured to present locally stored or streaming media, digital video disc players, other media players, etc.). The display device may monitor the input sources to distinguish content being presented from an OTA source from content being presented from an input source (e.g., a High-Definition Multimedia Interface (HDMI) input, a DisplayPort input, a Universal Serial Bus (USB) input, etc.). However, the display device cannot determine what type of content is being presented from an input source or identify the particular content being presented.

[0011]

[0021] The present disclosure includes systems and methods for detecting boundaries between media types presented by a display device. The boundaries may occur when a display device switches from presenting a first type of media to presenting a second type of media. For example, the boundaries may occur when a display device switches from presenting streaming media to a video game. The methods and systems described herein may detect a first boundary when the presentation of the video game media begins and detect a second boundary when the video game media ends (e.g., when the display device switches to another input source, broadcast media, back to streaming media, when the display device is turned off, etc.). The display device may distinguish between any type of media (e.g., television shows, movies, commercials, video games, streaming media, etc.) using processes executed locally by the display device. The display device may also use local and / or remote processes to identify the particular media being presented by the display device.

[0012]

[0022] In some examples, a display device may receive a cue representing one or more frames of a video. The cue may include characteristics of the one or more frames of the video, and in some cases, characteristics of audio components corresponding to the one or more frames of the video. The display device may extract one or more sets of features from the cue and generate one or more corresponding feature vectors. The one or more features may include video features (e.g., pixel values, etc.). The display device may run a first machine learning model using the one or more feature vectors to generate a prediction corresponding to a content type of the media represented by the one or more frames of the video. Examples of content types include, but are not limited to, movies, television programs, streaming media, advertisements or commercials, video games, music (e.g., with or without a video component), etc. In some examples, the display device may generate the prediction at regular intervals, periodically, in response to detecting an event, etc.

[0013]

[0023] In some examples, each time the first machine learning model generates a prediction corresponding to a new content type (e.g., different from the content type of the immediately preceding prediction) with a confidence greater than a threshold, the display device may store a timestamp. The timestamp may be recorded along with an identification of the new content type. The display device may determine that during the time interval from the first timestamp to the second timestamp (e.g., stored when the display device was turned on or shortly thereafter), the display device was presenting content corresponding to the new content type. Each subsequent timestamp can be paired with the previous one to identify the type of content being presented between each pair of timestamps.

[0014]

[0024] In some instances, the display device may use a second machine learning model to identify an end boundary (e.g., a frame of the video where the display device switches to content corresponding to a different content type than the previous frame of the video). For example, the second machine learning model may be a rolling average classifier that analyzes the current cue and one or more previous cues to determine a probability that the current cue is derived from a frame of the video presenting the same type of content or a different type of content. The rolling average classifier may be a non-Bayesian machine learning model that defines an average from features of a set of cues received over a lookback window (e.g., a window starting at the current time and extending backwards a predetermined time interval). In some examples, an average may be defined from a set of cues. In other examples, an average may be defined for each feature of the cue (e.g., such that multiple averages may be generated). The rolling average classifier may then determine a Euclidean distance of the feature of the current cue from the average (or multiple averages). Using the Z-score (e.g., a fractional representation of the standard deviation from the mean) and the lookback window, the rolling average classifier generates a probability that the feature of the current cue indicates a reading of the end of the session. Alternatively, using the Z-score and a lookback window, the rolling average classifier may generate a probability for each feature of the current cue, which indicates the likelihood that the feature is boundary (i.e., the content presented in the current frame of the video is of a different type than the content presented in the video frame in which the previous cue was received).

[0015]

[0025] The display device may define a size of the lookback window based on one or more target accuracy metrics for the first and / or second machine learning model (e.g., accuracy, precision, area under the curve, log loss, F1 score, mean absolute error, mean squared error, etc.), characteristics of the cue, characteristics of the predictions generated by the first and / or second machine learning model, a frequency at which the cue is received by the display device, a frequency at which predictions are generated by the first and / or second machine learning model, characteristics of the video frames being analyzed, a predetermined time interval, combinations thereof, etc. In some examples, the lookback window may have a size to include a predetermined amount of the cue.

[0016]

[0026] The size of the lookback window may be changed to increase the accuracy of the predictions generated by the rolling average classifier (e.g., by increasing the length of the window to include more cues) at the expense of increasing the processing resources used to generate the predictions. The size of the lookback window may be modified to decrease the processing resources used by the display device to generate the predictions (e.g., by decreasing the length of the window to include fewer cues to analyze) at the expense of increasing the accuracy of the predictions generated by the rolling average classifier. The display device may dynamically increase or decrease the display window based on the current state of the display device, the current processing load of the display device's processing resources, the threshold accuracy of the rolling average classifier, user input, time, the time interval since the previous prediction, the time interval since the previous prediction showed a boundary, combinations thereof, and the like.

[0017]

[0027] The lookback window may be a rolling window such that with each new cue that contributes to the average (or averages), the oldest cue that contributed to the average (or averages) is removed so that it no longer contributes to the average (or averages). The rolling window reduces the sensitivity of the average (or averages) to noise or makes the average useless. For example, if the rolling window is static and calculated from cues corresponding to too many different content types, a new cue may appear to correspond to a new content type, thus introducing errors. In some examples, the average (or values) may be recalculated each time a new cue corresponds to a different content type than the previous cue.

[0018]

[0028] The display device may include a supervisor class configured to manage the operation of the rolling average classifier and process the output. For example, a session object may begin running the rolling average classifier based on initialization parameters passed to the session object. The object may output a video frame identifier corresponding to the most recent video frame processed until a HasSessionEnded function returns true (e.g., indicating that the rolling average classifier has detected a boundary corresponding to a prediction that the current frame corresponds to a different type of content than the preceding frame). The session object may retain an identifier of the last video frame corresponding to the first video frame of content corresponding to a different type.

[0019]

[0029] The identifier of the last video frame may be retrieved via the function RetrieveSessionEnd. A reset function may reinitialize the rolling average classifier using initialization parameters, allowing the same model to be used again to impute different sessions. In order to continuously detect boundaries and content types in the media being presented by the display device, a loop may be defined to cause the session object to be reanalyzed each time a boundary video frame is detected.

[0020]

[0030] The display device may perform one or more functions upon detecting a boundary (e.g., with a prediction greater than a threshold and / or when it generates two or more predictions corresponding to content of the same content type after the boundary, etc.) The one or more functions may be performed each time a boundary is detected, when the first machine learning model and / or the second machine learning model indicate that the cue represents content of a particular content type, when content is identified, when the display device generates two consecutive boundaries usable to indicate a time interval during which predicted content of a particular content type is presented, combinations thereof, and so forth.

[0021]

[0031] The one or more functions may include, but are not limited to, altering one or more settings of the display device based on the display device predicting that the content being presented corresponds to a particular content type (e.g., to improve presentation of content of a particular content type), storing or transmitting an indication of the display device's prediction that the content being presented corresponds to a particular content type, storing or transmitting a time interval during which the display device predicts that the content being presented corresponds to a particular content type, identifying the content being presented by the display device (e.g., by matching one or more cues corresponding to the content being presented against reference cues stored in a local or remote memory), combinations thereof, etc.

[0022]

[0032] In an illustrative example, a display device may receive a first cue representing one or more frames of a video of media being presented by the display device. The display device may define a set of features extracted from the first cue. The set of features may represent characteristics of one or more frames of the video and / or characteristics of audio segments corresponding to one or more frames of the video. In some instances, the display device may include a frame buffer as part of the processing pipeline of the display device. The display device may identify a particular one or more frames of the video from the frame buffer and generate the cue from the one or more frames of the video. Alternatively, the display device may receive the cue from a component of the display device (e.g., such an integrated system on chip (SOC), a device connected to the display device, a process executed by the display device, a process executed by another device connected to the display device, a remote process (e.g., executed by a server, via a cloud network or service), etc.), from a local or remote memory, and / or the like.

[0023]

[0033] The display device may run a machine learning model using the first cue. For example, the display device may define one or more feature vectors from the first cue and run the machine learning model using the one or more feature vectors as input. The machine learning model may generate a prediction of a content type (e.g., video games, television programs, movies, streaming services, advertisements or commercials, music, etc.) corresponding to the media being presented by the display device. The machine learning model may also generate a corresponding confidence value with each prediction indicating the degree to which the feature vector of the first cue corresponds to the predicted output. The confidence value may indicate the probability that the prediction is accurate. If the machine learning model generates multiple predictions for the input cue, the display device may select the prediction with the highest corresponding confidence value to be the output prediction.

[0024]

[0034] The machine learning model may be, but is not limited to, a clustering model (e.g., naive Bayes, K-means, mean shift, density-based spatial clustering, Gaussian mixture model, agglomerative hierarchical clustering, etc.), a random forest model, a rolling average classifier, a support vector machine, a naive Bayes, a logistic regression model, and an ensemble model including a logistic regression model and a random forest model, a neural network (e.g., a convolutional neural network, a recurrent neural network, a deep neural network), a deep learning network, etc. The machine learning model may be trained using supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc., depending on the particular machine learning model used, the type of output expected, and / or the target accuracy of the machine learning model.

[0025]

[0035] The machine learning model may be trained using cues received from a reference database (e.g., with labels for supervised learning or unlabeled for unsupervised learning). Alternatively, the machine learning model may be trained using cues received from media presented by the display device over time. The machine learning model may be trained (via a background process on the display device) for a predetermined time interval or until a target accuracy is reached. Once trained, the machine learning model may begin generating predictions. In some examples, the machine learning model may be trained remotely (e.g., by a server, another display device, etc.) and stored in or downloaded to the memory of the display device during manufacture.

[0026]

[0036] In some examples, the display device may perform one or more functions (as previously described) based on a particular predicted content type. For example, the display device may send a communication to a content server indicating that the display device is presenting content of a particular content type. Alternatively, or in addition, the display device may modify the display device to improve the presentation of content corresponding to the display type. For example, if the display device is presenting video game content, the display device may increase the frame rate or response time of the display device to improve the appearance of the display device and the user's ability to interact with the video game. If the display device is presenting music or streaming content, the display device may reduce the frame rate to reduce processing resource consumption.

[0027]

[0037] The display device may receive a second cue representing one or more subsequent frames of video to be displayed by the display device. The one or more subsequent frames of video may be presented at some point after the one or more video frames. For example, the display device may receive the cue periodically, at regular intervals, upon detection of an event (e.g., activation of a control of the display device, receiving an instruction from a remote control, when the display device switches to an input source such as HDMI, when the display device is presented with content from an input source, detection of a boundary, etc.), etc. When the cue is received at regular intervals (e.g., 1 second, 5 seconds, 10 seconds, etc.), the display device may receive a cue corresponding to one or more frames separated by the regular interval length.

[0028]

[0038] In some cases, the display device may modify the regular intervals (referred to as modified time intervals). For example, when an event is detected (e.g., activation of a control of the display device, receiving an instruction from a remote control, when the display device switches to an input source such as HDMI, when the display device is presented with content from an input source, boundary detection, etc.), the display device may increase the rate at which cues are received for a predetermined time interval, such that if the display device receives cues periodically every 5 seconds, in response to detecting the event, the display device may receive cues every second for the next predetermined time interval (e.g., 30 seconds, 1 minute, etc.). The regular time intervals and modified time intervals may be predetermined or selected based on user input, machine learning models, the type of event detected, the content type currently being presented, and / or the like.

[0029]

[0039] The display device may use the machine learning model and the second cue to generate a second prediction of a second content type represented by one or more frames of the video. In some cases, the display device may generate the prediction using a second machine learning model in addition to or instead of the machine learning model. The second machine learning model may be, for example, a rolling average classifier that uses simple clustering to determine whether a new cue is different from a previous cue (and thus represents content of a different content type). For example, the rolling average classifier may define an average value from a set of cues (or an average value for each feature derived from each cue in the set of cues). The rolling average classifier then determines, for each new cue, the Euclidean distance of the new cue from the mean (or multiple mean values). A probability (e.g., a confidence value, etc.) may be calculated that the new cue was obtained from a frame corresponding to content of a different content type than the frame corresponding to the cue used to derive the mean. The probability may be calculated based on the Euclidean distance or from a z-score derived from the Euclidean distance. This probability may then be used to generate a prediction of whether the new cue was obtained from a frame representing the same type of content as the previous frame or from a frame representing a different type of content than the previous frame. For example, if the probability is greater than a threshold, the rolling average classifier may predict that the new cue corresponds to a frame representing content of the new content type.

[0030]

[0040] The display device may determine a probability that the first content type does not match the second content type based on the first prediction and the second prediction. The display device may determine the probability that the first content type does not match the second content type by comparing the first prediction to the second prediction, and in some cases, comparing a confidence value corresponding to the first prediction with a confidence value corresponding to the second prediction.

[0031]

[0041] The display device may perform one or more functions based on the probability that the first content type does not match the second content type. The display device may perform any function (e.g., as described above). In some examples, the particular one or more functions performed may be based on the particular content type associated with the probability. In other examples, the display device may select one or more functions based on the particular content type associated with the probability and one or more of the following: characteristics of the machine learning model (and / or the second machine learning model), a prediction (or confidence or probability) generated by the machine learning model (and / or the second machine learning model), characteristics of the display device (e.g., a currently selected input source, input from a remote control, user input, display device settings, etc.), an identification of a previous content type presented by the display device, historical content types presented by the display device, etc.

[0032]

[0042] In some instances, the display device may store a timestamp corresponding to each prediction corresponding to a new content type. The display device may then determine the time interval during which the content type was presented by the display device. The display device may send a report to the content server indicating one or more content types presented by the display device (e.g., the current content type and / or one or more immediately preceding content types, historical content types, etc.) along with the time interval corresponding to each content type.

[0033]

[0043] FIG. 1 illustrates a block diagram of exemplary components connected to a display device for a system in which boundary detection may be performed, according to an embodiment of the present disclosure. The display device 104 may be a device configured to display media (e.g., audio and / or video) from a connected device, from the Internet (e.g., streaming media, downloaded media, etc.), and / or the like. The display device may be a device configured to display visual and / or auditory media from one or more sources. For example, the display device may be a television, a smart television, a computer monitor, etc. Examples of components that may provide media to the display device 104 include, but are not limited to, broadcast media 108 (e.g., a digital antenna, a cable box, etc.), a game console 112 (e.g., a computing device configured to execute a video game application that may be presented in whole or in part by the display device 104). A user may interact with the game console 112 using one or more game controllers 116, an Internet set-top 120 (e.g., a device configured to receive content from one or more remote networks, such as the Internet, and present the content using the display device 104), etc.

[0034]

[0044] The components may be connected to the display device 104 through a wired connection (e.g., via an HDMI cable, optical cable, DisplayPort cable, USB cable, coaxial cable, digital antenna, etc.) or a wireless connection (e.g., Wi-Fi, Bluetooth, etc.). In some instances, the components may be applications (e.g., streaming applications, etc.) executed by the display device 104. The display device 104 may detect components connected to the display device 104 by detecting a signal through a particular input port. The display device 104 may determine when media to be presented is received through a particular input port. The display device 104 may perform a boundary detection process to determine a content type corresponding to the media being presented. As a result, the display device may determine when the display device 104 is presenting a television program, a movie, streaming content, a video game, etc. In some examples, the display device 104 may identify the media being presented in response to determining the content type of the media (e.g., using an automatic content recognition system, etc.).

[0035]

[0045] 2 illustrates a block diagram of an exemplary computing device configured to detect boundaries associated with content being presented by a display device according to aspects of the disclosure. The display device 204 may include one or more processing components (e.g., a system on a chip, a central processing unit, an application specific integrated circuit, a field programmable gate array, etc.), memory (e.g., volatile and non-volatile memory, a database, etc.), a network processor (e.g., including a Wi-Fi transceiver, a Bluetooth transceiver, and / or other transceivers, etc.), and one or more sensors.

[0036]

[0046] The display device 204 may be configured to present media to one or more users using the display 208 and / or one or more wireless devices (e.g., other display devices, mobile devices, tablets, etc.) connected via a network processor. The computing device 204 may retrieve media from a media database 252 (or alternatively receive media from one or more broadcast sources, remote sources via a network processor, external devices, etc.). The media may be loaded by a media player 248, which may process the media based on the video's container (e.g., MPEG-4, QuickTime Movie, Wavefile Audio File Format, Audio Video Interleave, etc.). The media player 248 may pass the media to a video decoder 244, which decodes the video into a sequence of video frames that may be displayed by the display 208. The sequence of video frames may be passed to the video frame processor 240 in preparation for display. Alternatively, the media may be generated by an interactive service operating within the app manager 236. The app manager 236 may pass the sequence of frames generated by the interactive service to the video frame processor 240.

[0037]

[0047] The sequence of video frames may be passed to a system on chip (SOC") 212. The SOC 212 may include processing components configured to enable presentation of the sequence of video and / or audio components. The SOC 212 may include a central processing unit (CPU) 224, a graphics processing unit (GPU) 220, volatile memory (e.g., random access memory), non-volatile memory (e.g., flash, etc.), input / output interfaces (e.g., collectively, volatile memory, non-volatile memory, and input / output interfaces correspond to block 228), an artificial intelligence processor 232 (e.g., including one or more machine learning models, training data sets, feature extractors, etc.), and a video frame buffer 216.

[0038]

[0048] The SOC 212 may generate cues from one or more video frames stored in the video frame buffer 216 before or as the one or more video frames are presented by the video display 208. The cues may be generated from one or more pixel arrays (also referred to as pixel patches) of the video frames. The pixel patches may be any shape or pattern, such as, but not limited to, a y×z pixel array that includes y pixels horizontally and z pixels vertically from the video frames. The pixels may include color values, such as red, green, and blue values. For example, the pixel 306 is shown as having a red-green-blue (RGB) color value. The color value of a pixel may be represented by an 8-bit binary value for each color. Other suitable color values ​​that may be used to represent the color of a pixel include luma and chroma (also referred to as Y, Cb, Cr, YUV) values ​​or any other suitable color value.

[0039]

[0049] The display device may derive an average value for each pixel patch. The average value may be a 24-bit data record representing the pixel patch. The display device may generate the cue by aggregating the average value for each pixel patch and adding a timestamp corresponding to the frame from which the pixel patch was obtained. The timestamp may correspond to an epoch time (e.g., which may represent a total elapsed time in fractions of a second since midnight on January 1, 1970), a predetermined start time, an offset time (e.g., from the beginning of the media being presented, or when the display device was powered on, etc.), etc. The cue may also include metadata, which may include any information about the media being presented, such as a program identifier, a program time, a program length, or any other information (if known).

[0040]

[0050] In some examples, the cue may be derived from any number of pixel patches obtained from a single video frame. Increasing the amount of pixel patches included in the queue increases the data size of the queue, which may increase the processing load of the display device and one or more cloud networks that may operate to identify the content. For example, a cue derived from 25 pixel patches may correspond to 600 bits of data (24 bits per pixel patch x 25 pixel patches), not including a timestamp and any metadata. Increasing the amount of video patches obtained from a video frame may increase the accuracy of boundary detection and content identification at the expense of increasing the processing load. Reducing the amount of video patches obtained from a video frame may decrease the accuracy of boundary detection and content identification, while also decreasing the processing load of the display device. The display device may dynamically determine whether to generate the cue using more or fewer pixel patches based on the target accuracy and / or processing load of the display device.

[0041]

[0051] The display device 204 may pass the cue to a machine learning model of the AI ​​processor 232. The machine learning model may generate a prediction indicating whether the frame from which the cue was obtained represents a different type of content than one or more preceding cues (e.g., the immediately preceding cue, an average generated from a set of previous cues, etc.). The display device 204 may perform one or more actions based on the prediction. For example, the display device 204 may modify the operation of the CPU 224 and / or the GPU 220 to improve the presentation of media of the predicted content type and / or to reduce resource consumption of the video display device.

[0042]

[0052] In some instances, the display device 204 may send the cue to a server configured to match the cue to a reference cue stored in a reference database. The reference cue may be associated with an identified television program, movie, video game, song, etc. If the server identifies a matching reference cue, the identifier associated with the matching reference cue may correspond to the media being presented by the display device 204.

[0043]

[0053] In some instances, detection of non-broadcast programming (e.g., content from other OTA television, cable, etc.) may be used to reduce the processing load on the server. For example, if the AI ​​processor 232 predicts that the current cue corresponds to a non-broadcast program, the display device 104 may send a communication to the server informing the server that a non-broadcast program is being presented by the display device 104. The server may instruct the display device 104 to temporarily reduce the amount of cues generated during a predetermined interval (e.g., reduce the frequency of cues from 10 per second to 1 every 30 seconds, etc.). Because the server will receive fewer cues for matching, the processing load on the server may be reduced while maintaining content identification operations. If the AI ​​processor 232 predicts that the new cue corresponds to a broadcast program, the display device 104 may notify the server and return to generating cues at the original frequency. Regardless of whether the AI ​​processor 232 predicted that the cue corresponds to a broadcast program, once the predetermined time interval has elapsed, the display device 104 may return to generating cues at the original frequency. If the AI ​​processor 232 later predicts that the cue corresponds to a non-broadcast program, the process may occur again. As a result, the display device 104 may reduce the processing load on the server when presenting non-broadcast media (such as, for example, a video game) for which the server does not have a reference cue for identification purposes or may be instructed to ignore.

[0044]

[0054] FIG. 3A illustrates an example Random Forest ROC graph for Test 1, according to an embodiment of the present disclosure. FIG. 3A depicts a graph representing the accuracy metric of a Random Forest machine learning model, where the false positive rate is graphed against the false positive rate. The area under the curve (AUC) represents an aggregate measure of the performance across all possible classification thresholds of the machine learning model. Thus, the AUC is an indication of the accuracy of the Random Forest machine learning model in predicting that a feature vector derived from cues obtained from a video frame represents content of a particular content type. The AUC of the Random Forest machine learning model for this test is .84.

[0045]

[0055] 3B illustrates an example logistic regression ROC graph for Test 2, according to an embodiment of the present disclosure. FIG. 3B illustrates a graph representing an accuracy metric for a logistic regression machine learning model. The AUC represents the accuracy of the logistic regression machine learning model in predicting that a feature vector derived from cues obtained from a video frame represents content of a particular content type. The AUC for the logistic regression machine learning model for this test is .68.

[0046]

[0056] FIG. 3C shows an example ensemble ROC graph for Test 1 according to an embodiment of the disclosure. FIG. 3B depicts a graph representing the accuracy metric of an ensemble machine learning model including a random forest model and a logistic regression model. The AUC represents the accuracy of the ensemble machine learning model in predicting that a feature vector derived from a cue obtained from a video frame represents content of a particular content type. The AUC of the ensemble machine learning model for this test is .90, indicating that for the test set, the ensemble model was able to predict the content type associated with the cue better than the random forest and logistic regression models alone.

[0047]

[0057] 4A-4C show histogram graphs of the distribution of distances from the mean for different video games, as shown in accordance with an embodiment of the present disclosure. The rolling average model machine learning model generates probabilities based on the distance of a cue (or its features) from the mean. The distribution of distances from the mean indicates the sensitivity of the rolling average model to detect that a new cue corresponds to a frame representing a particular video game (e.g., example video game 1 as shown in FIG. 4A, example video game 2 as shown in FIG. 4B, or example video game 3 as shown in FIG. 4C).

[0048]

[0058] 5A and 5B show exemplary ensemble models (smoothed) and rolling average models (smoothed), respectively, according to an embodiment of the present disclosure, and FIG. 5C and FIG. 5D show ensemble model ROC curves and rolling average model ROC curves, respectively. The ensemble model (shown in FIG. 5A) compared to the rolling average model (FIG. 5B) shows that both models perform well when predicting content type. In some examples, the ensemble model has a higher general accuracy. For example, the ensemble model has an AUC of .98 (as shown by the AUC in FIG. 5C), while the rolling average model has an AUC of .94 (as shown by the AUC in FIG. 5D). The rolling average model compensates for its slightly lower accuracy by being able to detect changes in content type faster than the ensemble model. The display device may switch between the ensemble model and the rolling average model based on a calculated tradeoff between accuracy and speed.

[0049]

[0059] FIG. 6 illustrates a block diagram of a boundary detection system configured to detect a boundary between a first type of media and a second type of media, according to an embodiment of the disclosure. The boundary detection system may output a notification indicating the boundary (e.g., a frame representing a different type of content than the immediately preceding frame). The boundary detection system may be a component of or operate in conjunction with an automatic content recognition (ACR) system of the display device. The client ACR cue generator 604 may generate cues from video frames stored in a video buffer. The client ACR cue generator 604 may generate cues at a default frequency (e.g., 10 cues per second, etc.). The default frequency may be determined by the ACR system, the display device, a machine learning model (as described above), user input, etc.

[0050]

[0060] The cue (eg, corresponding to unknown media) may be passed to the ACR process 608 . The ACR process 608 can determine whether the unknown cue matches a reference cue in a reference database. Each reference cue may be associated with an identifier that corresponds to a known media segment. If a match is found, the identifier of the matching reference cue can be attributed to the unknown cue (so that the unknown cue becomes a known cue).

[0051]

[0061] The boundary detection 616 may receive cues from the client ACR cue generator 604 and store coefficients derived from the cues in an audio fingerprint cache 620 that stores coefficients corresponding to the audio component of the cue, a video fingerprint cache 624 that stores coefficients corresponding to the video component of the cue, and / or an infrared (IR) command cache 628 that stores an indication of input from an infrared remote controller. In some examples, the IR command cache 628 may store an indication of input from another remote controller (e.g., Bluetooth, wired controller, etc.). The IR command cache 628 may generate an event when the input source of the display device is changed (e.g., from a coaxial input to an HDMI input, etc.), which may indicate a change in the type of content being presented by the display device. The audio fingerprint cache 620 stores a predetermined amount of audio coefficients from each of the one or more cues. In some examples, the audio fingerprint cache 620 may store 35 coefficients for each audio cue. The video fingerprint cache 624 may store a predetermined amount of video coefficients from each of the one or more cues. In some examples, the video fingerprint cache 624 may store a larger amount of video coefficients for each cue than the audio fingerprint cache 620. In those examples, the video fingerprint cache 624 may store 75 coefficients per cue and the audio fingerprint cache 620 may store 35. Per queue factor.

[0052]

[0062] The coefficients for audio and video may be processed together or separately. When processed together, the coefficients are aggregated together to form a feature vector that is passed as an input in the machine learning model (as described above). When processed separately, the video coefficients may form a first feature vector and the audio coefficients may form a second feature vector. The machine learning model may be trained to generate predictions according to a particular media type (e.g., video or audio). The display device may calibrate the machine learning model to generate predictions from a given media type (e.g., length of lookback window, threshold for prediction, learning period to approximate clusters or averages, etc.).

[0053]

[0063] Boundary detection 616 may determine that the current video frame corresponds to a particular type of content (e.g., non-broadcast programming, such as video games, etc.). Because the ACR system is configured to identify broadcast sources, the ACR system may not include cues corresponding to non-broadcast sources. If the current content being displayed corresponds to a non-broadcast program, the ACR system may be paused or set to a reduced execution mode to avoid consuming processing resources attempting to identify a match for an unknown cue that has no match. Once a boundary is detected, boundary detection 616 may send a notification to ACR client TV ACR enable / disable 612 to disable ACR processing 608. By disabling ACR processing 608, client ACR cue generator 604 generates cues at a reduced frequency. Returning to the previous example, at a reduced frequency, client ACR cue generator 604 may generate one cue every 30 seconds. Client ACR processing 608 may continue to operate in reduced execution mode for a predetermined time interval or until a frame corresponding to a broadcast program is identified. For example, if the ACR processing 608 identifies an unknown cue by matching the unknown cue to a known reference cue while operating in a reduced execution mode, the client ACR cue generator 604 may revert to generating cues at a default frequency and may disable boundary detection for a predetermined interval before continuing operation.

[0054]

[0064] FIG. 7 illustrates a flowchart of an example process for detecting boundaries for linear programming according to an aspect of the disclosure. At block 704, the display device may receive a first cue representing one or more video frames of media being presented by the display device. The first cue may include a set of features derived from the one or more video frames. The set of features may represent characteristics of the one or more frames of video and / or characteristics of audio segments corresponding to the one or more frames of video. In some instances, the display device may include a frame buffer as part of the processing pipeline of the display device. The display device may identify a particular one or more frames of video from the frame buffer and generate the cue from the one or more frames of video. Alternatively, the display device may receive the cue from a component of the display device (e.g., such an integrated system on chip (SOC), a device connected to the display device, a process executed by the display device, a process executed by another device connected to the display device, a remote process (e.g., executed by a server, via a cloud network or service), etc.), from a local or remote memory, and / or the like.

[0055]

[0065] At block 708, the display device may use the trained machine learning model and the first cue to generate a first prediction of a first content type represented by one or more frames of the video. For example, the display device may define one or more feature vectors from the first cue and run the machine learning model using the one or more feature vectors as inputs. The machine learning model may generate a prediction of a content type (e.g., video games, television programs, movies, streaming services, advertisements or commercials, music, etc.) corresponding to the media being presented by the display device. The machine learning model may also use each prediction to generate a corresponding confidence value indicating the degree to which the feature vector of the first cue corresponds to the predicted output. The confidence value may indicate the accuracy of the prediction. If the machine learning model generates multiple predictions for a given input cue, the display device may select the prediction with the highest corresponding confidence value to be the output prediction.

[0056]

[0066] The machine learning model may be, but is not limited to, a clustering model (e.g., naive Bayes, K-means, mean shift, density-based spatial clustering, Gaussian mixture model, agglomerative hierarchical clustering, etc.), a random forest model, a rolling average classifier, a support vector machine, a naive Bayes, a logistic regression model, and an ensemble model including a logistic regression model and a random forest model, a neural network (e.g., a convolutional neural network, a recurrent neural network, a deep neural network), a deep learning network, etc. The machine learning model may be trained using supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc., depending on the particular machine learning model used, the type of output expected, and / or the target accuracy of the machine learning model.

[0057]

[0067] In some examples, the display device may perform one or more functions (as previously described) based on the predicted content type. For example, the display device may send a communication to a content server indicating that the display device is presenting content of a particular content type. Alternatively, or in addition, the display device may modify the display device to improve the presentation of content corresponding to the display type. For example, if the display device is presenting video game content, the display device may increase the frame rate or response time of the display device to improve the appearance of the display device and the user's ability to interact with the video game. If the display device is presenting music or streaming content, the display device may reduce the frame rate to reduce processing resource consumption, etc.

[0058]

[0068] At block 712, the display device may receive a second cue representing one or more subsequent video frames to be displayed by the display device. The one or more video frames may be presented at some point after the one or more video frames. For example, the display device may receive the cue periodically, at regular intervals, upon detection of an event (e.g., activation of a control of the display device, receiving an instruction from a remote control, when the display device switches to an input source such as HDMI, when the display device is presented with content from an input source, boundary detection, etc.).

[0059]

[0069] In some cases, the display device may modify the regular intervals (referred to as modified time intervals). For example, when an event is detected (e.g., activation of a control of the display device, receiving an instruction from a remote control, when the display device switches to an input source such as HDMI, when the display device presents content from an input source, boundary detection, etc.), the display device may increase the rate at which cues are received in a predefined time interval, such that if the display device receives 10 cues per second periodically, in response to the detection of the event, the display device may receive 24 cues per second in the next predefined time interval (e.g., 30 seconds, 1 minute, etc.). The regular time intervals and modified time intervals may be predetermined or selected based on user input, machine learning models, the type of event detected, the content type currently being presented, and / or the like.

[0060]

[0070] At block 716, the display device may use the machine learning model and the second cue to generate a second prediction of a second content type represented by the one or more video frames. In some instances, the display device may generate a prediction using a second machine learning model in addition to or instead of the machine learning model. The second machine learning model may be, for example, a rolling average classifier that uses simple clustering to determine whether a new cue is different from a previous cue (and thus represents content of a different content type). For example, the rolling average classifier may define a mean value from a set of cues (or a mean value for each feature derived from each cue in the set of cues). The rolling average classifier then determines, for each new cue, a Euclidean distance from the mean value (or multiple mean values) of the new cue. A probability (e.g., a confidence value, etc.) may be calculated that the new cue was obtained from a frame corresponding to content of a different content type than the frames used to derive the mean value. The probability may be calculated based on the Euclidean distance or from a z-score derived from the Euclidean distance. This probability can then be used to generate a prediction of whether the new cue is taken from a frame representing the same type of content as the previous frame, or from a frame representing a different type of content than the previous frame. For example, if the probability is greater than a threshold, the rolling average classifier may predict that the new cue corresponds to a frame representing content of the new content type.

[0061]

[0071] At block 720, the display device may determine a probability that the first content type does not match the second content type based on the first prediction and the second prediction, for example, by comparing the first prediction with the second prediction and, in some cases, comparing a confidence value corresponding to the first prediction with a confidence value corresponding to the second prediction.

[0062]

[0072] At block 724, the display device may perform one or more functions based on the probability that the first content type does not match the second content type. The display device may perform any function (e.g., as described above). In some examples, the particular one or more functions performed may be based on the particular content type associated with the probability. In other examples, the display device may select one or more functions based on the particular content type associated with the probability and one or more of characteristics of the machine learning model (and / or the second machine learning model), a prediction (or confidence or probability) generated by the machine learning model (and / or the second machine learning model), characteristics of the display device (e.g., a currently selected input source, input from a remote control, user input, display device settings, etc.), an identification of a previous content type presented by the display device, historical content types presented by the display device, etc.

[0063]

[0073] For example, functions performed by the display device in response to detecting a single boundary (e.g., block 708, etc.) or two boundaries (e.g., blocks 708 and 716, etc.) may include modifying the operation of an ACR system of the display device. The ACR system includes some processes (e.g., cue generation, etc.) that run locally on the display device and some processes (e.g., cue matching, etc.) that run remotely. If the display device determines that the content being displayed is of a particular type (e.g., non-broadcast programming such as video games), the display device may temporarily modify the frequency at which cues are generated by the ACR system. Modifying the frequency may reduce the amount of cues generated over a unit time interval, which may reduce the processing resources needed to process the cues to determine if there is a match. Reducing the frequency at which cues are generated reduces the processing resources of both the display device and the server.

[0064]

[0074] In some instances, the display device may store a timestamp corresponding to each prediction corresponding to a new content type. The display device may then determine the time interval during which the content type was presented by the display device. The display device may send a report to the content server indicating one or more content types presented by the display device (e.g., the current content type and / or one or more immediately preceding content types, historical content types, etc.) along with the time interval corresponding to each content type.

[0065]

[0075] The process of Figure 7 may be executed locally on the processing hardware of a display device (e.g., an example of which is shown in Figure 2). The process may be executed once, multiple times, continuously, or in a similar manner. For example, the process of Figure 1 may begin executing when the display device is powered on, and continue to execute and record boundaries until the display device is powered off.

[0066]

[0076] FIG. 8 illustrates an exemplary computing device according to an aspect of the disclosure. For example, the computing device 800 can implement any of the systems or methods described herein. In some examples, the computing device 800 can be a component of or included within a media device. The components of the computing device 800 are shown to communicate electrically with each other using a connection 806, such as a bus. The exemplary computing device architecture 800 includes a processor (e.g., a CPU, processor, etc.) 804 and a connection 806 (e.g., a bus, etc.) configured to couple components of the computing device 800, such as, but not limited to, a memory 820, a read-only memory (ROM) 818, a random access memory (RAM) 816, and / or a storage device 808 to a processing unit 810.

[0067]

[0077] The computing device 800 may include a cache 802 of high-speed memory directly connected to, adjacent to, or integrated within the processor 804. The computing device 800 may copy data from the memory 820 and / or storage device 808 to the cache 802 for quick access by the processor 804. In this manner, the cache 802 may provide a performance boost that avoids delays while waiting for the processor 804 to data. Alternatively, the processor 804 may access data directly from the memory 820, ROM 817, RAM 816, and / or storage device 808. The memory 820 may include multiple types of homogenous or heterogeneous memory (e.g., but not limited to, magnetic, optical, solid state, etc.).

[0068]

[0078] The storage device 808 may include one or more non-transitory computer readable media, such as volatile and / or non-volatile memory. The non-transitory computer readable media may store instructions and / or data accessible by the computing device 800. The non-transitory computer readable media may include, but are not limited to, a magnetic cassette, a hard disk drive (HDD), a flash memory, a solid-state memory device, a digital versatile disk, a cartridge, a compact disk, a random access memory (RAM) 825, a read-only memory (ROM) 820, combinations thereof, and the like.

[0069]

[0079] The storage device 808 may store one or more services, such as service 1 810, service 2 812, and service 3 814, that are executable by the processor 804 and / or other electronic hardware. The one or more services include instructions executable by the processor 804 to perform operations, such as any of the techniques, steps, processes, blocks, and / or operations described herein, to control the operation of devices in communication with the computing device 800, to control the operation of the processing unit 810 and / or any dedicated processors, combinations thereof, etc. The processor 804 may be a system on a chip (SOC) that includes one or more cores or processors, buses, memory, clocks, memory controllers, caches, other processor components, and / or the like. Multi-core processors may be symmetric or asymmetric.

[0070]

[0080] The computing device 800 may include one or more input devices 822, which may represent any number of input mechanisms, such as a microphone, a touch-sensitive screen for graphical input, a keyboard, a mouse, motion input, speech, a media device, a sensor, combinations thereof, etc. The computing device 800 may include one or more output devices 824 that output data to a user. Such output devices 824 may include, but are not limited to, a media device, a projector, a television, a speaker, combinations thereof, etc. In some instances, a multimodal system may enable a user to provide multiple types of input / output for communicating with the computing system 800. The communication interface 826 may be configured to manage user input and computing device output. The communication interface 826 may also be configured to manage communications (e.g., establish connections, receive / send communications, etc.) with remote devices via one or more communication protocols and / or via one or more communication mediums (e.g., wired, wireless, etc.).

[0071]

[0081] Computing device 800 is not limited to the components shown in Figure 8. Computing device 800 may include other components not shown and / or components shown may be omitted.

[0072]

[0082] The following examples illustrate various aspects of the present disclosure. As used hereinafter, any reference to a series of examples should be understood as a reference to each of those examples disjunctively (e.g., "Examples 1-4" should be understood as "Examples 1, 2, 3, or 4").

[0073]

[0083] Example 1 is a method comprising receiving a first cue from one or more frames of a video of media being presented by a display device, where the first cue includes a set of features derived from the one or more frames of the video, generating a first prediction of a first content type represented by the one or more frames of the video using a trained machine learning model and the first cue; receiving a second cue from one or more subsequent frames of the video being presented by the display device, where the second cue is received after the first cue, generating a second prediction of a second content type represented by the one or more frames of the video using the trained machine learning model and the second cue; determining a probability that the first content type is inconsistent with the second content type based on the first prediction and the second prediction; and performing a function of the display device based on the probability that the first content type is inconsistent with the second content type.

[0074]

[0084] Example 2 is the method of example 1, wherein one of the first and second content types corresponds to a video game.

[0075]

[0085] Example 3 is the method of any of Examples 1-2, wherein receiving a first cue from one or more frames of a video of the media displayed by the display device includes identifying one or more sets of pixels from the one or more frames of the video of the media and extracting one or more features corresponding to pixel values ​​from each of the one or more sets of pixels.

[0076]

[0086] Example 4 is the method of any of Examples 1-3, wherein the machine learning model is an ensemble model derived from two or more machine learning models.

[0077]

[0087] Example 5 is the method of any of Examples 1-4, where performing the functionality of the display device includes facilitating an identification of the first content type and transmission to the server including a duration for which the first content type is to be presented by the display device.

[0078]

[0088] Example 6 is the method of any of Examples 1-5, wherein generating a first prediction of a first content type represented by one or more frames of the video includes identifying media corresponding to the first content type.

[0079]

[0089] Example 7 is the method of any of Examples 1-6, further comprising modifying video display or audio settings of the display device based on the first prediction and the first content type to improve presentation of media corresponding to the first content type.

[0080]

[0090] Example 8 is a system comprising one or more processors and a machine-readable storage medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including receiving a first cue from one or more frames of a video of media being presented by a display device, where the first cue includes a set of features derived from the one or more frames of the video, generating a first prediction of a first content type represented by the one or more frames of the video using the trained machine learning model and the first cue; receiving a second cue from one or more subsequent frames of the video being presented by the display device, where the second cue is received after the first cue, generating a second prediction of a second content type represented by the one or more frames of the video using the trained machine learning model and the second cue; determining a probability that the first content type is inconsistent with the second content type based on the first prediction and the second prediction; and performing a function of the display device based on the probability that the first content type is inconsistent with the second content type.

[0081]

[0091] Example 9 is the system of example 8, wherein one of the first and second content types corresponds to a video game.

[0082]

[0092] Example 10 is the system of any of Examples 8-9, wherein receiving a first cue from one or more frames of a video of the media displayed by the display device includes identifying one or more sets of pixels from the frames of the video of the one or more frames of the video, and extracting one or more features corresponding to pixel values ​​from each of the one or more sets of pixels.

[0083]

[0093] Example 11 is the system of any of Examples 8-10, wherein the machine learning model is an ensemble model derived from two or more machine learning models.

[0084]

[0094] Example 12 is the system of any of Examples 8-11, where performing the functionality of the display device includes facilitating an identification of the first content type and transmission to the server including a duration for which the first content type is presented by the display device.

[0085]

[0095] Example 13 is the system of any of Examples 8-12, wherein generating a first prediction of a first content type represented by one or more frames of the video includes identifying media corresponding to the first content type.

[0086]

[0096] Example 14 is the system of any of Examples 8-13, where the operations further include modifying video display or audio settings of the display device based on the first prediction and the first content type to improve presentation of media corresponding to the first content type.

[0087]

[0097] Example 15 is one or more non-transitory machine-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including receiving a first cue from one or more frames of a video of media being presented by a display device, where the first cue includes a set of features derived from the one or more frames of the video, generating a first prediction of a first content type represented by the one or more frames of the video using the trained machine learning model and the first cue; receiving a second cue from one or more subsequent frames of the video being presented by the display device, where the second cue is received after the first cue, generating a second prediction of a second content type represented by the one or more frames of the video using the trained machine learning model and the second cue; determining a probability that the first content type is inconsistent with the second content type based on the first prediction and the second prediction; and performing a function of the display device based on the probability that the first content type is inconsistent with the second content type.

[0088]

[0098] Example 16 is the non-transitory machine-readable storage medium of example 15, wherein one of the first content type and the second content type corresponds to a video game.

[0089]

[0099] Example 17 is the non-transitory machine-readable storage medium of any of Examples 15-16, wherein receiving a first cue from one or more frames of a video of the media being displayed by the display device includes identifying one or more sets of pixels from the frame of video of the one or more frames of the video, and extracting one or more features corresponding to pixel values ​​from each of the one or more sets of pixels.

[0090]

[0100] Example 18 is the non-transitory machine-readable storage medium of any of Examples 15-17, wherein the machine learning model is an ensemble model derived from two or more machine learning models.

[0091]

[0101] Example 19 is the non-transitory machine-readable storage medium of any of Examples 15-18, wherein performing the functionality of the display device includes facilitating transmission to the server including an identification of the first content type and a duration for which the first content type is presented by the display device.

[0092]

[0102] Example 20 is the non-transitory machine-readable storage medium of any of Examples 15-19, wherein generating a first prediction of a first content type represented by one or more frames of the video includes identifying media corresponding to the first content type.

[0093]

[0103] Example 21 is the non-transitory machine-readable storage medium of any of Examples 15-20, wherein the operations further include modifying video display or audio settings of the display device based on the first prediction and the first content type to improve presentation of the media corresponding to the first content type.

[0094]

[0104] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or transporting instruction(s) and / or data. Computer-readable media may include non-transitory media in which data may be stored in a form that does not include a carrier wave and / or a transitory electronic signal. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory or memory devices. A computer-readable medium may store code and / or machine-executable instructions, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, or the like.

[0095]

[0105] Some portions of this description describe examples in terms of algorithms and symbolic representations of operations on information. These operations, while described functionally, computationally, or logically, may be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, arrangements of operations may be referred to as modules without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0096]

[0106] Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In some examples, a software module is implemented in a computer-readable medium that stores computer program code, which can be executed by a computer processor to perform any or all of the steps, operations, or processes described.

[0097]

[0107] Some examples may relate to an apparatus or system for performing any or all of the described steps, operations, or processes. The apparatus or system may be specially constructed for the required purposes and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a memory of the computing device. The memory may be or may include a non-transitory tangible computer-readable storage medium, which may be coupled to a bus, or any type of medium suitable for storing electronic instructions. Additionally, any computing system referred to herein may include a single processor or multiple processors.

[0098]

[0108] Although the subject matter has been described in detail with respect to certain examples, it will be recognized that, upon understanding the foregoing, one skilled in the art may readily create alternatives, modifications, and equivalents to such embodiments. Many specific details have been described herein to provide a thorough understanding of the claimed subject matter. However, one skilled in the art will understand that the claimed subject matter may be practiced without these specific details. In other instances, methods, apparatus, or systems that would be known by one skilled in the art have not been described in detail so as not to obscure the claimed subject matter. As a result, it is to be understood that the present disclosure is presented for purposes of illustration rather than limitation, and does not preclude inclusion of such modifications, variations, and / or additions to the subject matter as would be readily apparent to one skilled in the art.

[0099]

[0109] For clarity of explanation, in some cases, the disclosure may be presented as including individual functional blocks, including devices, device components, steps or routines in a method implemented in software, or functional blocks comprising a combination of hardware and software. Additional functional blocks other than those shown in the drawings and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0100]

[0110] Particular examples may be described herein as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps that are not shown. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0101]

[0111] The processes and methods according to the examples described above can be implemented using computer-executable instructions stored on or otherwise available from a computer-readable medium. Such instructions can include, for example, instructions and data that cause or otherwise configure a general purpose computer, a special purpose computer, or a processing device to perform a certain function or group of functions. A portion of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc.

[0102]

[0112] Devices implementing the methods and systems described herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) to perform the necessary tasks may be stored in a computer-readable or machine-readable medium. The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated circuits or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, a conventional processor, a controller, a microcontroller, a state machine, or a combination thereof. A processor may also be implemented as a combination of computing components (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors coupled to a DSP core, or any other such configuration). Thus, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein. The functionality described herein may also be embodied in a peripheral device or an add-in card. Such functionality may also be implemented across different chips on a circuit board, or across different processes running on a single device, as further examples.

[0103]

[0113] In the foregoing description, aspects of the present application have been described with reference to specific examples thereof, but those skilled in the art will recognize that the present application is not limited thereto. From this, it should be understood that although exemplary embodiments of the present disclosure have been described in detail herein, the inventive concept may be embodied and used in various other ways, and the appended claims are intended to be construed to include such variations except as limited by the prior art. The various features and aspects of the present disclosure described above may be used individually or in any combination. Moreover, the examples may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the present disclosure. The present disclosure and drawings are therefore to be considered illustrative and not restrictive.

[0104]

[0114] The various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

[0105]

[0115] Unless otherwise indicated, throughout this specification, discussions utilizing terms such as "processing," "computing," "calculating," "determining," and "identifying" are understood to refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities in the memory, registers, or other information storage, transmission, or media devices of a computing platform. As used herein, the terms "adapted to" or "configured to" are meant to be open and inclusive language that does not exclude devices that are adapted to or configured to perform additional tasks or steps. Additionally, use of the term "based on" is meant to be open and inclusive in that a process, step, calculation, or other action "based on" one or more recited conditions or values ​​may in fact be based on additional conditions or values ​​other than those recited. The headings, lists, and numbering contained herein are for ease of explanation only and are not meant to be limiting.

[0106]

[0116] The foregoing detailed description of the technology has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the technology to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The described embodiments were chosen to best explain the principles of the technology, its practical application, and to enable others skilled in the art to utilize the technology in various embodiments and with various modifications suitable for the particular use contemplated. It is intended that the scope of the technology be defined by the claims.

Claims

1. 1. A method comprising: receiving a first cue from one or more frames of a video of media being presented by a display device, where the first cue includes a set of features derived from the one or more frames of the video; generating a first prediction of a first content type represented by one or more frames of the video using a trained machine learning model and the first cue; receiving a second cue from one or more subsequent frames of video being presented by the display device, where the second cue is received after the first cue; generating a second prediction of a second content type represented by one or more frames of the video using the trained machine learning model and the second cue; and determining a probability that the first content type is inconsistent with the second content type based on the first prediction and the second prediction; and performing a function of the display device based on the probability that the first content type does not match the second content type.

2. The method of claim 1 , wherein one of the first and second content types corresponds to a video game.

3. Receiving the first cue from one or more frames of a video of a media being displayed by the display device includes: identifying one or more sets of pixels from one or more frames of the video; and extracting one or more features corresponding to pixel values ​​from each of the one or more sets of pixels.

4. The method of claim 1 , wherein the machine learning model is an ensemble model derived from two or more machine learning models.

5. Performing the functions of the display device includes:

10. The method of claim 1, further comprising facilitating transmission to a server including an identification of the first content type and a duration that the first content type is to be presented by the display device.

6. Generating the first prediction of the first content type represented by one or more frames of the video includes: The method of claim 1 , further comprising identifying the media that corresponds to the first content type.

7. 10. The method of claim 1 , further comprising modifying video display or audio settings of the display device based on the first prediction and the first content type to improve presentation of media corresponding to the first content type.

8. 1. A system comprising: one or more processors; and a machine-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: receiving a first cue from one or more frames of a video of media being presented by a display device, where the first cue includes a set of features derived from the one or more frames of the video; generating a first prediction of a first content type represented by one or more frames of the video using a trained machine learning model and the first cue; receiving a second cue from one or more subsequent frames of video being presented by the display device, where the second cue is received after the first cue; generating a second prediction of a second content type represented by one or more frames of the video using the trained machine learning model and the second cue; and determining a probability that the first content type is inconsistent with the second content type based on the first prediction and the second prediction; and performing an operation of the display device based on the probability that the first content type does not match the second content type.

9. The system of claim 8 , wherein one of the first and second content types corresponds to a video game.

10. Receiving the first cue from one or more frames of a video of a media being displayed by the display device includes: identifying one or more sets of pixels from a frame of video among the one or more frames of the video; and extracting one or more features corresponding to pixel values ​​from each of the one or more sets of pixels.

11. The system of claim 8 , wherein the machine learning model is an ensemble model derived from two or more machine learning models.

12. Performing the functions of the display device includes:

10. The system of claim 8, further comprising facilitating transmission to a server including an identification of the first content type and a duration that the first content type is to be presented by the display device.

13. Generating the first prediction of the first content type represented by one or more frames of the video includes: The system of claim 8 , further comprising identifying the media that corresponds to the first content type.

14. Here, the operation is 10. The system of claim 8, further comprising modifying video display or audio settings of the display device based on the first prediction and the first content type to improve presentation of media corresponding to the first content type.

15. One or more non-transitory machine-readable storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to: receiving a first cue from one or more frames of a video of media being presented by a display device, where the first cue includes a set of features derived from the one or more frames of the video; generating a first prediction of a first content type represented by one or more frames of the video using a trained machine learning model and the first cue; receiving a second cue from one or more subsequent frames of video being presented by the display device, where the second cue is received after the first cue; generating a second prediction of a second content type represented by one or more frames of the video using the trained machine learning model and the second cue; and determining a probability that the first content type is inconsistent with the second content type based on the first prediction and the second prediction; and performing a function of the display device based on the probability that the first content type does not match the second content type.

16. 20. The non-transitory machine-readable storage medium of claim 15, wherein one of the first and second content types corresponds to a video game.

17. Receiving the first cue from one or more frames of a video of a media being displayed by the display device includes: identifying one or more sets of pixels from one or more frames of the video; and extracting one or more features corresponding to pixel values ​​from each of the one or more sets of pixels.

18. 16. The non-transitory machine-readable storage medium of claim 15, wherein the machine learning model is an ensemble model derived from two or more machine learning models.

19. Performing the functions of the display device includes:

16. The non-transitory machine-readable storage medium of claim 15, further comprising facilitating transmission to a server including an identification of the first content type and a duration that the first content type is to be presented by the display device.

20. The operation includes:

16. The non-transitory machine-readable storage medium of claim 15, further comprising modifying video display or audio settings of the display device based on the first prediction and the first content type to improve presentation of media corresponding to the first content type.

Images