Method, system, and computer-readable medium for non-occlusion video overlay

Abstract

Methods, systems, and apparatus, including a computer program encoded on a computer storage medium for overlaying content on a video stream. In one aspect, a category of a video is identified. Confidence scores are determined, each confidence score indicating the likelihood that a location in a frame includes a feature of a feature type. A weight is determined for each feature type based on the category, the weight reflecting the importance of not occluding features of the feature type. The confidence score is adjusted for each feature type based on the weight of the feature type to generate an adjusted confidence score. The adjusted confidence scores are aggregated for each location of each frame to generate an aggregated and adjusted confidence score. A location at which overlay content is to be placed during video display is determined based on the aggregated and adjusted confidence scores. The overlay content is provided for display at the determined location in the video.

Description

Technical Field

[0001] The present specification relates generally to data processing and overlaying content on top of different types of video streams while avoiding areas of the video stream that feature specific types of content. Background Art

[0002] The video streamed to the user may include additional content superimposed on the original video stream. The superimposed content may be provided to the user in a rectangular area that superimposes and blocks a portion of the original video screen. In some methods, the area for providing the superimposed content is located at any position on the video screen, such as the middle and lower part of the screen. If the important content of the original video stream is located in the middle and lower part of the video screen, it may be blocked or obstructed by the superimposed content. Summary of the invention

[0003] In general, an innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following operations: for a video with which overlay content is to be displayed, identifying a video category of the video from a set of predefined video categories; for each video frame in a set of sampled video frames of the video: for each video feature type in a set of video feature types and for each position in a plurality of positions in the video frame, determining a confidence score, the confidence score indicating the likelihood that the position in the video frame includes a feature of the video feature type; determining a weight for each video feature type based on the video category, the weight reflecting the likelihood that the video feature type will be displayed when the video is displayed. The invention relates to a method for determining the importance of not obstructing the video feature of the video feature type when viewing a video of a certain category; and for each video feature type in the set of video feature types, adjusting the confidence scores of the multiple positions in the video frame based on the weight determined for the video feature type to generate an adjusted confidence score; aggregating the adjusted confidence scores for each position of each video frame in the set of sampled video frames to generate an aggregated and adjusted confidence score; determining a position where superimposed content will be placed during video display based on the aggregated and adjusted confidence scores; and providing superimposed content for display at the determined position in the video. Other embodiments of this aspect include corresponding methods, apparatus, and computer programs encoded on a computer storage device and configured to perform the actions of the method. These and other embodiments may each feasibly include one or more of the following features.

[0004] In some implementations, video feature types may include human faces, human torsos, text, moving objects, or changes in color variance.

[0005] In some implementations, at least some confidence scores for different video feature types may be determined in parallel.

[0006] In some implementations, adjusting the confidence scores of the plurality of locations in the video frame for each of the set of video feature types may include adjusting the confidence scores of locations in the video frame that are within a predefined neighborhood of a center of the video frame.

[0007] In some embodiments, determining the location at which overlay content will be placed during video display based on the aggregated and adjusted confidence scores can also include determining the location at which overlay content will be placed during video display based on a specified size of the overlay content and a frame duration or number of frames within which the overlay content will be provided within the video.

[0008] In some implementations, determining the location at which the overlay content is to be placed may include determining a plurality of locations, and providing the overlay content at the determined location includes providing the overlay content for display at one of the plurality of locations.

[0009] In some implementations, the method may further include sampling the video based on a sampling rate by the video processing system to obtain the set of sampled video frames. The sampling rate may be determined based on a processing capability of the video processing system.

[0010] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0011] The technology described in this specification provides resource-efficient technology for displaying video content and superimposed content displayed with the video content. For example, when a user is watching a video stream that fills the video screen, the content that is important to the user in the video screen area may not fill the entire area of ​​the video screen. Important content can include video features that the user is likely to be most interested in. For example, important features (e.g., face, text, or significant objects (such as foreground objects or moving objects)) may only occupy a portion of the video screen area. Therefore, there is an opportunity to display additional content to the user in the form of superimposed content that does not obstruct the portion of the video screen area that contains the following important content.

[0012] Aspects of the present disclosure provide the advantage of identifying locations in a video frame that contain features from which superimposed content is excluded, because superimposed content above these locations can block or obscure important content (e.g., content classified as important) included in the video stream below, which can result in wasting computing resources by delivering video to the user when the user cannot perceive the important content, thereby making the delivery of the video incomplete or ineffective. In some cases, a machine learning engine (such as a Bayesian classifier, an optical character recognition system, or a neural network) can identify important features within a video stream, such as faces or other human parts, text, or other significant objects (such as foreground objects or moving objects). Areas covering these important features can be identified; superimposed content can then be displayed outside of these identified areas, for example, at (multiple) locations that have been determined to not have important features (or at least have the minimum possibility of having important features). As a result, the user can receive superimposed content without obstructing the important content of the video stream below, thereby not wasting computing resources required to deliver the video. This results in a more efficient video distribution system that prevents computing system resources (e.g., network bandwidth, memory, processor cycles, and limited client device display space) from being wasted by delivering video in which important content is obscured or otherwise not perceptible to the user.

[0013] This technical solution has the further advantage of improving the efficiency of the screen area in terms of bandwidth for important content delivered to the viewer. If a user is watching a video, where the important content of the video typically only occupies a portion of the viewing area, the available bandwidth for delivering the important content to the viewer is not fully utilized. By using a machine learning system to identify that portion of the viewing area that contains the important content of the underlying video stream, aspects of the present disclosure provide for superimposing additional content outside of that portion of the viewing area, resulting in more efficient use of the screen area for delivering important content to the viewer. In other words, for the same or similar resource overhead, more content is delivered.

[0014] In some approaches, the overlay content includes a box or other icon that the viewer can click to remove the overlay content, for example, if the overlay content obstructs important content in the underlying video. A further advantage of the present disclosure is that because the overlay content is less likely to obstruct important content in the underlying video, there is less interruption in the viewing experience and a greater likelihood that the viewer will not "click away" from the overlay content that has been displayed.

[0015] In addition, identifying important features to avoid occlusion can be further refined and customized based on the category of the video, which can result in changes to each of the above-mentioned technical advantages. For example, different video features may be more important for different video categories. For example, not blocking certain types of video features may be more important for some video categories, and identifying the most important video features for a category can improve the efficiency of content distribution (e.g., by distributing videos in which important content is occluded without wasting resources). As another example, for some video categories, certain video features may be less important (or unimportant), and viewing areas including low-importance content or non-important content can be used to overlay content, thereby improving the efficient use of screen area to deliver important content to users.

[0016] In addition, the techniques described in this specification provide various methods for reducing the processing time (and resource usage) for determining the location at which the superimposed content will be displayed. For example, various downsampling and parallelization methods can be used. Thus, the identification of the appropriate location at which the superimposed content will be displayed can be performed using fewer computing resources, while still benefiting from the other technical advantages of the present invention that occur when the superimposed content is displayed at the identified (multiple) locations.

[0017] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram of an example environment in which digital content is distributed and provided for display on a client device.

[0019] Figure 2 An example video frame is shown.

[0020] Figure 3 is a block diagram of an example environment in which a video processing system determines a location of overlay content within a video.

[0021] Figure 4 An example heat map is shown.

[0022] Figure 5 An example confidence score visualization is shown.

[0023] Figure 6 is a flow chart of an example process for determining a location within a video to display overlay content.

[0024] Figure 7 is a block diagram of an example computer system that may be used to perform the described operations. DETAILED DESCRIPTION

[0025] The present disclosure generally relates to overlaying content on top of different types of video streams while avoiding areas of the video stream with important content.

[0026] As summarized below and described in more detail throughout this document, video systems generally display overlay content (e.g., text, images, etc.) at a static, predetermined location within a video (such as in the lower-middle portion of the video). However, for some videos (or for some types of videos), displaying overlay content at a static, predetermined location may obscure important content of the video. For example, a sports-related video may display a scoreboard in the lower-middle portion of the video, and overlay content displayed at a preconfigured location (e.g., in the lower-middle portion) may obscure the scoreboard, which may frustrate the user and cause the following video content to be less important to the user.

[0027] The technology described herein provides a video system that can be configured to identify a portion of a video where superimposed content can be provided without obstructing important video content. Important content can include, for example, video features, such as text, human faces, human torsos, moving objects, or portions of a video that undergo a color variance change between video frames. Important content can include the type of video feature on which displaying superimposed content would be undesirable for a user because the user wishes to view the content and finds the content interesting. In some embodiments, a video system can be configured to automatically identify video features of a video feature type corresponding to important content within a set of frames of a video. The video system can automatically determine a location where superimposed content will be displayed by identifying a location where a video feature is not identified. For example, a confidence score can be determined for each video feature type and for each of a plurality of locations in a video frame. Each confidence score can indicate the likelihood that a location in a video frame includes a feature of a particular video feature type. Confidence scores can be aggregated and used to determine a location where superimposed content will be displayed.

[0028] In some embodiments, depending on the type or category of the video, some features of the video may be important because superimposing content on those features will interfere with the viewing of the video, while other features may not be so important because superimposing content on those features will not interfere with the viewing of the video. Therefore, the video system can be configured to adjust the confidence values ​​of different features based on the category or type of the video. In some embodiments, for videos of different types / categories, the video system can be configured to assign different importance levels (e.g., weights on a numerical scale from 1 to 5) to each detected feature type. The high importance / weight of a specific feature type indicates that the specific feature type should not be blocked by the superimposed content, while the low importance / weight indicates that the specific feature type can be blocked by the superimposed content. For example, for talk show videos, high importance (e.g., a weight of 4 in the range of 1 to 5) can be assigned to the part of the video that is not superimposed on the human face. For videos related to sports, high importance (e.g., a weight of 4 or 5 in the range of 1 to 5) can be assigned to the text part of the video that is not superimposed (e.g., a part including a score) and to the part of the video that is not superimposed on the moving part.

[0029] Based on the weights assigned to each feature type, the confidence values ​​of the various feature types can be adjusted based on the video category weights, thereby reducing the likelihood of superimposing content on feature types that are more important for the video category. Based on the adjusted confidence values ​​for the different feature types, the video system can identify one or more locations where superimposed content can be displayed. The video system can then provide the superimposed content for display at one of these identified locations.

[0030] Refer to the following Figure 1-7 These and additional features are described in more detail further on.

[0031] In addition to the description throughout this document, controls may be provided to the user that allow the user to make choices about whether and when the systems, programs, or features described herein can collect user information (e.g., information about the user's social network, social behavior or activities, occupation, the user's preferences, or the user's current location) and whether to send content or communications from a server to the user. In addition, certain data may be processed in one or more ways before being stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be summarized (such as to a city, zip code, or state level) where location information is obtained so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how that information is used, and what information is provided to the user.

[0032] Figure 11 is a block diagram of an example environment 100 in which digital content is distributed and provided for display on a client device. Example environment 100 includes a network 104. Network 104 may include a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. Network 104 may also include any type of wired and / or wireless network, satellite network, cable network, Wi-Fi network, mobile communication network (e.g., 3G, 4G, etc.), or any combination thereof. Network 104 may utilize communication protocols, including packet-based and / or datagram-based protocols, such as Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), or other types of protocols. Network 104 may also include multiple devices that facilitate network communications and / or form a network hardware foundation, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, or a combination thereof.

[0033] The network 104 connects the client devices 102, the content platforms 106, the content providers 108, and the video processing systems 110. The example environment 100 may include many different content platforms 106, video processing systems 110, client devices 102, and content providers 108.

[0034] Content platform 106 is a computing platform capable of distributing content (such as, for example, Figure 7 1. The content platform 106 may be a web server or another data processing device as described herein. Example content platforms 106 include search engines, social media platforms, video sharing platforms, news platforms, data aggregation platforms, or other content sharing platforms. Each content platform 106 may be operated by a content platform service provider.

[0035] The content platform 106 may publish its own content and make its own content available. For example, the content platform 106 may be a news platform that publishes its own news articles. The content platform 106 may also display content provided by one or more content providers 108 that are not part of the content platform 106. In the above example, the news platform may also display third-party content provided by one or more content providers 108. As another example, the content platform 106 may be a data aggregation platform that does not publish its own content but aggregates and displays third-party content provided by different content providers 108.

[0036] In some embodiments, the content platform 106 may store certain information about the client device (e.g., device preference information, content consumption information, etc.). Such user information may be used by the content platform, for example, to customize the content provided to the client device 102 or to enable convenient access to specific content frequently accessed by the client device 102. In some embodiments, the content platform 106 may not store such device information on the platform; however, the content platform 106 may still provide such information for storage on a specific server (separate from the content platform). Therefore, the content platform 106 (also referred to herein as the content platform / server 106 or simply the server) refers to the content platform that stores such device information or the server (separate from the content platform) that stores such device information.

[0037] In some embodiments, the content platform 106 is a video service through which a user can watch streaming video content. The video streamed to the user may include additional content (e.g., provided by the content provider 108) superimposed on the original video stream. Generally, it is desirable to provide superimposed content on the underlying video stream to provide additional content to viewers of the video stream and to increase the amount of content transmitted in the viewing area for a given video stream bandwidth. In addition to or in lieu of the video stream scenario, the content platform 106 may include a video processor that processes a video file to modify the video file so as to include superimposed content, wherein the processed video file with superimposed content is provided to the client device 102 for display on the client device 102.

[0038] However, there is a technical problem of determining how to place the overlay content so that it does not obstruct important content in the underlying video. This is a particularly difficult problem in the context of overlaying content on a video, because the location of important content in the video can change rapidly over time. Therefore, even if a particular location within the video is a good candidate for overlay content at one point in time (e.g., in one frame), that location may become a poor candidate for overlay content at a later point in time / in a subsequent frame (e.g., due to movement of a character within the video).

[0039] Using the position information of the important content determined by the video processing system 110, the content platform 106 can superimpose the content on the video stream, while avoiding the area with important content in the video stream below the video screen, such as the area containing face, text or significant objects (such as moving objects) in the original video stream. The video processing system 110 can, for example, include a machine learning method and an engine, which can identify the position in the video that is unlikely to include important content, so that the superimposed content displayed at those positions is unlikely to hinder the important content in the following video. As described in more detail below, different types of (e.g., categories) of videos can include different types of important content. Therefore, the video processing system 110 can prioritize the position of the superimposed content based on avoiding video features that are particularly important for the video category that has been determined for the video.

[0040] Additional structural and operational aspects of these components of the example environment 100 are described in detail in Figure 3 to describe.

[0041] Figure 2 An example video frame 200 is shown. Video frame 200 includes various features, such as person 202 and various objects (e.g., cup 204, laptop 206, sign 208, text label 209, and social media post 210). In addition to text label 209, some other objects included in video frame 200 may include other text. For example, cup 204 has a text label, laptop 206 has a sticker 212 including text, sign 208 includes text 214, and social media post 210 includes text 216.

[0042] When overlay content is displayed on top of video frame 200, it may be desirable to not obstruct important features displayed in frame 200. Some features may be more important to the user than other features. In some cases, feature size may determine importance. For example, the face of person 202 may be more important than face 218 in a social media post based on the face of person 202 being larger than face 218. As another example, the face of person 202 may be more important than face 218 due to the movement of person 202 during the video.

[0043] As another example, feature importance can be based on the category of the video. Figure 3 As described above, the category of the video can be determined. For example, the video frame 200 can be a frame from a talk show video, and the video can have a talk show or entertainment category. The person 202 can be, for example, a talk show host.

[0044] For talk show videos, certain types of features, such as human faces or torsos, may be particularly important. Other types of features, such as text, may also be important, but not as important as human features. Compared with other video categories, talk show videos can give different types of importance to certain features. For example, for sports videos, static texts such as scoreboards and moving objects (e.g., players) can have the highest importance. In sports (e.g., baseball) videos, small text items such as ball counts, hit counts, and out counts can be considered important, while for talk show videos, small text items such as text on cup 204 can be considered not to be particularly important (although larger text items, such as text 214 on sign 208 or text 216 in social media post 210 may still be of great significance).

[0045] As described in more detail below, the video processing system 110 can determine, based on a video category (e.g., talk show) of the video that includes the video frame 200 and based on the detected locations of features that are important to the video type, one or more locations at which to display overlay content in the video so as not to obscure features that are important to the video category.

[0046] Figure 3 300 is a block diagram of an example environment 300 in which a video processing system 110 determines a location within a video at which to display overlay content. The video processing system 110 can receive or access an input video 302. For example, the video processing system 110 can receive a video file or a video stream, or a link to a video file or a video stream. In some implementations, the video processing system 110 can process multiple video inputs in parallel (and generate corresponding outputs for each video input).

[0047] In some embodiments, the video processing system 110 includes a preprocessor 304. The preprocessor 304 can, for example, preprocess the input video 302 to provide consistency in one or more of frame rate, video size, video quality, video resolution, or video format. The output of the preprocessor 304 can be a video stream in a standard format for further processing by other sub-engines of the video processing system 110.

[0048] The video classifier 306 can identify the category of the input video 302 from a set of predefined video categories. To cite a few examples, the video category may include entertainment, games, lifestyle, sports, knowledge and society. The video classifier 306 can determine the category of the input video 302 based on, for example, the metadata of the input video 302. The metadata may include a video title or may include a video category or genre. The category or genre included in the video metadata may be mapped to one of the predefined video categories. The metadata 302 may be received together with the input video 302, or may be accessed by the video classifier 302 as part of the video classification. In some embodiments, the video category is provided to the video processing system 110 as an input (e.g., together with the input video 302). Other types of video classification may be performed, such as techniques involving analyzing a subset of frames of the input video 302 and determining the video category based on the content of the analyzed frames.

[0049] The video sampler 308 can capture a subset of the frames of the input video 302. For example, the video sampler 308 can determine a sampling rate, which can be, for example, frames per second or seconds per frame. For example, the video sampler 308 can capture three frames per second, one frame per second, or one frame every three seconds. The sampling rate can be determined based on various factors. In some cases, the sampling rate can be determined based on the video category. For example, some types of video content can be more dynamic (e.g., in terms of content changes between frames) than other types of content that are more static. For example, previous analysis can show that entertainment videos are generally more dynamic than knowledge videos. Therefore, the sampling rate for entertainment videos can be higher than the sampling rate for knowledge videos.

[0050] In some cases, the sampling rate can be determined based on the processing capacity of the video processing system 110. For example, a particular instance of the video processing system 110 receiving a given input video 302 can have a preconfigured processing capacity. As another example, the processing capacity can be dynamically determined, such as based on current (or recent) resource utilization. For example, recent resource utilization can be based on the recent processing of other input videos by the video processing system 110. In some embodiments, the sampling rate can be a parameter that can be configured by an administrator. Reducing the sampling rate can result in resource efficiency by reducing the number of frames processed.

[0051] The sampled frame may be processed by a set of feature detectors 310. The feature detectors 310 include a face detector 310a, a human joint detector 310b, a body contour detector 310c, a text detector 310d, a moving object detector 310e, and a color signal variance detector 310f. In some embodiments, the feature detectors 310 are run at least partially in parallel. A separate task may be initiated for each feature type, such as where a master task performs coordination and monitoring of parallel tasks. Some or all feature detectors 310 may perform certain optimizations specific to a given feature detector 310. For example, for some (or all) types of feature detection, the frame image may be downsampled (e.g., to a lower resolution).

[0052] Each feature detector 310 may determine a confidence score for each position in each sampled frame, where a given confidence score indicates the confidence that a given position in a given frame includes a feature corresponding to a given feature detector 310. The confidence score may range from 0 to 1, e.g., where 0 indicates the lowest confidence in the feature detection and 1 indicates the strongest confidence in the feature detection.

[0053] In some approaches, detected features may be differentiated by limiting feature detection to larger features, i.e., features that are in the foreground and closer to the video viewpoint than background features. For example, larger human faces corresponding to people in the foreground of the video may be detected by the face detector 310a, while smaller human faces corresponding to people in the background of the video (such as faces in a crowd) may be excluded from detection by the face detector 310a or may receive a lower confidence score than the larger faces.

[0054] The confidence scores may be stored in a confidence score map or matrix. For example, the matrix dimensions and addressing may correspond to the dimensions and positions of a video frame. For example, the positions may correspond to a single pixel or a block of multiple pixels. For example, the positions may be a 10×10 block of pixels. The block size may be a configurable or dynamically determined parameter. Larger or smaller block sizes may be configured based on, for example, processing power or desired accuracy or processing time. As mentioned above, processing power may be determined based on a preconfigured resource description or based on dynamic performance. For example, higher processing power or a higher level of desired accuracy may result in a smaller block size. For example, lower processing power or a smaller desired processing time may result in a larger block size.

[0055] The face detector 310 may be configured to detect human faces in the sampled frames. The face detector 310a may detect blurred faces, side profiles, and closed-eye faces. The face detector 310a may create a confidence map for each sampled frame, wherein the confidence score for the region within the detected face is non-zero (where the higher the confidence of the detected face, the higher the non-zero confidence score) and the confidence score for the region of the frame outside the detected face is zero. The face detector 310a may be or include a computer vision system, such as a machine learning system, for example a Bayesian image classifier or a convolutional neural network (CNN) image classifier. For efficiency purposes, the face detector 310a may find frame locations where human faces appear in the frame without actually identifying the identity of the person displayed within those locations (e.g., identifying the face of a specific person displayed within those regions).

[0056] The human joint detector 310b, which may be or include a machine learning computer vision engine, may detect other important human body parts, such as the neck, shoulders, elbows, wrists, or hands. In some embodiments, certain body parts (such as those of the upper body) may be treated as more important than lower body parts. In some cases, the human joint detector 310b is configured to weight the upper body parts more importantly, while in other embodiments, the human joint detector 310b may exclude the lower body parts from detection. Once a joint is detected, an area (e.g., a circle) around the joint may be identified, and the area within the circle may be considered to include the location of an important human body part. The size of the circle may be based on the detected size of the person to whom the joint belongs (e.g., a larger person in the frame may have a larger circle area around the detected joint). The size of the circle (or the ratio of the person size to the circle size) may be a configurable parameter. Different joints may have circles of different sizes compared to other types of joints. For example, the shoulder joint may be considered to be more important than the elbow joint, so a larger circle may be used for the shoulder compared to the elbow (or as another example, the shoulder joint may be represented by a larger circle because it is generally a larger joint than the elbow joint). The joint detector 310b can create a confidence map for each sampled frame, where the confidence score of the area within the joint-related circle is non-zero (where the higher the confidence of the detected joint, the higher the non-zero confidence score) and the confidence score of the area of ​​the frame outside the joint-related circle is zero.

[0057] The body contour detector 310c can determine the contour of the person detected in the sample frame. The body contour detector 310c can be or include a CNN. In some embodiments, in addition to the face detector 310a and the human joint detector 310b, the body contour detector 310c can be used to identify any area of ​​the frame occupied by a human body rather than just important human body parts (such as a face or important joints). In some embodiments, the body contour detector 310c generates a confidence map, but includes a score whose weighting is less than the high confidence score generated by the face detector 310a or the human joint detector 310b. For example, the face detector 310a can score a position with a detected face with a confidence score of 1 for high confidence in the detected face, the human joint detector 310b can score a position with a detected joint with a confidence score of 0.8 for high confidence in the detected joint, and the body contour detector 310c can score a position within the body contour with a confidence score of 0.6. The net result of having different score weights between the face detector 310a, the human joint detector 310b, and the body contour detector 310c can be to handle avoiding occlusion of important body parts according to body part priority, where faces are treated most cautiously and important joints are treated more cautiously than body parts other than faces and important joints. Ultimately, whether to place overlay content at a given frame location is based on the aggregated (and adjusted) confidence scores aggregated across the feature detectors 310, as described in more detail below.

[0058] The text detector 310d can detect text in the sampled frames to avoid obstructing important text content, such as subtitles, game scores, or titles. The text detector 310d can detect text features within the video frame, such as text that appears on product labels, road signs, whiteboards on the screen in school lecture videos, etc. The text detector 310d can detect text in multiple languages ​​and can recognize text of various sizes or distorted text. It is worth noting that the text features detected within the frame are part of the video stream itself, compared to superimposed content that is separate from the video stream.

[0059] The text detector 310d can be or include a machine learning system, such as an optical character recognition (OCR) system. For efficiency purposes, the text detector 310d can find frame locations where text appears without actually recognizing the value of the text displayed in those locations. The text detector 310d can create a confidence map for each sampled frame, where the confidence score of the location including the text is non-zero (where the higher the confidence of the detected text, the higher the non-zero confidence score) and the confidence score of the area of ​​the frame outside the area including the text is zero. In some embodiments, a fill area (e.g., twenty pixels) is added to the location including the text as a transition gradient, where the fill area is considered to be part of the text area.

[0060] Other types of important features that the user may prefer not to be obscured may be moving objects. For example, objects in motion are generally more likely to convey important content to the viewer than static objects, and are therefore generally less suitable for being obscured by superimposed content. The moving object detector 310e can detect moving objects by detecting movement between adjacent sample frames based on the color space difference between the frames. Processing can be performed on each frame (except the first and last frames) using the previous frame, the current frame, and the next frame (wherein a confidence score for the current frame is calculated). Moving object detection can include calculating the average color value of pixels within a frame position (for example, when a position corresponds to multiple pixels), and calculating the Euclidean difference between the average color value of the position and the average color value calculated for the corresponding position in the previous frame and the next frame. A higher difference value can correspond to a higher confidence score, which indicates the possibility that the position corresponds to moving (and therefore important) content. Moving object detection can be based on the following equation:

[0061]

[0062] in

[0063] Where S = all integer pairs i,j∈[x:x+10),[y:y+10)

[0064] Among them A t = RGB matrix representing the frame at time t

[0065] where |M0[x,y,t0]-M0[x,y,t1] = the RGB Euclidean distance between two 3×1 vectors Maximum possible Euclidean distance

[0066] and where i,j,C = color value C at coordinates i,j.

[0067] Color variance may be another type of feature. Color variance between frames at a location may indicate important content, such as foreground content. Low or no color variance between frames may indicate less important content, such as background areas (e.g., walls, sky, etc.). The color signal variance detector 310f may determine the color variance between frames and calculate a confidence score for each location in each respective frame indicating whether the location is subject to the color variance of the frame relative to the previous and subsequent (multiple) frames. As described above, the location with respect to the color variance location may be a pixel block. The color variance may be calculated according to the following formula:

[0068]

[0069] where S = all integer pairs i,j∈[x,x+50),[y,y+50)

[0070] Among them A t = RGB matrix representing the frame at time t

[0071] The confidence value adjuster 312 can adjust the confidence value generated by the feature detector 310 based on the video category determined by the video classifier 306. The confidence value adjuster 312 can identify or determine the weight of each feature type based on the video category. As mentioned above, some feature types are more important for some video categories. The weight of each feature type of a category can be, for example, a value between 0 and 1. A weight value of 1 can indicate that the feature type is important for the video category. A weight value of a feature type of a video category less than 1 (e.g., 0.5) can indicate that the feature type is not very important for the video category. For example, for a talk show video, the weight of a human facial feature can be 1 and the weight of a text feature can be 0.8. A sports video category can have a weight of 1 for a moving object feature, a weight of 1 for a text feature (e.g., for a scoreboard or statistics), and a weight of 0.8 for a human facial feature. In some embodiments, the confidence value adjuster 312 uses a lookup table to retrieve the weight of a given category. The lookup table can store the weight values ​​of each of the various features for each category. The lookup table may include weight values ​​that are preconfigured by an administrator, or the lookup table may include weight values ​​that are dynamically determined or adjusted by a machine learning system that is trained to determine weight adjustments for different video categories.

[0072] The confidence value aggregator 314 can aggregate the adjusted confidence scores that have been adjusted by the confidence value adjuster 312 to generate an aggregated and adjusted confidence score for each position of each sampling frame. The aggregated and adjusted confidence scores can be stored in a matrix for each sampling frame. For each position and for each sampling frame, the aggregation can include summing the confidence values ​​calculated by each feature detector 310 for the position. If the sum is greater than 1 for a position of the sampling frame, the confidence value aggregator 314 can limit the sum to 1. Therefore, the aggregated and adjusted confidence score can have a value between 0 and 1, where a value of 1 indicates the highest confidence of the important feature at a position, and a value of 0 indicates the lowest confidence of the important feature at the position.

[0073] The confidence value aggregator 314 (or in some embodiments, the confidence value adjuster 312) can further adjust the confidence scores to improve the confidence scores of locations at or near the center of the frame. Based on the general understanding that important content of the video is likely to occupy the central area of ​​the frame (and that overlay content placed in the center may cause the user's attention to be too distracted from the main content of the video), the center-based adjustment can be made to make it less likely that a center location will be selected for overlay content. For example, a constant two-dimensional Gaussian distribution can be applied as a multiplier to the aggregated and adjusted confidence scores, for example according to the following formula:

[0074]

[0075] in

[0076] where x,y∈[-width,width),[-height,height)

[0077] The aggregated and adjusted confidence scores can be used for various purposes. For example, in some embodiments and as described below with respect to Figure 4 As shown, the visualizer 316 can generate and present a heat map for each of the one or more sample frames. The heat map of the frame can display confidence values ​​presented as different colors. For example, red can indicate a high confidence value and green can indicate a low confidence value. As another example, the heat map of the frame can be combined with the frame image (e.g., superimposed on the frame image) and presented as a merged image, as shown below with respect to Figure 5 Individual and / or overlaid heat maps may be viewed by an administrator or developer for, for example, monitoring, troubleshooting, or debugging purposes.

[0078] As another example, in some implementations, the aggregated and adjusted confidence scores may be used to calculate quality scores for one or more existing (e.g., in-use) content slots currently used to display overlay content on the input video 302. A determination may be made based on the calculated quality scores of the existing content slots whether to continue to display the overlay content in the existing content slots (e.g., a higher quality score may indicate that the existing content slot is suitable for displaying the overlay content, while a lower quality score may indicate that the existing content slot is not ideal for displaying the overlay content).

[0079] The aggregated and adjusted confidence scores may be used by the overlay position identifier 318 to determine one or more outputs 320 based on, for example, one or more overlay content and size inputs 322 received as input, each output 320 comprising a recommended overlay content position 320a and a corresponding time offset 320b for the input video 302. Each overlay content and size input 322 may specify a desired size and display duration for an overlay content item to be overlaid on the input video 302 for a specified display duration.

[0080] The detected features (as represented by the aggregated and adjusted confidence scores) may be located at different locations within different frames occupying a particular duration of the input video 302. The overlay content position 320a may be determined so that the overlay content that begins display at the time offset 320b and is displayed for a specified duration including multiple frames is not located at any location where the detected features are located across the multiple frames. The identified overlay content position 320a may be a location that is generally or substantially outside of the location of the identified features in a series of video frames corresponding to the input duration. The location including the identified features may be a location that should not (if possible) be obscured by the overlay content (e.g., to ensure that important content of the input video 302 below is not obscured). As another example, the overlay content position 320a may be a location corresponding to a confidence score that is below a predetermined threshold.

[0081] The overlay position identifier 318 can determine a candidate overlay content slot for a given overlay content item size and duration (e.g., based on the content item size and the duration of the content item, the duration of the content item corresponds to a specific number of consecutive / contiguous video frames in the input video 302). The overlay position identifier 318 can determine a quality score for each candidate slot for each frame in the series for each series of different series of consecutive frames corresponding to the desired duration. The quality score of the candidate slot of the frame can be the sum of the aggregated and adjusted confidence scores of all positions located within the candidate slot. A total quality score can be calculated for a series of candidate slots by calculating the sum of all quality scores of the candidate slots of all frames in the series. The series can be identified by the time offset of the first frame of the series. A set of one or more highest ranked total quality scores can be identified.

[0082] Based on the one or more highest ranked total quality scores, the overlay position identifier 318 can identify one or more outputs 320, wherein each output 320 includes an overlay content position 320a (e.g., position and size) of a candidate slot having one of the highest ranked quality scores, and a corresponding time offset 320b indicating a starting time offset at which the overlay content item is to be displayed.

[0083] The output(s) 320 may be provided to, for example, the content platform 106 to enable the content platform 106 to display the overlay content item(s) corresponding to the output(s) at the recommended overlay content position(s) 320a, starting at the corresponding time offset(s) 320b, during playback of the input video 302. As another example, such as during development or troubleshooting, the output(s) 320 may be provided to the visualizer 316, and the visualizer 316 may present, for example, an overlay content outline (or in some cases, the actual overlay content) for monitoring or troubleshooting purposes, e.g., for viewing by an administrator or developer.

[0084] Figure 4 An example heat map 400 is shown. Figure 3 The heat map 400 generated by the visualizer 316 of the can be used to visualize the aggregated and adjusted confidence scores generated by the confidence value aggregator 314. For example, the heat map 400 can be used by an administrator or developer for troubleshooting or informational purposes. The heat map 400 is a diagram of the above description of the confidence value aggregator 314. Figure 2 A visualization of a matrix of aggregated and adjusted confidence scores for each sampled video frame 200 is depicted.

[0085] Different colors may be used to display different confidence levels. For example, red may indicate a high confidence value and green may indicate a low confidence value. Regions 402a-402f of the heat map 400 that may be colored red correspond to high aggregated and adjusted confidence scores based on confidence scores generated by the face detector 310a, the human joint detector 310b, and / or the body contour detector 310c. Depending on the talk show video category determined by the video classifier 306, the regions 402a-402f may have a red color (e.g., corresponding to a high confidence score) based on the adjustment / weighting performed by the confidence value adjuster 312. Regions 404a-404e of heat map 400 may also be colored red (e.g., with the same or lighter shade(s) of red) based on (relatively) high aggregated and adjusted confidence scores corresponding to text detector 310d detecting text on text label 209, social media post 210, cup 204, sticker 212, and sign 208. Other regions of heat map 400, such as region 406 and region 408, may be colored green (e.g., shades of green) to reflect lower aggregated and adjusted confidence scores. As described above and as shown in Figure 5 As shown, overlay position identifier 318 may determine position(s) corresponding to low confidence scores that persist across a set of frames for displaying overlay content for a set of frames equal to the desired duration.

[0086] Figure 5 An example confidence score visualization 500 is shown. The confidence score visualization 500 shows Figure 4 At least a portion of the heat map 400 is Figure 2 As with heat map 400, confidence score visualization 500 may be (e.g., by Figure 3 316) is optionally generated for troubleshooting, monitoring, or other informational purposes. For example, visualization 500 includes shaded regions 502a corresponding to features of person 202 that have been identified by face detector 310a, human joint detector 310b, and / or body contour detector 310c. Shaded region 502b corresponds to detection of face 202 in social media text 210. Shaded text regions 504a-504e correspond to detection of text label 209, text 216 on social media post 210, text on cup 204, text on sticker 212, and text 214 on sign 208 by text detector 310d. Other shaded regions may be displayed, or shaded regions corresponding to confidence scores above a predetermined threshold may be displayed (e.g., so that only regions with the highest confidence scores are displayed).

[0087] The content slot 506 located outside the colored area corresponding to the high confidence score corresponds to a recommended location for displaying the superimposed content item within a specified duration, which begins at (or at least includes) a time offset of the talk show video corresponding to the video frame 200. The content slot 506 can have a size equal to the desired superimposed content size. For informational purposes, a quality score 508 of the content slot 506 is displayed, which can correspond to the highest ranked quality score of the frame 200 determined by the superimposed position identifier 318. As mentioned, the position information describing the location of the content slot 506 and the time offset corresponding to the frame 200 can be provided to the content platform 106 for displaying the superimposed content item at that location, for example, when the end user selects to watch the talk show video.

[0088] Although a single frame and a single content slot visualization are shown, multiple frame visualizations are generated (e.g., as a collection of still images or as a modified video stream) and displayed to an administrator, developer, or content operator, for example, by the visualizer 216. Furthermore, each frame visualization may include a presentation of more than one candidate content slot, such as the top N content slots identified for the current frame.

[0089] Figure 6 6 is a flow chart of an example process 600 for determining a location within a video where overlay content will be displayed. The operation of process 600 is described below as follows: Figure 1 and Figure 3The operations of process 600 are performed by components of the system described and depicted in the description. The operations of process 600 are described below for illustration purposes only. The operations of process 600 may be performed by any suitable device or system (e.g., any suitable data processing device). The operations of process 600 may also be implemented as instructions stored on a computer-readable medium, which may be non-transitory. The execution of the instructions causes one or more data processing devices to perform the operations of process 600.

[0090] The video processing system 110 identifies (at 602) a video category for a video with which the overlay content is to be displayed (e.g., on which the overlay content is to be displayed) from a set of predefined video categories. For example, as described above with reference to Figure 3 As described, the video classifier 306 can identify the video category of the input video 302. The video category can include entertainment, games, lifestyle, sports, knowledge and society, among other categories.

[0091] The video processing system 110 performs processing on each video frame in a set of sampled video frames of the video (at 604). For example, the video processing system 110 determines a confidence score for each video feature type in a set of video feature types and for each position in a plurality of positions in the video frame, the confidence score indicating the likelihood that the position in the video frame includes a feature of the video feature type (at 606). The video feature types may include a human face, a human torso, text, a moving object, or a change in color variance. For example, as described above with reference to Figure 3 As described, each of the feature detectors 310 including the face detector 310a, the human joint detector 310b, the body contour detector 310c, the text detector 310d, the moving object detector 310e, and the color signal variance detector 310f can determine a confidence score for each position of each frame. The positions within the frame can be pixels or pixel blocks. At least some confidence scores of different video feature types can be determined in parallel. For example, two or more of the feature detectors 310 can run in parallel.

[0092] The video processing system 110 determines a weight for each video feature type based on the video category, the weight reflecting the importance of not obscuring the video features of the video feature type when displaying a video of the video category (at 608). Figure 3 As described above, the confidence value adjuster 312 may determine a weight for each feature type of features associated with the feature detector 310 based on the video category determined by the video classifier 306 .

[0093] The video processing system 110 adjusts the confidence scores of the plurality of locations in the video frame for each video feature type in the set of video feature types based on the weight determined for the video feature type to generate adjusted confidence scores (at 610). For example, as described above with reference to Figure 3 As described, the confidence value adjuster 312 adjusts the confidence score determined by the feature detector 310 based on the corresponding weight of the video feature type determined according to the video category determined by the video classifier 306. In some embodiments, the confidence score of the position in the video frame within a predefined neighborhood of the center of the video frame can be further adjusted (e.g., improved) to prevent the superimposed content from being positioned in the center of the frame or attached.

[0094] The video processing system 110 aggregates the adjusted confidence scores for each position of each video frame in the set of sampled video frames to generate an aggregated and adjusted confidence score (at 612). For example, as described above with reference to Figure 3 As described, confidence value aggregator 314 aggregates the adjusted confidence scores generated by confidence score adjuster 312 to generate an aggregated and adjusted confidence score.

[0095] The video processing system 110 determines, based on the aggregated and adjusted confidence scores, where the overlay content will be displayed during the video display (at 614). Figure 3 As described above, the overlay position identifier 312 can determine an overlay content position 320a and a time offset 320b of the input video 302 at which to begin displaying the overlay content as an overlay on the input video 302 based on the size of the overlay content and the desired display duration. Determining the position at which the overlay content is to be displayed during video display can include determining the position based on a specified size of the overlay content and a frame duration or number of frames within which the overlay content is to be provided in the video. For example, with respect to Figure 3 , the overlay position identifier can determine the overlay position based on the overlay content size and duration input 322.

[0096] The video processing system 110 provides the overlay content for display at the determined location in the video (at 616). Figure 3 As described, the video processing system 110 may provide the overlay content position 320a and the corresponding time offset 320b of the overlay content item to the content platform 106 (in some embodiments, the overlay content item itself) so that the content platform 106 presents the overlay content item at the overlay content position 320a starting from the time offset 320b as an overlay on the input video 302.

[0097] Thus, in this manner, the video processing system 110 and / or content platform 106 may display overlay content items on top of video content without obstructing important video content, and importantly avoiding content that is most important to the video of the video category determined for the video content.

[0098] Figure 7 7 is a block diagram of an example computer system 700 that can be used to perform the above operations. System 700 includes a processor 710, a memory 720, a storage device 730, and an input / output device 740. Each of components 710, 720, 730, and 740 can be interconnected, for example, using a system bus 750. Processor 710 is capable of processing instructions for operation within system 700. In some embodiments, processor 710 is a single-threaded processor. In another embodiment, processor 710 is a multi-threaded processor. Processor 710 is capable of processing instructions stored in memory 720 or stored on storage device 730.

[0099] The memory 720 stores information within the system 700. In one embodiment, the memory 720 is a computer readable medium. In some embodiments, the memory 720 is a volatile memory unit. In another embodiment, the memory 720 is a non-volatile memory unit.

[0100] The storage device 730 can provide mass storage for the system 700. In some embodiments, the storage device 730 is a computer-readable medium. In various embodiments, the storage device 730 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices over a network (e.g., a cloud storage device), or some other mass storage device.

[0101] The input / output device 740 provides input / output operations for the system 700. In some embodiments, the input / output device 740 may include one or more of a network interface device, such as an Ethernet card, a serial communication device (e.g., an RS-232 port) and / or a wireless interface device (e.g., an 802.11 card). In another embodiment, the input / output device may include a driver device configured to receive input data and send output data to a peripheral device 760, such as a keyboard, a printer, and a display device. However, other embodiments may also be used, such as a mobile computing device, a mobile communication device, a set-top box TV client device, etc.

[0102] Although already Figure 7 An example processing system is described in the specification, but the subject matter and functional operations described in this specification may be implemented in other types of digital electronic circuits or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents) or in a combination of one or more of them.

[0103] The subject matter described in this specification and the embodiment of operation can be realized in digital electronic circuit or in computer software, firmware or hardware (including the structure disclosed in this specification and its structural equivalent) or in one or more combinations thereof. The embodiment of the subject matter described in this specification can be implemented as one or more computer programs, i.e. one or more computer program instruction modules, encoded on multiple computer storage media (or a computer storage medium) for data processing device operation or control of the operation of data processing device. Alternatively or additionally, the program instruction can be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical signal, optical signal or electromagnetic signal), which is generated to encode information for transmission to a suitable receiver device so as to be operated by a data processing device. The computer storage medium can be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof, or can be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof. Furthermore, although a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium can also be one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices), or can be included in one or more separate physical components or media.

[0104] The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0105] The term "data processing apparatus" encompasses all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or a plurality or combination of the above. The apparatus may include dedicated logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an operating environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and operating environment may implement a variety of different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0106] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it may be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file holding other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0107] The processes and logic flows described in this specification may be performed by one or more programmable processors that run one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by a dedicated logic circuit (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)), and the apparatus may also be implemented as a dedicated logic circuit.

[0108] Processors suitable for running computer programs include, for example, general-purpose microprocessors and special-purpose microprocessors. Generally, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic element of a computer is a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more large-capacity storage devices (for example, magnetic disks, magneto-optical disks or optical disks) for storing data, or can be operatively connected to receive data from one or more large-capacity storage devices or transmit data to one or more large-capacity storage devices, or can include one or more large-capacity storage devices for storing data and can be operatively connected to receive data from one or more large-capacity storage devices or transmit data to one or more large-capacity storage devices. However, a computer does not have to have such a device. In addition, a computer can be embedded in another device (for example, to name a few, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device (for example, a universal serial bus (USB) flash drive)). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0109] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; input from the user may be received in any form, including sound input, voice input, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser in response to a request received from the web browser on the user's client device.

[0110] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internets (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0111] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a client device (e.g., for displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., a result of a user interaction) can be received from the client device at the server.

[0112] Although this specification contains many specific implementation details, these details should not be interpreted as limiting the scope of any invention or the scope of the content that can be claimed, but should be interpreted as a description of the characteristics unique to a specific embodiment of a specific invention. Certain features described in the context of a separate embodiment in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. In addition, although the features may be described above as working in certain combinations, and even initially claimed as such, in some cases, one or more features from the claimed combination may be excluded from the combination, and the claimed combination may be about a sub-combination or a variation of a sub-combination.

[0113] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the particular order shown or in a sequential order or performing all operations shown to achieve desired results. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of each system component in the above-described embodiment should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and system generally can be integrated together in a single software product or packaged into multiple software products.

[0114] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

Claims

1. A computer-implemented method for non-occluded video overlay, comprising: identifying, for a video with which the overlay content is to be displayed, a video category of the video from a set of predefined video categories; For each video frame in a set of sampled video frames of the video: For each video feature type in a set of video feature types and for each position in a plurality of positions in the video frame, determining a confidence score indicating a likelihood that the position in the video frame includes a feature of the video feature type; Determining a weight for each video feature type based on the video category, the weight reflecting the importance of not obstructing video features of the video feature type when displaying a video of the video category; as well as For each video feature type in the set of video feature types, adjusting the confidence scores of the plurality of locations in the video frame based on the weight determined for the video feature type to generate an adjusted confidence score; aggregating the adjusted confidence score for each position of each video frame in the set of sampled video frames to generate an aggregated and adjusted confidence score; determining a location at which to place overlay content during display of the video based on the aggregated and adjusted confidence scores; as well as Overlay content is provided for display at the determined location in the video.

2. The computer-implemented method of claim 1, wherein the video feature types include human faces, human torsos, text, moving objects, or changes in color variance. 3 . The computer-implemented method of claim 1 , wherein at least some confidence scores for different video feature types are determined in parallel.

4. The computer-implemented method of claim 1 , wherein adjusting the confidence scores of the plurality of locations in the video frame for each of the set of video feature types comprises adjusting the confidence scores of locations in the video frame that are within a predefined neighborhood of a center of the video frame.

5. A computer-implemented method according to claim 1, wherein determining the location at which the overlay content will be placed during the video display based on the aggregated and adjusted confidence scores also includes determining the location at which the overlay content will be placed during the video display based on a specified size of the overlay content and a frame duration or number of frames within which the overlay content will be provided within the video. 6 . The computer-implemented method of claim 1 , wherein determining a location at which the overlay content is to be placed comprises determining a plurality of locations, and providing the overlay content at the determined location comprises providing the overlay content for display at one of the plurality of locations.

7. The computer-implemented method of claim 1, further comprising sampling, by a video processing system, the video based on a sampling rate to obtain the set of sampled video frames, wherein the sampling rate is determined based on a processing capability of the video processing system.

8. A system for non-occluded video overlay, comprising: one or more memory devices storing instructions; as well as One or more data processing apparatuses are configured to interact with the one or more memory devices and to perform operations including the following when executing instructions: identifying, for a video with which the overlay content is to be displayed, a video category of the video from a set of predefined video categories; For each video frame in a set of sampled video frames of the video: For each video feature type in a set of video feature types and for each position in a plurality of positions in the video frame, determining a confidence score indicating a likelihood that the position in the video frame includes a feature of the video feature type; Determining a weight for each video feature type based on the video category, the weight reflecting the importance of not obstructing video features of the video feature type when displaying a video of the video category; as well as For each video feature type in the set of video feature types, adjusting the confidence scores of the plurality of locations in the video frame based on the weight determined for the video feature type to generate an adjusted confidence score; aggregating the adjusted confidence score for each position of each video frame in the set of sampled video frames to generate an aggregated and adjusted confidence score; determining a location at which to place overlay content during display of the video based on the aggregated and adjusted confidence scores; as well as Overlay content is provided for display at the determined location in the video.

9. The system of claim 8, wherein the video feature types include human faces, human torsos, text, moving objects, or changes in color variance.

10. The system of claim 8, wherein at least some confidence scores for different video feature types are determined in parallel.

11. The system of claim 8, wherein adjusting the confidence scores of the plurality of locations in the video frame for each of the set of video feature types comprises adjusting the confidence scores of locations in the video frame that are within a predefined neighborhood of a center of the video frame.

12. The system of claim 8, wherein determining a location at which superimposed content will be placed during video display based on aggregated and adjusted confidence scores further comprises determining a location at which superimposed content will be placed during video display based on a specified size of the superimposed content and a frame duration or number of frames within which the superimposed content will be provided within the video.

13. The system of claim 8, wherein determining the location at which the overlay content is to be placed comprises determining a plurality of locations, and providing the overlay content at the determined location comprises providing the overlay content for display at one of the plurality of locations.

14. The system of claim 8, wherein the operations further comprise sampling, by a video processing system, the video based on a sampling rate to obtain the set of sampled video frames, wherein the sampling rate is determined based on a processing capability of the video processing system.

15. A computer readable medium storing instructions which, when executed by one or more data processing devices, cause the one or more data processing devices to perform operations comprising: identifying, for a video with which the overlay content is to be displayed, a video category of the video from a set of predefined video categories; For each video frame in a set of sampled video frames of the video: For each video feature type in a set of video feature types and for each position in a plurality of positions in the video frame, determining a confidence score indicating a likelihood that the position in the video frame includes a feature of the video feature type; Determining a weight for each video feature type based on the video category, the weight reflecting the importance of not obstructing video features of the video feature type when displaying a video of the video category; as well as For each video feature type in the set of video feature types, adjusting the confidence scores of the plurality of locations in the video frame based on the weight determined for the video feature type to generate an adjusted confidence score; aggregating the adjusted confidence score for each position of each video frame in the set of sampled video frames to generate an aggregated and adjusted confidence score; determining a location at which to place overlay content during display of the video based on the aggregated and adjusted confidence scores; as well as Overlay content is provided for display at the determined location in the video.

16. The computer-readable medium of claim 15, wherein the video feature types include human faces, human torsos, text, moving objects, or changes in color variance.

17. The computer-readable medium of claim 15, wherein at least some confidence scores for different video feature types are determined in parallel.

18. The computer-readable medium of claim 15, wherein adjusting the confidence scores of the plurality of locations in the video frame for each of the set of video feature types comprises adjusting the confidence scores of locations in the video frame that are within a predefined neighborhood of a center of the video frame.

19. The computer-readable medium of claim 15, wherein determining a location at which superimposed content will be placed during video display based on aggregated and adjusted confidence scores further comprises determining a location at which superimposed content will be placed during video display based on a specified size of the superimposed content and a frame duration or number of frames within which the superimposed content will be provided within the video.

20. The computer-readable medium of claim 15, wherein determining the location at which the overlay content is to be placed comprises determining a plurality of locations, and providing the overlay content at the determined location comprises providing the overlay content for display at one of the plurality of locations.

Images