System and method for overlaying video on video

Abstract

This provides a system and method for overlaying video onto other video. [Solution] The second video may be overlaid on top of or inserted into the first video in a non-intrusive manner. The position of the second video within the first video may be determined based on an analysis of non-essential positions within consecutive frames of the first video. The second video may be selected from a group of candidate videos based on parameters of the second video such as size, duration, and context. Thus, the second video may be displayed simultaneously with the first video by nesting it within the first video, providing a streamlined viewing experience and minimizing distractions.

Description

Background Art

[0001] Video advertisements are often placed before the start or after the end of a video. Sometimes, to present an advertisement, the video is interrupted, and when the advertisement is completed, the video is resumed, so that the advertisement is intermittently placed throughout the video. This interruption disrupts the viewer's viewing experience, giving the viewer the opportunity to completely abandon watching the video or to ignore the advertisement until the video resumes. This creates user frustration and an impaired viewing experience.

Summary of the Invention

Means for Solving the Problems

[0002] The present disclosure provides covering a first video that plays desired content for a viewer with a second video such as a video advertisement. The first and second videos may be played simultaneously. The placement of the second video within the first video may be determined by analyzing positions within the first video and selecting the least distracting location for covering the second video. Such analysis may include comparing consecutive frames and identifying positions with minimal change and that do not depict a particular object such as a person or an animal. Such positions may be candidate positions for the second video and may each have an associated set of parameters such as size, length of time, context of the content within the relevant frame of the first video, etc. The second video may be selected based on one or more of such parameters.

[0003] One aspect of the present disclosure provides a method for placing a second video within a first video. The method may include the steps of: identifying candidate frame positions in a sequence of frames within a first video using one or more processors; comparing candidate frame positions among frames in the sequence of frames using one or more processors; selecting candidate video positions based on the comparison using one or more processors, wherein the candidate video positions include frame positions that are consistent among the candidate frame positions in the sequence of frames; selecting a second video based on one or more parameters associated with the candidate video positions; and inserting the second video within the candidate video positions of the first video for simultaneous playback with the first video. The step of identifying candidate frame positions may include identifying two-dimensional positions such that the difference between content displayed at the two-dimensional position in the first frame and content displayed at the two-dimensional position in the second frame is less than a threshold. The method may further include the step of dividing each frame into a grid containing a plurality of cells, wherein the two-dimensional positions include a subset of the plurality of cells.

[0004] In some examples, the step of identifying candidate frame locations may include, for each frame, calculating a local average for every pixel; for each row of the frame, storing the length of the longest line whose local average remains above a first threshold; and for each column of the frame, storing the largest rectangle whose local average remains above a second threshold. The method may further include identifying stored rectangles having an area greater than the minimum area required by the second video. The method may further include identifying one or more sequences of stored rectangles having consecutive locations in the first video across at least N consecutive frames. Furthermore, the method may include calculating weights for each sequence of stored rectangles, and the step of selecting candidate video locations is based on these weights.

[0005] In some examples, the method may further include the steps of storing candidate video positions and dynamically selecting a second video during playback of a first video based on at least one of the size or duration of the candidate video positions.

[0006] The second video may be selected from multiple candidate videos based on the duration of the candidate video position in the frame sequence and the duration of the second video.

[0007] In some examples, the method may further include the steps of identifying the context of a first video and selecting a second video from a group of candidate videos based on the context of the second video that corresponds to the identified context of the first video.

[0008] In some examples, the method may further include the step of annotating a first video with a first tag corresponding to a specified context, and the step of selecting a second video includes identifying a second tag of the second video that overlaps with the first tag. Such annotation of the first video with the first tag may include annotating the first video with tags relevant to the first video as a whole. In other examples, such annotation of the first video with the first tag may include annotating the first video with tags relevant to individual moments or frames within the first video.

[0009] In some examples, the method may further include a step of comparing candidate frame locations, which involves running a machine learning model. The step of training the machine learning model may include collecting training samples, wherein the training samples include a set of frames in a first video containing annotated sequences of non-essential areas, and feeding the training samples into a deep neural network classifier, thereby training the classifier to assess, for each point on the set of frames, how likely each point is to be a non-essential location. The machine learning model may be trained to identify the presence of humans in the video and exclude locations in frames containing humans from the candidate frame locations.

[0010] Another aspect of the present disclosure provides a system including memory and one or more processors communicating with the memory. The one or more processors may be configured to identify candidate frame positions in a sequence of frames in a first video, compare candidate frame positions among frames in the sequence of frames, select candidate video positions based on the comparison, wherein the candidate video positions include frame positions that are consistent among the candidate frame positions in the sequence of frames, select a second video based on one or more parameters associated with the candidate video positions, and insert the second video into the candidate video positions of the first video for simultaneous playback with the first video.

[0011] Candidate frame positions may include two-dimensional positions, where the difference between the content displayed at the two-dimensional position in the first frame and the content displayed at the two-dimensional position in the second frame is less than a threshold. When identifying candidate frame positions, one or more processors may be further configured to calculate a local average value for every pixel for each frame. For each row of a frame, one or more processors may store the length of the longest line for which the local average value remains above a first threshold, and for each column of a frame, they may store the longest rectangle for which the local average value remains above a second threshold.

[0012] The second video may be selected from multiple candidate videos based on the duration of the candidate video position in the frame sequence and the duration of the second video.

[0013] Another aspect of the present disclosure provides a non-temporary computer-readable medium for storing instructions executable by one or more processors to perform a method. Such a method may include the steps of: identifying candidate frame locations in a sequence of frames in a first video; comparing the candidate frame locations among frames in the sequence of frames; selecting a candidate video location based on the comparison, wherein the candidate video location includes frame locations that are consistent among the candidate frame locations in the sequence of frames; selecting a second video based on one or more parameters associated with the candidate video location; and inserting the second video into the candidate video location of the first video for simultaneous playback with the first video. [Brief explanation of the drawing]

[0014] [Figure 1A] This figure shows examples of candidate frame positions within various frames of a first video according to the embodiments of this disclosure. [Figure 1B] This figure shows examples of candidate frame positions within various frames of a first video according to the embodiments of this disclosure. [Figure 1C]This figure shows an example of identifying candidate video positions for potential insertion of a second video into a first video, based on comparing candidate frame positions of various frames, according to an aspect of this disclosure. [Figure 1D] This figure shows an example of a second video inserted into a first video for simultaneous playback, according to the embodiments of this disclosure. [Figure 2A] This figure shows an example of a candidate frame position identified using a cell grid according to an aspect of this disclosure. [Figure 2B] This figure shows an example of a candidate frame position identified using a cell grid according to an aspect of this disclosure. [Figure 2C] This figure shows an example of a candidate frame position identified using a cell grid according to an aspect of this disclosure. [Figure 3] This block diagram shows the training of a machine learning model according to the manner of disclosure. [Figure 4] This block diagram shows the execution of a machine learning model for identifying candidate video locations according to the aspects of this disclosure. [Figure 5] This figure shows an example of multiplying weights assigned to candidate frame positions identified based on different methods, according to an aspect of this disclosure. [Figure 6] This figure shows an example of identifying a candidate video position based on a candidate frame position according to an aspect of this disclosure. [Figure 7] This figure shows an example of selecting a second video based on the parameters of the candidate video position according to an aspect of this disclosure. [Figure 8] This figure shows another example of selecting a second video based on the parameters of the candidate video position according to an aspect of this disclosure. [Figure 9] This figure shows another example of selecting a second video based on the parameters of the candidate video position according to an aspect of this disclosure. [Figure 10] Block diagram illustrating an exemplary system according to the aspects of this disclosure. [Figure 11] This is a flowchart illustrating an exemplary method according to an aspect of the present disclosure.

Best Mode for Carrying Out the Invention

[0015] This disclosure generally relates to overlaying one video on another in a non-intrusive manner. This disclosure provides systems and methods for identifying candidate regions within a first video. This disclosure further provides systems and methods for selecting a second video to overlay on the first video based on the identified candidate regions.

[0016] When determining where and when to overlay a second video on the first video, a frame range within the first video may be selected for placement of the second video, or a two-dimensional position on a frame within the frame range may be selected. The selection of the frame range and frame position may be performed using a low user disturbance technique. Additionally or alternatively, the selection of the frame range and frame position may be performed using a contextual moment selection technique.

[0017] In a low user disturbance technique, video positions are identified as being non-essential to the viewer. For example, a non-essential position may be one that minimally or not at all obstructs the content of the first video that the viewer is likely to want to see. For example, if the first video depicts a person playing soccer, a non-essential position may be one that does not block the person, ball, or goal, but instead may cover depictions of the sky, grass, etc. In some examples, non-essential positions may be scored based on how obstructive or distracting they are, with a better score correlating to being less obstructive or distracting.

[0018] Calculating candidate frame positions, such as non-essential areas within individual frames, may be performed using various techniques such as static content detection techniques, machine learning techniques, hybrid techniques, etc.

[0019] According to static content detection methods, given a sequence of video frames, the difference between consecutive video frames is calculated. Each resulting frame may be divided into a grid, and for each cell in the grid, the sum of the differences between consecutive frames is calculated.

[0020] According to machine learning techniques, a set of video frames is selected to serve as training samples. Frames may be selected randomly or based on various features, etc. Each of the selected video frames may be manually scrutinized to identify candidate frame locations or non-essential locations. Such candidate frame locations may be annotated. In other examples, the training sample video may be manually scrutinized within or throughout a segment to identify sequences of non-essential areas or candidate video locations, and those locations may be annotated. Annotated frames and / or videos may be used to train a deep neural network classifier to assess, for each point on the frame, how likely it is that that point is a non-essential location.

[0021] In a hybrid approach, a machine learning model may be used to provide additional information. For example, the machine learning model may be trained to identify essential elements of the first video, and all locations containing such essential elements may be excluded from consideration as candidate locations in the second video. For example, one possible heuristic is that a person is an essential element of the first video. The classifier may calculate the probability that each location in the frame contains a person.

[0022] As a result of any of the above methods, positions within a frame or video segment may be scored or weighted. For example, weights may be assigned to each cell in a grid, each pixel, or any other area of ​​a frame defined in any way. The weights may correspond to the likelihood of an area being non-essential, such as not interfering with or distracting the placement of a second video. In some examples, weighted results from different methods may be combined.

[0023] Candidate video positions for a second video may be identified within the first video using candidate frame positions. For example, calculated weights for each pixel, cell, or other area may be used to calculate a local average for every pixel, cell, or other area in the frame. This may blur the image and make the determination of candidate positions less susceptible to noise. For each row of the frame, the length of the longest line whose local average does not fall below a first threshold may be stored. For each column of the frame, the largest rectangle whose local average does not fall below a second threshold may be stored. For each frame, the N largest rectangles based on such calculations may be stored.

[0024] In some examples, candidate video positions may be selected based on the requirements of the second video. For example, such requirements may include size, length, etc. Candidate frame positions may be selected that are larger than the minimum area required by the second video, have a minimum number of consecutive frames corresponding to the duration of the second video, and so on. For each sequence of candidate frame positions, a weight may be calculated as the average weight of all areas encompassed by the candidate frame position in all frames of the sequence. From those sequences, the sequence that best fits the criteria of the second video while having the largest weight for each size / duration pair may be selected as the video position for the second video. The selected video position may be stored for retrieval during playback so that the second video may be dynamically selected, retrieved, and inserted into the first video.

[0025] Contextual moment selection may involve identifying the context of a particular frame into which a second video is inserted, and correlating such context with the context of the second video. For example, frames in the first video may be annotated with relevant contextual tags. Such contextual tags may indicate that the content within the frame is relevant to a particular theme, subject, or object. For example, contextual tags may indicate that the frame is relevant to a particular sport, activity, product, or state of mind. The second video may also be assigned relevant tags. Corresponding tags between frames in the first video and the second video may be identified, and based on such corresponding tags, the second video may be selected for insertion into the first video. Such corresponding tags may overlap, for example, if some subset of tags match between frames in the first video and the second video. In another example, such corresponding tags may be complementary, for example, if the contextual tags in the first video include food and the tags in the second video include dining room decor.

[0026] Figure 1A shows an exemplary video frame 110 of a first video. Video frame 110 may be one of several consecutive frames that together form the first video. Each frame 110 may contain content 120, such as objects, people, or other features depicted in the video. A video progress bar 130 may indicate the length or duration of the first video and the point in time within the duration corresponding to the video frame being viewed at a given moment. For example, a progress indicator 132 may move from one end of the progress bar 130, such as the left edge, to the other end. In some examples, as the progress indicator 132 moves, a completion indicator 134 may follow to indicate which portion of the video has already been viewed. For example, the completion indicator 134 may make the progress bar 130 a different color, shape, thickness, etc., with respect to the portion of the progress bar 130 corresponding to a frame earlier than the frame 110 corresponding to the progress indicator 132.

[0027] A first candidate frame position 150 of video frame 110 may be identified. The candidate frame position 150 may correspond to an area within the frame that is not essential, unobtrusive, or distracting. For example, such an area may not significantly overlap with the main features of the content 120. The candidate frame position 150 may encompass an area of ​​any of a variety of shapes and sizes. In this example, one candidate frame position 150 is shown, but in other examples, a frame may have multiple candidate frame positions.

[0028] Figure 1B shows the second video frame 111 of the first video, which is later in the sequence of frames that make up the first video, as indicated by the movement of the indicator 132 on the video progress bar 130. In this second video frame 111, the content 120 has changed to move to a different area within the second frame 111 compared to the first frame 110. Therefore, candidate frame position 151 may have a different size, shape, and / or position compared to candidate frame position 150 in the first frame 110, as the non-essential area in the second frame differs based on the change in the displayed content 120. Furthermore, in this example, an additional candidate frame position 152 is available in another part of the second frame 111, to the left of the content 120.

[0029] Figure 1C shows an example of comparing candidate frame positions 150, 151, and 152 across multiple frames. While this example only shows a few candidate frame positions from two frames, the comparison may include any number of candidate frame positions from multiple consecutive frames. When comparing candidate regions, it is determined which regions are consistent across multiple consecutive frames. In the example shown, region 155 may be consistent across multiple consecutive frames. Therefore, region 155 may be identified as a candidate video position for a second video.

[0030] Identifying candidate frame positions in each frame may be performed using one of several techniques. Several examples of such techniques are described below in relation to Figures 2-5.

[0031] Figure 1D shows an example of inserting a second video within a candidate video position identified in the first video. As shown, the second video 190 contains second content which may differ from the content 120 of the first video. The second video 190 may have a duration equal to or shorter than the duration of the first video. For example, the duration of the second video may be between a subset of frames of the first video. In some examples, the second video 190 may pop up within the first video and disappear automatically when the second video 190 finishes. In other examples, the second video 190 may contain an icon 192 or other interactive element which may be used to close the second video 190 so that only the first video is shown. In further examples, the second video 190 may contain other interactive elements such as controls for adjusting the speed of the second video 190, controls for pausing the second video 190, controls for annotating the second video 190, and game elements.

[0032] Figures 2A and 2C illustrate an example of a static content detection method. This method calculates the difference between consecutive video frames, given a sequence of video frames. Each resulting frame may be divided into a grid, and for each cell in the grid, the sum of the differences between consecutive frames is calculated.

[0033] As shown in Figure 2A, the first frame 110 is divided into a grid of cells 202. Similarly, as shown in Figures 2B-2C, frames 111-112 are also divided into a grid of cells. Cells may have a consistent size with respect to each frame of the first video, but their size may differ in the analysis of one video from the size in the analysis of another video. In some examples, each cell may be one pixel. In other examples, each cell may contain multiple pixels. Cells may be compared between frames 110-112 to calculate the difference between cells. Such differences may include changes in the content depicted in each cell, such as changes in color, light, or subject. Cells with minimal frame-to-frame differences may be considered to correspond to non-essential areas of the frame. In some examples, cells spanning frames with minimal differences may be assigned a value, but values ​​below a threshold indicate the possibility that the cell is included in a candidate frame location. A set of adjacent cells, all with values ​​below a threshold, may form a candidate frame location.

[0034] In some examples, a calculated weight for each pixel, cell, or other area may be used to calculate a local average for every pixel, cell, or other area in the frame. This may blur the image and make the determination of candidate positions less susceptible to noise. For each row of the frame, the length of the longest line whose local average does not fall below a first threshold may be stored. For each column of the frame, the largest rectangle whose local average does not fall below a second threshold may be stored. For each frame, the N largest rectangles based on such calculations may be stored.

[0035] In some cases, machine learning models may be used to identify candidate frame locations within each frame and / or candidate video locations that include candidate frame locations across multiple consecutive frames. Figure 3 shows an example of training machine learning model 330.

[0036] The machine learning model 330 may be, for example, a neural network, a convolutional neural network, a generalized additive model, or any of various other types of machine learning models. To generate a model, the machine learning engine may acquire feature data and signal data associated with several content item impressions and train the model using the feature data and corresponding signal data for each historical content item impression. For example, the machine learning engine may consist of regression analysis techniques that use statistical processes to estimate the relationship between a feature or combination of features and a signal. The feature or combination of features can be a predictor or an independent variable, and the signal can be a dependent or criterion variable that changes when the feature is changed. In some cases, the machine learning engine may estimate or determine the conditional expectation of the dependent variable given an independent variable, such as the mean of the dependent variable when the independent variable is fixed, or other parameters or indicators of the conditional distribution of the dependent variable or signal given an independent variable or feature. The predicted signal is a function of the independent variable and may be called a regression function. Furthermore, machine learning engines can identify, determine, or characterize the variability of a dependent variable around a regression function that can be described by a probability distribution. The machine learning engine can then use the probability distribution to generate a confidence score for the predicted signal value, or it can use the probability distribution as the confidence score.

[0037] In the example shown, the machine learning model 330 may be trained using multiple training samples 310-312. While only a few training samples are shown, it should be understood that any number of training samples may be used. In this example, each training sample 310-312 is tagged with information identifying candidate frame positions 350-352. Such tagging may be performed by a manual process. Tagging may include annotating each frame, for example, by drawing or highlighting an area of ​​the frame. In other examples, tagging may include updating metadata associated with the image to identify the position within the image that is the candidate frame position. For example, the position may be identified using coordinates, pixel identifiers, or any other information.

[0038] Training samples 310-312 may be input to the machine learning model 330. Based on such input, the machine learning model 330 may learn to recognize candidate frame locations. For example, the machine learning model 330 may recognize patterns in the annotated candidate frame locations 350-352 of the training samples 310-312 and learn to detect such patterns in other frames to identify candidate frame locations. The machine learning model 330 may provide an output that identifies the candidate frame locations. According to some examples, the output may be evaluated as part of a feedback loop used to update or fine-tune the machine learning model 330.

[0039] Model 100 can repeat the training steps multiple times until one or more stopping criteria are met. These stopping criteria may include, for example, the maximum number of iterations of the training steps, and / or, for supervised or semi-supervised learning, the number of iterations of backpropagation, gradient descent, and model parameter updates. Additionally or alternatively, stopping criteria may define the minimum improvement between training steps. For semi-supervised training, an example might be the relative or absolute decrease in the calculated error between the output predicted by Model 100 and the corresponding ground-truth labels on the training data set aside for validation and / or testing. For unsupervised learning, an exemplary loss function might be least squares.

[0040] In some examples, Model 100 may be trained for a large number of epochs, e.g., 1000 epochs, with early termination occurring if the error between the predicted output and the ground truth error does not improve over 10 epochs. Other termination criteria may be based on the maximum amount of computing resources allocated for training, e.g., the amount of training time exceeded, e.g., the total amount of wall clock time, or the total number of processing cycles consumed, after which training is terminated.

[0041] As shown in Figure 4, a trained machine learning model may be used to predict candidate video locations within a video input. For example, when a video is input, the machine learning model may identify candidate frame locations within a series of consecutive frames by assessing how likely each point on the frame is to be a non-essential location. The machine learning model may further identify candidate video locations based on consistent candidate frame locations across multiple consecutive frames. If more than one candidate video location is detected, the model may identify each candidate video location. In some examples, multiple candidate video locations may be scored and / or ranked. In other examples, parameters of the candidate video locations, such as duration and size, may be provided with the output.

[0042] In some examples, machine learning models may be trained to detect other types of information. For instance, a machine learning model may be trained to predict essential areas within a frame, such as areas depicting people, animals, or featured objects. Such essential areas may therefore be excluded from consideration when determining candidate frame locations.

[0043] In some cases, a hybrid of multiple different methods may be used. For example, Figure 5 shows an example of a hybrid method. For instance, frame 510 may be evaluated using a combination of two methods. Frame 512 represents frame 510 evaluated using the static content detection method described above in relation to Figures 2A-2C. Frame 514 represents frame 510 evaluated using a machine learning model, as described above in relation to Figures 3-4. Evaluating pixels, cells, or areas of frames 512, 514 under each method may involve assigning weights to each pixel, cell, or area, where the weights indicate the likelihood that the pixel / cell / area is an area that is not essential. The weights assigned to a given portion of each frame 512, 514 may differ based on the evaluation technique. For example, a particular pixel in frame 512 may be weighted by value x based on the evaluation of static content detection, while the same particular pixel in frame 514 may be weighted by value y based on the evaluation of machine learning. In this example, the weights are multiplied so that both evaluation techniques are taken into consideration. Using the multiplied weights for each pixel / cell / area, a candidate frame position 550 can be identified with respect to frame 510.

[0044] Candidate video positions for a second video may be identified within the first video using candidate frame positions. Figure 6 shows an example of identifying candidate video positions based on candidate frame positions. A first candidate frame position 150 of video frame 110 may be identified. For example, candidate frame position 150 may be determined by comparing a pixel or other sub-part of a frame with a corresponding pixel or sub-part of more or more adjacent frames. Weights may be assigned to each pixel or other sub-part based on the degree of difference between frames. In some examples, the calculated weights for each pixel, cell, or other area may be used to calculate a local average for every pixel, cell, or other area of ​​the frame. For each row of a frame, the length of the longest line whose local average does not fall below a first threshold may be stored. For each column of a frame, the largest rectangle whose local average does not fall below a second threshold may be stored. For each frame, N largest rectangles based on such calculations may be stored. The second video frame 111 of the first video is similarly analyzed, and the second video frame 111 is later in the sequence of frames that make up the first video. Furthermore, each frame between the first frame and the second frame may be similarly analyzed to determine the N largest rectangles within each frame. The N largest rectangles within each frame may be compared to determine which positions within the frame are consistent among the rectangles of at least the lower portion of the first video. In some examples, slight discrepancies may be acceptable, such as when one or more frames in the sequence have small deviations with respect to the N largest rectangles. In this example, the candidate video position 155 represents the region that was consistent within the N largest rectangles in each of the frames in the sequence corresponding to a time span of 135.

[0045] In some cases, a second video inserted into or overlaid on a first video may be selected from multiple videos. For example, the second video may be an advertisement inserted into the first video. The first video may be, for example, streaming content. Various advertisements may be available. In some cases, different advertisements may be selected for each playback of streaming content by the same or different users.

[0046] Figure 7 shows an example where a second video is selected based on size. For example, candidate video position 760 may have a defined size and shape. Candidate second videos 781, 782, and 783 may each have different sizes. Some of the candidate second videos, such as candidate second video 783, may be larger than candidate video position 760. Such larger second videos may be excluded from selection because they encompass a portion of the first video beyond the defined candidate position 760. Candidate second videos 781 and 782 are both suitable, so either may be selected. According to some examples, the larger second video 782 may be selected to utilize a large portion of candidate video position 760.

[0047] Figure 8 shows an example of selecting a second video based on the length of time a candidate video position is available within the first video. As shown, the candidate video position 835 within the first video has a duration of 15 seconds. Therefore, one of the candidate second videos 881-883 having a duration of 15 seconds or less may be selected. Other second videos with longer durations may be excluded.

[0048] Figure 9 shows an example of selecting a second video based on the content of the first video. A context relevant to a specific frame of the first video into which the second video is inserted may be determined, or a context relevant to other parts or the entirety of the first video may be determined. In some examples, frames of the first video may be annotated with relevant contextual tags. Tagging may be performed manually, using image or object recognition techniques, or using machine learning. Such contextual tags may indicate that the content within the frame is relevant to a particular theme, subject, or object. For example, contextual tags may indicate that a frame is relevant to a particular sport, activity, product, or state of mind. Candidate second videos 981-983 may also be assigned relevant tags. One of the second candidate videos 981-983 may be selected based on the number of overlapping tags. For example, the second video with the most matching or corresponding tags to those of the first video may be selected. In the example shown, frame 910, which contains the candidate video position, is related to soccer and may therefore be tagged with indicators such as soccer, sports, field, outdoors, athletics, determination, motivation, and practice. Candidate second video 981 may be related to cooking and may therefore be tagged with food, cooking, tools, pasta, butter, etc. Candidate second video 982 may be related to sneakers and may be tagged with sports, athletics, comfort, style, and blue, etc. Candidate second video 983 may be related to furniture and may be tagged with wood, mahogany, traditional, and luxury, etc. Since candidate second video 982 contains at least two tags that overlap with the tags of the first video—sports and athletics—candidate second video 982 may be selected to be included in the candidate video position.

[0049] Although size, duration, and context are shown separately in the example above, the second video may be selected based on a combination of these and / or other parameters. These parameters can be used to dynamically select the second video for insertion into the first video.

[0050] Figure 10 shows an exemplary system for performing candidate video location identifiers and the selection of a second video. In particular, the system includes one or more client devices 1001, 1002, 1003 that communicate with one or more servers 1020 through network 1050. For example, each of several different client devices 1001-1003 may receive first video content from server 1020. The server may determine where in the first video a second video can be inserted to minimize interruption during simultaneous playback, and may select the second video to be inserted. While several client devices 1001-1003 are shown, it should be understood that any number of client devices may communicate with one or more servers 1020 through network 1050.

[0051] Server 1020 includes one or more processors 1070. A processor 1070 can be any ordinary processor, such as a commercially available CPU. Alternatively, a processor can be a specialized component, such as an application-specific integrated circuit ("ASIC") or other hardware-based processor. While not required, Server 1020 may include specialized hardware components for performing specific computing processes.

[0052] Memory 1060 can store information accessible by processor 1070, including instructions that can be executed by processor 1070 and can be retrieved, manipulated, or stored by processor 1070.

[0053] Instructions can be a set of instructions that are executed directly by the processor 1070, such as in machine code, or indirectly, such as in a script. In this regard, the terms “instruction,” “step,” and “program” may be used interchangeably herein. Instructions may be stored in an object code format for direct processing by the processor 1070, or in a script or other type of computer language, including a set of independent source code modules that are interpreted on demand or pre-compiled. The functions, methods, and routines of instructions will be described in more detail in the examples above and in the illustrative manner below.

[0054] The data may be retrieved, stored, or modified by the processor 1070 according to instructions. The data may also be formatted in a computer-readable format, such as binary, ASCII, or Unicode, but not limited to these. Furthermore, the data may contain enough information to identify relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to other data stored in memory including other network locations, or information used by functions for calculating the relevant data.

[0055] Figure 10 functionally shows the processor, memory, and other elements of server 1020 as being in the same block; however, a processor, computer, computing device, or memory may actually include multiple processors, computers, computing devices, or memory that may or may not be housed in the same physical housing. For example, memory could be a hard drive or other storage medium located in a different housing from that of server 1020. Thus, references to processors, computers, computing devices, or memory are understood to include references to a collection of processors, computers, computing devices, or memory that may or may not operate in parallel. For example, server 1020 may include server computing devices operating as a load-balanced server farm, distributed system, etc. Furthermore, while some functions described below are shown as being performed on a single computing device having a single processor, various aspects of the subject matter described herein may be performed by multiple computing devices, for example, transmitting information over a network.

[0056] Memory 1060 can store information accessible by processor 1070, including instructions 1062 that can be executed by processor 1070. Memory may also include data 1064 that can be retrieved, manipulated, or stored by processor 1070. Memory 1060 may be any type of non-primary computer-readable medium that can store information accessible by processor 1070, such as a hard drive, solid-state drive, tape drive, optical storage, memory card, ROM, RAM, DVD, CD-ROM, write-capable, and read-only memory. Processor 1070 can be a well-known processor or any other less well-known type of processor. Alternatively, processor 1070 can be a dedicated controller such as an ASIC.

[0057] Instruction 1062 can be a set of instructions that are executed directly by the processor 1070, such as machine code, or indirectly, such as a script. In this regard, the terms “instruction,” “step,” and “program” may be used interchangeably herein. Instruction 1062 may be stored in an object code format for direct processing by the processor 1070, or in a script or other type of computer language, including a set of independent source code modules that are interpreted on demand or pre-compiled. Instruction 1062 may be executed to identify a region in a sequence of video frames for inserting a second video, as described above. Furthermore, instruction 1062 may be executed to select a second video for insertion into or overlay on a first video during simultaneous playback of the first and second videos.

[0058] Data 1064 may be retrieved, stored, or modified by processor 1070 in accordance with instruction 1062. For example, although the system and method are not limited by a specific data structure, data 1064 may be stored in a computer register, in a relational database as a table with multiple different fields and records, or in an XML document. Data 1064 may also be formatted in a computer-readable format, such as binary values, ASCII, or Unicode, but not limited to these. Furthermore, data 1064 may contain enough information to identify relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to other data stored in memory including other network locations, or information used by functions for calculating the relevant data.

[0059] Server 1020 may be further coupled to external storage 1080, such as a database. External storage 1080 may store content for distribution to client devices 1001-1003. External storage 1080 may also store banners for rendering on client devices 1001-1003 along with the content. Such banners may contain advertisements or other information. Although external storage 1080 is presented as a single database, it should be understood that the physical structure of external storage 1080 can include multiple storage devices, and such multiple devices may communicate with each other like a distributed storage system.

[0060] Each client device 1001, 1002, and 1003 may be configured similarly to each other and to the server 1020, in that each includes a processor 1091 and memory 1092 containing data 1093 and instructions 1094 that can be executed by the processor 1091. The structure of the processor 1091 and memory 1092 may be similar to the structure of the processor 1070 and memory 1060 described above, respectively. Client devices 1001-1003 may be any type of personal computing device, such as a laptop, desktop computer, tablet, game console, telephone, augmented reality or virtual reality headset, smartwatch, smart glasses, home assistant hub, or any other computing device including a display for outputting content along with one or more banners. Each client device 1001-1003 may further include one or more user input devices 1095. Such user input devices 1095 may include a touchscreen, touchpad, keypad, camera, microphone, joystick, or any other device adapted to capture input signals from the user.

[0061] In addition to the exemplary systems described above, exemplary methods are described below. Such methods may be performed using the systems described above, modifications thereof, or various systems with different configurations. It should be understood that the operations included in the following methods are not necessarily performed in the strict order described. Rather, various operations may be processed in different orders or simultaneously, and operations may be added or omitted.

[0062] Figure 11 illustrates an exemplary method for determining the placement of a second video within a first video. The method may be performed, for example, by a computing device or module, or by a system of one or more distributed server processes. While the operations are described in a specific order, it should be understood that the order may be modified. Furthermore, operations may be performed concurrently, and operations may be added or omitted.

[0063] In block 1110, the candidate frame position is identified within each of several consecutive frames in the first video.

[0064] In block 1120, candidate frame positions are compared across consecutive frames. The comparison determines where within the sequence of frames a consistent frame position for the second video exists. Identified candidate frame positions may be weighted according to a confidence level that includes areas where the position is not essential.

[0065] In block 1130, a candidate video position is selected based on the candidate frame position, for example, by identifying a portion of the frame that consistently contains the candidate frame position across multiple adjacent frames.

[0066] In block 1140, a second video is selected based on several parameters, including but not limited to size, length, and context. Once the candidate video position and candidate frame position are determined, they reveal specific parameters that must be met by the second video. There are also additional parameters that must be met, which may be indicated, for example, by the ad provider or video provider.

[0067] In block 1150, the selected video is inserted within the selected video position and selected frame range. Then, the second video may be played simultaneously with the first video.

[0068] While some of the examples above refer to overlaying video ads on a video for simultaneous playback, the techniques described above may be similarly applied to overlaying other types of videos on a first video for simultaneous playback.

[0069] Unless otherwise noted, the alternative examples described above are not mutually exclusive and may be implemented in various combinations to achieve their own advantages. Since these and other variations and combinations of the features discussed above may be utilized without departing from the subject matter defined by the claims, the foregoing description should be taken as illustrative rather than limiting to the subject matter defined by the claims. Furthermore, the provision of examples described herein, and sections expressing "such as..." or "including...", should not be interpreted as limiting the subject matter of the claims to specific examples; rather, the examples are intended to illustrate only one of many possible examples. Moreover, the same reference number in different drawings may identify the same or similar elements. [Explanation of Symbols]

[0070] 110 video frames 111 Second video frame 112 frames 120 contents 130 Video Progress Bar 134 Completion Indicator 150 First candidate frame position 151 Candidate frame positions 152 Additional candidate frame positions 155 regions, candidate video positions 190 Second video 192 icons 202 cells 310-312 Training Samples 330 Machine Learning Models Candidate frame positions 350-352 510 frames 512 frames 514 frames 550 Candidate frame position 760 Candidate video location 781-783 Second video of the candidates 835 Duration 881-883 Second video of the candidates 910 frame 981-983 Second video of the candidates 1001-1003 Client Devices 1020 Servers 1050 Network 1060 memory 1062 instructions 1064 data 1070 processor 1080 External Storage 1091 processor 1092 memory 1093 data 1094 Instructions 1095 User Input Device

Claims

1. A method for placing a second video within a first video, The steps include: identifying candidate frame positions in a sequence of frames within the first video using one or more processors; The steps include: comparing the candidate frame positions among the frames in the sequence of frames using one or more processors; A step of selecting candidate video positions based on the comparison by one or more processors, wherein the candidate video positions include frame positions that are consistent among the candidate frame positions in the sequence of frames; The steps include selecting a second video based on one or more parameters related to the candidate video position, The steps include inserting the second video into the candidate video position of the first video for simultaneous playback with the first video, and Methods that include...

2. The step of identifying the candidate frame position includes identifying a two-dimensional position, The method according to claim 1, wherein the difference between the content displayed at the two-dimensional position in the first frame and the content displayed at the two-dimensional position in the second frame is less than a threshold.

3. The process further includes the step of dividing each frame into a grid containing multiple cells, The method according to claim 2, wherein the two-dimensional position includes a subset of the plurality of cells.

4. The step of identifying the candidate frame position is, Regarding each frame, Calculating the local average value for every pixel, For each row of the frame, the length of the longest line in which the local average value remains above a first threshold is stored. With respect to each column of the frame, the largest rectangle in which the local average value remains above the second threshold is stored. The method according to claim 1, including the method described in claim 1.

5. The method according to claim 4, further comprising the step of identifying a stored rectangle having an area larger than the minimum area required by the second video.

6. The method according to claim 4, further comprising the step of identifying one or more sequences of stored rectangles having consecutive positions in the first video over at least N consecutive frames.

7. The process further includes the step of calculating the weight for each sequence of stored rectangles, The method according to claim 6, wherein the step of selecting the candidate video position is based on the weights.

8. The steps include storing the candidate video positions, A step of dynamically selecting the second video during playback of the first video based on at least one of the size or duration of the candidate video position; The method according to claim 1, further comprising:

9. The method according to claim 1, wherein the second video is selected from a plurality of candidate videos based on the duration of the candidate video position in the sequence of frames and the duration of the second video.

10. The method according to claim 1, further comprising the steps of identifying the context of the first video and selecting the second video from a plurality of candidate videos based on the context of the second video corresponding to the identified context of the first video.

11. The step further includes annotating the first video with a first tag corresponding to the identified context, The method according to claim 10, wherein the step of selecting the second video includes identifying a second tag of the second video that overlaps with the first tag.

12. The method according to claim 11, wherein the step of annotating the first video with a first tag includes annotating the first video with a tag that is relevant to the entire first video.

13. The method according to claim 11, wherein the step of annotating the first video with a first tag includes annotating the first video with tags that are relevant to individual moments or frames within the first video.

14. The method according to claim 1, wherein the steps of identifying and comparing candidate frame positions include running a machine learning model.

15. A step of training the aforementioned machine learning model, Collecting training samples, wherein the training samples include a set of frames in the first video containing an annotated sequence of non-essential areas. The training samples are input into a deep neural network classifier, thereby training the deep neural network classifier to evaluate how likely it is that each point on the set of frames is an indispensable location. The method according to claim 14, further comprising the step of including claim 14.

16. The method according to claim 14, wherein the machine learning model is trained to identify the presence of a human in the video and to exclude the position in the frame containing the human from the candidate frame positions.

17. Memory and One or more processors that communicate with the aforementioned memory The one or more processors are equipped with Identifying candidate frame positions within the sequence of frames in the first video, Comparing the candidate frame positions among the frames in the sequence of frames, Selecting candidate video positions based on the comparison, wherein the selected candidate video positions include frame positions that are consistent among the candidate frame positions within the sequence of frames. Selecting a second video based on one or more parameters related to the candidate video position, and To enable simultaneous playback with the first video, the second video is inserted within the candidate video position of the first video. A system configured to perform the following actions.

18. The candidate frame position includes a two-dimensional position, The system according to claim 17, wherein the difference between the content displayed at the two-dimensional position in the first frame and the content displayed at the two-dimensional position in the second frame is less than a threshold.

19. When identifying candidate frame positions, one or more processors Regarding each frame, Calculating the local average value for every pixel, For each row of the frame, the length of the longest line in which the local average value remains above a first threshold is stored, and For each column of the frame, store the largest rectangle in which the local average value remains above the second threshold. The system according to claim 17, further configured to perform the following:

20. The system according to claim 17, wherein the second video is selected from a plurality of candidate videos based on the duration of the candidate video position in the sequence of frames and the duration of the second video.

21. A non-temporary computer-readable storage medium storing instructions that can be executed by one or more processors to perform a method, wherein the method is The first step is to identify candidate frame positions within a sequence of frames in the video, The steps include comparing the candidate frame positions among the frames in the sequence of frames, A step of selecting candidate video positions based on the comparison, wherein the candidate video positions include frame positions that are consistent among the candidate frame positions in the sequence of frames; The steps include selecting a second video based on one or more parameters related to the candidate video position, The steps include inserting the second video into the candidate video position of the first video for simultaneous playback with the first video, and Non-temporary computer-readable storage media, including [specific type of storage medium].

Images