SYSTEM AND METHOD FOR OVERLAYING VIDEO ON VIDEO - Patent application

Abstract

The second video may be overlaid or inserted into the first video in a non-intrusive manner. The location of the second video within the first video may be determined based on analyzing non-essential locations within consecutive frames of the first video. The second video may be selected from multiple candidate videos based on parameters of the second video such as size, duration, context, etc. Thus, by nesting the second video within the first video, the second video may be displayed simultaneously with the first video while providing a streamlined viewing experience and minimizing distractions.

Description

[Background technology]

[0001] Video ads are often placed before the start of a video or after the video has finished. Sometimes, ads are placed intermittently throughout a video by pausing the video to present the ad and then resuming the video once the ad is completed. This interruption disrupts the viewer's viewing experience and gives the viewer the opportunity to either abandon watching the video altogether or ignore the ad until the video resumes. This creates user frustration and a disrupted viewing experience. Summary of the Invention [Means for solving the problem]

[0002] The present disclosure provides for overlaying a second video, such as a video advertisement, on a first video playing a viewer's desired content. The first and second videos may be played simultaneously. The placement of the second video within the first video may be determined by analyzing locations within the first video to select the least distracting location for overlaying the second video. Such analysis may include comparison of consecutive frames and identifying locations that have minimal changes and do not depict specific objects, such as humans, animals, etc. Such locations may be candidate locations for the second video, each of which may have an associated set of parameters, such as size, length of time, context of content within the associated frame of the first video, etc. The second video may be selected based on one or more of such parameters.

[0003] One aspect of the disclosure provides a method of placing a second video within a first video. The method may include identifying, by one or more processors, candidate frame positions within a sequence of frames in the first video; comparing, by one or more processors, the candidate frame positions among frames in the sequence of frames; selecting, by one or more processors, the candidate video positions based on the comparison, the candidate video positions including frame positions that are consistent among the candidate frame positions in the sequence of frames; selecting a second video based on one or more parameters associated with the candidate video positions; and inserting the second video within the candidate video positions of the first video for simultaneous playback with the first video. Identifying the candidate frame positions may include identifying two-dimensional positions where a difference between content displayed at the two-dimensional positions in the first frame and content displayed at the two-dimensional positions in the second frame is less than a threshold. The method may further include dividing each frame into a grid including a plurality of cells, the two-dimensional positions including a subset of the plurality of cells.

[0004] According to some examples, the step of identifying candidate frame locations may include, for each frame, calculating a local average value for every pixel, for each row of the frame, storing the length of the longest line for which the local average value remains above a first threshold, and for each column of the frame, storing the largest rectangle for which the local average value remains above a second threshold. The method may further include identifying stored rectangles having an area larger than a minimum area required by the second movie. The method may further include identifying one or more sequences of stored rectangles having consecutive positions in the first movie across at least N consecutive frames. Furthermore, the method may include calculating a weight for each sequence of stored rectangles, and selecting the candidate movie location is based on said weights.

[0005] According to some examples, the method may further include storing the candidate video positions and dynamically selecting a second video during playback of the first video based on at least one of a size or duration of the candidate video positions.

[0006] A second video may be selected from a number of candidate videos based on a duration of the candidate video position within the sequence of frames and a duration of the second video.

[0007] According to some examples, the method may further include identifying a context of the first video and selecting a second video from the plurality of candidate videos based on a context of the second video that corresponds to the identified context of the first video.

[0008] According to some examples, the method may further include annotating the first video with a first tag corresponding to the identified context, and the step of selecting the second video includes identifying a second tag of the second video that overlaps with the first tag. Such annotation of the first video with the first tag may include annotating the first video with tags that are relevant to the first video as a whole. In other examples, such annotation of the first video with the first tag may include annotating the first video with tags that are relevant to individual moments or frames within the first video.

[0009] According to some examples, the method may further include a step of comparing the candidate frame locations including running a machine learning model. Training the machine learning model may include collecting training samples, the training samples including a set of frames in a first video including annotated sequences of non-essential areas, and inputting the training samples into a deep neural network classifier, thereby training the classifier to evaluate, for each point on the set of frames, how likely each point is to be a non-essential location. The machine learning model may be trained to identify the presence of a human in the video and exclude locations in the frames that include a human from the candidate frame locations.

[0010] Another aspect of the disclosure provides a system including a memory and one or more processors in communication with the memory, which may be configured to identify candidate frame positions within a sequence of frames in a first video, compare the candidate frame positions among frames in the sequence of frames, select a candidate video position based on the comparison, where the candidate video positions include frame positions that are consistent among the candidate frame positions in the sequence of frames, select a second video based on one or more parameters associated with the candidate video positions, and insert the second video within the candidate video position of the first video for simultaneous playback with the first video.

[0011] The candidate frame locations may include two-dimensional locations where a difference between content displayed at the two-dimensional locations in the first frame and content displayed at the two-dimensional locations in the second frame is less than a threshold. In identifying the candidate frame locations, the one or more processors may be further configured to calculate, for each frame, a local average value for every pixel. For each row of the frame, the one or more processors may store the length of the longest line for which the local average value remains above a first threshold, and for each column of the frame, store the largest rectangle for which the local average value remains above a second threshold.

[0012] A second video may be selected from a number of candidate videos based on a duration of the candidate video position within the sequence of frames and a duration of the second video.

[0013] Another aspect of the disclosure provides a non-transitory computer-readable medium storing instructions executable by one or more processors to perform a method. Such a method may include identifying candidate frame positions within a sequence of frames in a first video, comparing the candidate frame positions among frames in the sequence of frames, selecting a candidate video position based on the comparison, the candidate video positions including frame positions that are consistent among the candidate frame positions in the sequence of frames, selecting a second video based on one or more parameters associated with the candidate video positions, and inserting the second video within the candidate video positions of the first video for simultaneous playback with the first video. [Brief description of the drawings]

[0014] [Figure 1A] A diagram showing examples of candidate frame positions within various frames of a first moving image according to an aspect of the present disclosure. [Figure 1B] A diagram showing examples of candidate frame positions within various frames of a first moving image according to an aspect of the present disclosure. [Figure 1C]A diagram illustrating an example of identifying candidate video positions for potential insertion of a second video into a first video based on comparing candidate frame positions of various frames according to an aspect of the present disclosure. [Figure 1D] 1A-1C illustrate examples of a second video inserted within a first video for simultaneous playback according to aspects of the present disclosure. [Figure 2A] 1A-1C are diagrams illustrating examples of candidate frame locations identified using a grid of cells, according to aspects of the present disclosure. [Figure 2B] 1A-1C are diagrams illustrating examples of candidate frame locations identified using a grid of cells, according to aspects of the present disclosure. [Figure 2C] 1A-1C are diagrams illustrating examples of candidate frame locations identified using a grid of cells, according to aspects of the present disclosure. [Diagram 3] FIG. 1 is a block diagram illustrating training of a machine learning model according to an aspect of the present disclosure. [Figure 4] FIG. 1 is a block diagram illustrating the execution of a machine learning model for identifying candidate video locations, according to aspects of the present disclosure. [Diagram 5] 11A-11C illustrate examples of multiplying assigned weights to candidate frame positions identified based on different approaches, according to aspects of the present disclosure. [Figure 6] 1A-1C illustrate examples of identifying candidate video positions based on candidate frame positions according to aspects of the present disclosure. [Figure 7] 13A-13C illustrate examples of selecting a second video based on parameters of a candidate video location according to aspects of the present disclosure. [Figure 8] FIG. 13 illustrates another example of selecting a second video based on parameters of a candidate video location according to aspects of the present disclosure. [Figure 9] FIG. 13 illustrates another example of selecting a second video based on parameters of a candidate video location according to aspects of the present disclosure. [Figure 10] FIG. 1 is a block diagram illustrating an example system according to an aspect of the present disclosure. [Figure 11] 1 is a flow diagram illustrating an exemplary method according to an aspect of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The present disclosure generally relates to overlaying one video onto another in an unobtrusive manner. The present disclosure provides a system and method for identifying candidate regions within a first video. The present disclosure further provides a system and method for selection of a second video for overlaying the first video based on the identified candidate regions.

[0016] In determining when and where to overlay the second video on the first video, a frame range within the first video may be selected for placement of the second video, and a two-dimensional location on the frame within the frame range may be selected. The selection of the frame range and the frame location may be performed using a low user disturbance technique. Additionally or alternatively, the selection of the frame range and the frame location may be performed using a contextual moment selection technique.

[0017] In a low user disturbance approach, video locations are identified as non-essential to the viewer. For example, a non-essential location may be a location that provides minimal or no disturbance to the content of the first video that the viewer is likely to want to see. For example, if the first video depicts a person playing soccer, a non-essential location may be a location that does not block the person, the ball, or the goal, but may instead obscure the depiction of the sky, grass, etc. According to some examples, non-essential locations may be scored based on how disturbing or distracting they are, with better scores correlating with less disturbance or distraction.

[0018] Calculating candidate frame locations, such as non-critical areas within individual frames, may be performed using a variety of techniques, such as static content detection techniques, machine learning techniques, hybrid techniques, etc.

[0019] According to static content detection techniques, given a sequence of video frames, the differences between successive video frames are calculated. Each resulting frame may be divided into a grid, and for each cell in the grid, the aggregate differences between successive frames are calculated.

[0020] According to machine learning techniques, a set of video frames is selected to serve as training samples. The frames may be selected randomly or based on various features, etc. Each of the selected video frames may be manually scrutinized to identify candidate frame locations or non-essential locations. Such candidate frame locations may be annotated. In other examples, training sample videos may be manually scrutinized within or throughout a segment to identify a sequence of non-essential areas or candidate video locations, which may be annotated. The annotated frames and / or videos may be used to train a deep neural network classifier to assess, for each point on a frame, how likely it is that the point is a non-essential location.

[0021] According to a hybrid approach, a machine learning model may be used to provide additional information. For example, a machine learning model may be trained to identify essential elements of a first video, and all locations containing such essential elements may be excluded from consideration as candidate locations for the second video. For example, one possible heuristic is that people are essential elements of the first video. A classifier may calculate the probability that each location in the frame contains a human.

[0022] As a result of any of the above techniques, locations within a frame or video segment may be scored or weighted. For example, a weight may be assigned to each cell of a grid, each pixel, or an area of ​​a frame defined in any other way. The weight may correspond to the likelihood of the area being non-essential, such as not intrusive or distracting, with respect to the placement of the second video. According to some examples, weighted results from different methods may be combined.

[0023] Candidate video locations for the second video may be identified within the first video using the candidate frame locations. For example, the calculated weights for each pixel, cell, or other area may be used to calculate a local average value for every pixel, cell, or other area of ​​the frame. This may blur the image and make the determination of the candidate locations less susceptible to noise. For each row of the frame, the length of the longest line whose local average value does not fall below a first threshold may be stored. For each column of the frame, the largest rectangle whose local average value does not fall below a second threshold may be stored. For each frame, the N largest rectangles based on such calculations may be stored.

[0024] According to some examples, the candidate video locations may be selected based on the requirements of the second video. For example, such requirements may include size, length, etc. Candidate frame locations may be selected that have consecutive locations in the first video for at least a minimum number of frames corresponding to the duration of the second video that are larger than the minimum area required by the second video, and so on. For each sequence of candidate frame locations, a weight may be calculated as the average weight of all areas encompassed by the candidate frame location in all frames of the sequence. From those sequences, the sequence that best fits the criteria of the second video while having the largest weight for each size / duration pair may be selected as the video location for the second video. The selected video locations may be stored for retrieval during playback, such that the second video may be dynamically selected, retrieved, and inserted into the first video.

[0025] Contextual moment selection may include identifying a context of a particular frame into which a second video is inserted and correlating such context with the context of the second video. For example, a frame of a first video may be annotated with relevant contextual tags. Such contextual tags may indicate that content within the frame is relevant to a particular theme, subject, object, etc. For example, the contextual tags may indicate that the frame is relevant to a particular sport, activity, product, state of mind, etc. The second video may also be assigned relevant tags. Corresponding tags between frames of the first video and the second video may be identified, and the second video may be selected for insertion into the first video based on such corresponding tags. Such corresponding tags may be overlapping, such as when some subset of tags match between frames of the first video and the second video. In another example, such corresponding tags may be complementary, such as when contextual tags of the first video include food and tags of the second video include dining room decor.

[0026] FIG. 1A illustrates an exemplary video frame 110 of a first moving image. The video frame 110 may be one of multiple consecutive frames that together form the first moving image. Each frame 110 may include content 120, such as an object, person, or other feature depicted in the moving image. A moving image progress bar 130 may indicate the length or duration of the first moving image and the point in the duration that corresponds to the video frame being viewed at a given moment. For example, a progress indicator 132 may move from one end of the progress bar 130, such as the left end, to the other end. In some examples, as the progress indicator 132 advances, it may be followed by a completion indicator 134 to indicate what portion of the moving image has already been viewed. For example, the completion indicator 134 may cause the progress bar 130 to be a different color, shape, thickness, etc., for portions of the progress bar 130 that correspond to frames prior to the frame 110 that corresponds to the progress indicator 132.

[0027] A first candidate frame location 150 of the video frame 110 may be identified. The candidate frame location 150 may correspond to a non-essential, non-intrusive, or non-distracting area within the frame. For example, such an area may be an area that does not significantly overlap with a major feature of the content 120. The candidate frame location 150 may encompass areas having any of a variety of shapes and sizes. Although one candidate frame location 150 is shown in this example, in other examples, a frame may have multiple candidate frame locations.

[0028] 1B shows a second video frame 111 of a first video, the second video frame 111 being later in the sequence of frames making up the first video, as indicated by the movement of the indicator 132 on the video progress bar 130. In this second video frame 111, the content 120 has changed such that it has moved to a different area in the second frame 111 compared to the first frame 110. Thus, the candidate frame location 151 may have a different size, shape, and / or location compared to the candidate frame location 150 in the first frame 110, as the non-essential areas in the second frame are different based on the change in the content 120 being displayed. Further in this example, an additional candidate frame location 152 is available in another portion of the second frame 111, to the left of the content 120.

[0029] 1C shows an example of comparing candidate frame positions 150, 151, 152 between multiple frames. Although this example shows only a few candidate frame positions from two frames, the comparison may include any number of candidate frame positions from multiple consecutive frames. In comparing the candidate regions, it is determined which regions are consistent between multiple consecutive frames. In the example shown, region 155 may be consistent across multiple consecutive frames. Thus, region 155 may be identified as a candidate video position for the second video.

[0030] Identifying candidate frame locations in each frame may be performed using any of a variety of techniques, some examples of which are described below in conjunction with Figures 2-5.

[0031] FIG. 1D illustrates an example of inserting a second video into a candidate video location identified in a first video. As shown, the second video 190 includes a second content that may be different from the content 120 of the first video. The second video 190 may have a duration equal to or less than the duration of the first video. For example, the duration of the second video may be during a subset of the frames of the first video. In some examples, the second video 190 may pop up within the first video and may disappear automatically when the second video 190 ends. In other examples, the second video 190 may include an icon 192 or other interactive element that may be used to close the second video 190 so that only the first video is shown. In further examples, the second video 190 may include other interactive elements, such as controls for adjusting the speed of the second video 190, controls for pausing the second video 190, controls for annotating the second video 190, game elements, and the like.

[0032] 2A-2C show an example of a static content detection technique, where, given a sequence of video frames, the differences between successive video frames are calculated. Each resulting frame may be divided into a grid, and for each cell in the grid, the aggregate differences between successive frames are calculated.

[0033] As shown in FIG. 2A, a first frame 110 is divided into a grid of cells 202. Similarly, as shown in FIG. 2B-2C, frames 111-112 are divided into a grid of cells. The cells may have a consistent size for each frame of the first video, but the size may vary in an analysis of one video from an analysis of another video. In some examples, each cell may be one pixel. In other examples, each cell may include multiple pixels. The cells may be compared between frames 110-112 to calculate the difference between the cells. Such differences may include changes in the content depicted in each cell, such as, for example, changes in color, light, objects, etc. Cells with minimal frame-to-frame differences may be considered to correspond to non-essential areas of the frames. In some examples, cells across frames with minimal differences may be assigned a value, where a value below a threshold indicates the likelihood of the cell being included in a candidate frame location. A collection of adjacent cells all having values ​​below a threshold may form a candidate frame location.

[0034] In some examples, the calculated weights for each pixel, cell, or other area may be used to calculate a local average value for every pixel, cell, or other area of ​​the frame. This may blur the image and make the determination of candidate locations less susceptible to noise. For each row of the frame, the length of the longest line whose local average value does not fall below a first threshold may be stored. For each column of the frame, the largest rectangle whose local average value does not fall below a second threshold may be stored. For each frame, the N largest rectangles based on such calculations may be stored.

[0035] In some examples, a machine learning model may be used to identify candidate frame locations within each frame and / or to identify candidate video locations, including candidate frame locations across multiple consecutive frames. Figure 3 shows an example of training a machine learning model 330.

[0036] The machine learning model 330 may be, for example, a neural network, a convolutional neural network, a generalized additive model, or any of a variety of other types of machine learning models. To generate the model, the machine learning engine may obtain feature data and signal data associated with several content item impressions and train the model using the feature data and corresponding signal data of each of the historical content item impressions. For example, the machine learning engine may be configured with regression analysis techniques that use statistical processes to estimate the relationship between a feature or a combination of features and a signal. The feature or combination of features may be the predictor or independent variable, and the signal may be the dependent or reference variable that may change when the feature is altered. In some cases, the machine learning engine may estimate or determine the conditional expectation of the dependent variable given the independent variable, such as the mean value of the dependent variable when the independent variable is fixed, or other parameters or indicators of the conditional distribution of the dependent variable or signal given the independent variable or feature. The predicted signal is a function of the independent variable and may be called a regression function. Additionally, the machine learning engine may identify, determine, or characterize the variation of the dependent variable around a regression function that may be described by a probability distribution. The machine learning engine may use the probability distribution to generate a confidence score for the predicted signal value or use the probability distribution as the confidence score.

[0037] In the illustrated example, the machine learning model 330 may be trained using multiple training samples 310-312. Although only a few training samples are shown, it should be understood that any number of training samples may be used. In this example, each training sample 310-312 is tagged with information identifying candidate frame locations 350-352. Such tagging may be performed by a manual process. Tagging may include annotating each frame, such as by drawing or highlighting areas of the frame. In other examples, tagging may include updating metadata associated with the image to identify locations within the image that are candidate frame locations. For example, the locations may be identified using coordinates, pixel identifiers, or any other information.

[0038] The training samples 310-312 may be input to the machine learning model 330. Based on such input, the machine learning model 330 may learn to recognize candidate frame locations. For example, the machine learning model 330 may recognize patterns among the annotated candidate frame locations 350-352 of the training samples 310-312 and learn to detect such patterns in other frames to identify the candidate frame locations. The machine learning model 330 may provide an output that identifies the candidate frame locations. According to some examples, the output may be evaluated as part of a feedback loop used to update or fine-tune the machine learning model 330.

[0039] The model 100 can repeat the training steps multiple times until one or more stopping criteria are met. The stopping criteria may include, for example, a maximum number of iterations of the training steps and / or, for supervised or semi-supervised learning, backpropagation, gradient descent, and model parameter updates. The stopping criteria may additionally or alternatively define a minimum improvement between training steps. For semi-supervised training, an example may be a relative or absolute reduction in the calculated error between the output predicted by the model 100 and the corresponding ground-truth label on training data set aside for validation and / or testing. For unsupervised learning, an exemplary loss function may be least squares.

[0040] In some examples, the model 100 may be trained for a number of epochs, e.g., 1000, with early stopping if the error between the predicted output and the ground truth error does not improve over 10 epochs. Other stopping criteria can be based on a maximum amount of computing resources allocated for training, e.g., training time exceeded, e.g., the total amount of wall clock time, or the total number of processing cycles consumed, after which training is terminated.

[0041] As shown in FIG. 4, the trained machine learning model may be used to predict candidate video locations within a video input. For example, when a video is input, the machine learning model may identify candidate frame locations within multiple consecutive frames, such as by evaluating, for each point on a frame, how likely the point is to be a non-essential location. The machine learning model may further identify candidate video locations based on consistent candidate frame locations across multiple consecutive frames. If more than one candidate video location is detected, the model may identify each candidate video location. According to some examples, the multiple candidate video locations may be scored and / or ranked. According to other examples, parameters of the candidate video locations, such as duration, size, etc., may be provided with the output.

[0042] According to some examples, the machine learning model may be trained to detect other types of information. By way of example, the machine learning model may be trained to predict essential areas within a frame, such as areas depicting people, animals, featured objects, etc. Such essential areas may therefore be removed from consideration for candidate frame locations.

[0043] In some examples, a hybrid of different approaches may be used. For example, FIG. 5 illustrates an example of a hybrid approach. For example, frame 510 may be evaluated using a combination of two approaches. Frame 512 represents frame 510 evaluated using the static content detection approach described above in connection with FIGS. 2A-2C. Frame 514 represents frame 510 evaluated using a machine learning model as described above in connection with FIGS. 3-4. Evaluating the pixels, cells, or areas of frames 512, 514 under each approach may include assigning a weight to each pixel, cell, or area, the weight indicating the likelihood that the pixel / cell / area is a non-essential area. The weights assigned to a given portion of each frame 512, 514 may differ based on the evaluation technique. For example, a particular pixel of frame 512 may be assigned a weight value of x based on the static content detection evaluation, while the same particular pixel of frame 514 may be assigned a weight value of y based on the machine learning evaluation. In this example, the weights are multiplied so that both evaluation techniques are taken into account. Using the multiplied weights for each pixel / cell / area, candidate frame locations 550 may be identified for frame 510.

[0044] Candidate video locations for the second video may be identified in the first video using the candidate frame locations. FIG. 6 illustrates an example of identifying candidate video locations based on the candidate frame locations. A first candidate frame location 150 of a video frame 110 may be identified. For example, the candidate frame location 150 may be determined by comparing a pixel or other sub-portion of the frame with a corresponding pixel or sub-portion of a larger or larger number of adjacent frames. A weight may be assigned to each pixel or other sub-portion based on the degree of difference between the frames. In some examples, the calculated weight for each pixel, cell, or other area may be used to calculate a local average value for every pixel, cell, or other area of ​​the frame. For each row of the frame, the length of the longest line whose local average value does not fall below a first threshold value may be stored. For each column of the frame, the largest rectangle whose local average value does not fall below a second threshold value may be stored. For each frame, the N largest rectangles based on such calculations may be stored. A second video frame 111 of the first video is analyzed in a similar manner, the second video frame 111 being later in the sequence of frames that make up the first video. Additionally, each frame between the first and second frames may be analyzed in a similar manner to determine the N largest rectangles in each frame. The N largest rectangles in each frame may be compared to determine which locations in the frame are consistent among rectangles of at least a subportion of the first video. According to some examples, slight discrepancies may be tolerated, such as when one or more frames in the sequence have a small deviation with respect to the N largest rectangles. In this example, the candidate video positions 155 represent areas that were consistently within the N largest rectangles in each of the frames in the sequence corresponding to the span of time 135.

[0045] A second video to be inserted or overlaid on a first video may be selected from multiple videos in some examples. As an example, the second video may be an advertisement inserted into the first video. The first video may be, for example, streamed content. Various advertisements may be available. In some examples, different advertisements may be selected for each playback of the streamed content by the same user or different users.

[0046] FIG. 7 illustrates an example where a second video is selected based on size. For example, a candidate video location 760 may have a defined size and shape. Candidate second videos 781, 782, 783 may each have a different size. A portion of a candidate second video, such as candidate second video 783, may be larger in size than the candidate video location 760. Such a larger second video may be excluded from selection because it encompasses a portion of the first video beyond the defined candidate location 760. Since each of the candidate second videos 781, 782 fits, either one may be selected. According to some examples, the larger second video 782 may be selected to utilize a large portion of the candidate video location 760.

[0047] 8 illustrates an example of selecting a second video based on the length of time a candidate video position is available within a first video. As shown, a candidate video position within a first video has a duration 835 of 15 seconds. Thus, one of the candidate second videos 881-883 having a duration of 15 seconds or less may be selected. Other second videos having longer durations may be eliminated.

[0048] FIG. 9 illustrates an example of selecting a second video based on the content of a first video. A context may be determined that is relevant to a particular frame of the first video into which the second video is inserted, or a context may be determined that is relevant to other parts or the entirety of the first video. According to some examples, frames of the first video may be annotated with relevant contextual tags. Tagging may be performed manually, using image or object recognition techniques, using machine learning, etc. Such contextual tags may indicate that the content in the frame is relevant to a particular theme, subject, subject, etc. For example, the contextual tags may indicate that the frame is relevant to a particular sport, act, product, state of mind, etc. The candidate second videos 981-983 may also be assigned relevant tags. One of the candidate second videos 981-983 may be selected based on the number of overlapping tags. For example, the second video with the largest number of tags that match or correspond to the tags of the first video may be selected. In the illustrated example, frame 910 including the candidate video location may be soccer related and thus tagged with indicators such as soccer, sports, field, outdoors, athletics, determination, motivational, practice, etc. Candidate second video 981 may be cooking related and thus tagged with food, cooking, utensils, pasta, butter, etc. Candidate second video 982 may be sneaker related and thus tagged with sports, athletics, comfort, style, blue, etc. Candidate second video 983 may be furniture related and may be tagged with wood, mahogany, traditional, luxury, etc. Candidate second video 982 may be selected for inclusion in the candidate video location because it included at least two tags--sports and athletics--that overlapped with tags of the first video.

[0049] Although size, duration, and context are shown separately in the above examples, the second video may be selected based on a combination of these and / or other parameters. The parameters may be used to dynamically select a second video for insertion into a first video.

[0050] FIG. 10 illustrates an exemplary system for implementing candidate video location identifiers and second video selection. In particular, the system includes one or more client devices 1001, 1002, 1003 that communicate with one or more servers 1020 through a network 1050. For example, each of several different client devices 1001-1003 may receive first video content from the server 1020. The server may determine where in the first video the second video may be inserted for minimal interruption during simultaneous playback, and may select the second video to be inserted. Although several client devices 1001-1003 are shown, it should be understood that any number of client devices may communicate with one or more servers 1020 through the network 1050.

[0051] The server 1020 includes one or more processors 1070. The processor 1070 can be any conventional processor, such as a commercially available CPU. Alternatively, the processor can be a dedicated component, such as an application specific integrated circuit ("ASIC") or other hardware-based processor. Although not required, the server 1020 may include specialized hardware components for executing specific computing processes.

[0052] The memory 1060 can store information accessible by the processor 1070, including instructions that can be executed by the processor 1070 and that can be retrieved, manipulated, or stored by the processor 1070.

[0053] The instructions can be a set of instructions that are executed by the processor 1070 directly, such as in machine code, or indirectly, such as in a script. In this regard, the terms "instructions," "steps," and "programs" may be used interchangeably herein. The instructions may be stored in object code format for direct processing by the processor 1070, or in other types of computer languages, including scripts or a collection of independent source code modules that are interpreted on demand or precompiled. The functions, methods, and routines of the instructions are described in more detail in the examples above and in the illustrative manner below.

[0054] The data may be retrieved, stored, or modified by the processor 1070 in accordance with the instructions. The data may also be formatted in a computer-readable format, such as, but not limited to, binary values, ASCII, or Unicode. Additionally, the data may include information sufficient to identify relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memory including other network locations, or information used by a function to calculate the relevant data.

[0055] Although FIG. 10 functionally illustrates the processor, memory, and other elements of the server 1020 as being in the same block, the processor, computer, computing device, or memory may actually include multiple processors, computers, computing devices, or memories that may or may not be housed in the same physical housing. For example, the memory can be a hard drive or other storage medium located in a housing different from that of the server 1020. Thus, reference to a processor, computer, computing device, or memory is understood to include reference to a collection of processors, computers, computing devices, or memories that may or may not operate in parallel. For example, the server 1020 may include server computing devices that operate as a load-balanced server farm, a distributed system, and the like. Additionally, although some functions described below are shown as occurring on a single computing device having a single processor, various aspects of the subject matter described herein may be implemented by multiple computing devices that communicate information over a network, for example.

[0056] The memory 1060 can store information accessible by the processor 1070, including instructions 1062 that can be executed by the processor 1070. The memory can also include data 1064 that can be retrieved, manipulated, or stored by the processor 1070. The memory 1060 can be any type of non-primary computer-readable medium that can store information accessible by the processor 1070, such as a hard drive, solid-state drive, tape drive, optical storage, memory card, ROM, RAM, DVD, CD-ROM, write-capable, and read-only memory. The processor 1070 can be a well-known processor or other lesser known types of processor. Alternatively, the processor 1070 can be a dedicated controller, such as an ASIC.

[0057] The instructions 1062 can be a set of instructions executed by the processor 1070 directly, such as in machine language, or indirectly, such as in a script. In this regard, the terms "instructions," "steps," and "program" may be used interchangeably herein. The instructions 1062 may be stored in object code format for direct processing by the processor 1070, or in other types of computer languages, including scripts or collections of independent source code modules that are interpreted on demand or precompiled. The instructions 1062 may be executed to identify an area in a sequence of video frames for inserting a second moving image, as described above. Additionally, the instructions 1062 may be executed to select a second moving image for insertion or overlay into a first moving image during simultaneous playback of the first and second moving images.

[0058] The data 1064 may be retrieved, stored, or modified by the processor 1070 in accordance with the instructions 1062. For example, the data 1064 may be stored in a computer register, in a relational database as a table with multiple different fields and records, or in an XML document, although the system and method are not limited by a particular data structure. The data 1064 may also be formatted in a computer readable format, such as, but not limited to, binary values, ASCII, or Unicode. Furthermore, the data 1064 may include sufficient information to identify related information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memory including other network locations, or information used by a function to calculate the related data.

[0059] The server 1020 may be further coupled to an external storage 1080, such as a database. The external storage 1080 may store content for delivery to the client devices 1001-1003. The external storage 1080 may further store banners for rendering at the client devices 1001-1003 along with the content. Such banners may include advertisements or other information. Although the external storage 1080 is shown as a single database, it should be understood that the physical structure of the external storage 1080 may include multiple storage devices, and such multiple devices may communicate with each other like a distributed storage system.

[0060] Each of the client devices 1001, 1002, 1003 may be configured similarly to each other and to the server 1020 in that it includes a processor 1091 and a memory 1092 including data 1093 and instructions 1094 executable by the processor 1091. The structure of the processor 1091 and the memory 1092 may be similar to the structure of the processor 1070 and the memory 1060, respectively, described above. The client devices 1001-1003 may be any type of personal computing device, such as a laptop, a desktop computer, a tablet, a game console, a phone, an augmented reality or virtual reality headset, a smart watch, a smart glass, a home assistant hub, or any other computing device including a display for outputting content together with one or more banners. Each of the client devices 1001-1003 may further include one or more user input devices 1095. Such user input devices 1095 may include a touch screen, a touch pad, a key pad, a camera, a microphone, a joystick, or any other device adapted to capture input signals from a user.

[0061] In addition to the exemplary systems described above, exemplary methods are described below. Such methods may be performed using any of the systems described above, modifications thereof, or various systems having different configurations. It should be understood that the operations included in the methods below do not necessarily have to be performed in the exact order described. Rather, various operations may be processed in different orders or simultaneously, and operations may be added or omitted.

[0062] 11 illustrates an exemplary method for determining placement of a second video within a first video. The method may be performed, for example, by a computing device or module, or a system of one or more distributed server processes. Although operations are described in a particular order, it should be understood that the order may be modified. Additionally, operations may be performed simultaneously, and operations may be added or omitted.

[0063] At block 1110, candidate frame locations are identified within each frame of a plurality of consecutive frames in a first moving image.

[0064] At block 1120, the candidate frame locations are compared between consecutive frames. The comparison determines where within the sequence of frames there are consistent frame locations for the second video. The identified candidate frame locations may be given a weight corresponding to a confidence level, including areas where the location is not critical.

[0065] At block 1130, a candidate video location is selected based on the candidate frame location, such as by identifying a portion of a frame that consistently includes the candidate frame location across multiple adjacent frames.

[0066] At block 1140, a second video is selected based on multiple parameters including, but not limited to, size, length, context, etc. Once the candidate video locations and candidate frame locations are determined, it reveals certain parameters that must be met by the second video. There are additional parameters that must also be met and may be indicated, for example, by the ad provider or video provider.

[0067] The selected video is inserted at the selected video location and within the selected frame range at block 1150. A second video may then be played simultaneously with the first video.

[0068] Although some of the examples above refer to overlaying a video with a video advertisement for simultaneous playback, the techniques described above may be similarly applied for other types of videos to be overlaid on a first video for simultaneous playback.

[0069] Unless otherwise noted, the foregoing alternative examples are not mutually exclusive and may be implemented in various combinations to realize unique advantages. These and other variations and combinations of the features discussed above may be utilized without departing from the subject matter defined by the claims, and the foregoing description should be taken as illustrative rather than limiting the subject matter defined by the claims. In addition, the provision of examples described herein, and clauses that refer to "such as" and "including" should not be construed as limiting the subject matter of the claims to any particular example, but rather, the example is intended to illustrate only one of many possible examples. Additionally, the same reference numbers in different drawings may identify the same or similar elements. [Explanation of symbols]

[0070] 110 Video Frames 111 Second video frame 112 frames 120 Contents 130 Video Progress Bar 134 Completion Indicator 150 First candidate frame position 151 candidate frame positions 152 additional candidate frame positions 155 regions, candidate video positions 190 2nd video 192 Icons 202 Cells 310~312 Training samples 330 Machine Learning Models 350~352 Candidate frame positions 510 Frame 512 frames 514 Frames 550 candidate frame positions 760 candidate video positions 781~783 Candidate's second video 835 Duration 881~883 Candidate's second video 910 Frame 981~983 Candidate's second video 1001~1003 Client devices 1020 Server 1050 Network 1060 Memory 1062 instructions 1064 Data 1070 processor 1080 External Storage 1091 processor 1092 Memory 1093 Data 1094 Instructions 1095 User Input Devices

Claims

1. 1. A method for placing a second moving image within a first moving image, comprising: identifying, by one or more processors, candidate frame locations within a sequence of frames in the first video; comparing, by the one or more processors, the candidate frame positions among frames in the sequence of frames; selecting, by the one or more processors, candidate video positions based on the comparison, the candidate video positions including frame positions that are consistent among the candidate frame positions in the sequence of frames; selecting a second video based on one or more parameters associated with the candidate video location; inserting the second video within the candidate video location of the first video for simultaneous playback with the first video; A method comprising:

2. wherein the step of identifying candidate frame locations includes identifying two-dimensional locations; The method of claim 1 , wherein a difference between a content displayed at the two-dimensional position in a first frame and a content displayed at the two-dimensional position in a second frame is less than a threshold.

3. further comprising dividing each frame into a grid including a plurality of cells; The method of claim 2 , wherein the two-dimensional location comprises a subset of the plurality of cells.

4. The step of identifying candidate frame locations comprises: For each frame, Calculating a local average value for every pixel; storing, for each row of the frame, the length of the longest line along which the local average value remains above a first threshold; storing, for each column of the frame, the largest rectangle in which the local mean value remains above a second threshold; 2. The method of claim 1, comprising:

5. The method of claim 4 , further comprising identifying a stored rectangle having an area greater than a minimum area required by the second moving image.

6. The method of claim 4 , further comprising identifying one or more sequences of stored rectangles having consecutive positions in the first moving image across at least N consecutive frames.

7. calculating a weight for each sequence of stored rectangles; The method of claim 6 , wherein selecting the candidate video positions is based on the weights.

8. storing the candidate video positions; dynamically selecting the second video during playback of the first video based on at least one of a size or duration of the candidate video locations; The method of claim 1, further comprising:

9. The method of claim 1 , wherein the second video is selected from a plurality of candidate videos based on a duration of the candidate video position within the sequence of frames and a duration of the second video.

10. 2. The method of claim 1, further comprising: identifying a context of the first video; and selecting the second video from a plurality of candidate videos based on a context of the second video that corresponds to the identified context of the first video.

11. annotating the first video with a first tag corresponding to the identified context; The method of claim 10 , wherein selecting the second video includes identifying a second tag of the second video that overlaps with the first tag.

12. 12. The method of claim 11, wherein annotating the first video with a first tag comprises annotating the first video with a tag that is relevant to the entirety of the first video.

13. 12. The method of claim 11, wherein annotating the first video with first tags comprises annotating the first video with tags that are associated with individual moments or frames within the first video.

14. The method of claim 1 , wherein the steps of identifying and comparing candidate frame locations include executing a machine learning model.

15. Training the machine learning model, collecting training samples, the training samples including a set of frames in the first video that include annotated sequences of non-essential areas; inputting the training samples into a deep neural network classifier, thereby training the deep neural network classifier to evaluate, for each point on the set of frames, how likely each point is to be a non-critical location; The method of claim 14 , further comprising the step of:

16. The method of claim 14 , wherein the machine learning model is trained to identify the presence of humans in the video and to exclude locations in frames that contain the humans from the candidate frame locations.

17. Memory, one or more processors in communication with the memory; wherein the one or more processors: Identifying candidate frame locations within a sequence of frames in a first moving image; comparing the candidate frame positions among frames in the sequence of frames; selecting candidate video positions based on the comparison, the candidate video positions including frame positions that are consistent among the candidate frame positions within the sequence of frames; selecting a second video based on one or more parameters associated with the candidate video location; and inserting the second video within the candidate video location of the first video for simultaneous playback with the first video. A system configured to:

18. the candidate frame positions include two-dimensional positions; 18. The system of claim 17, wherein a difference between content displayed at the two-dimensional location in a first frame and content displayed at the two-dimensional location in a second frame is less than a threshold.

19. In identifying candidate frame locations, the one or more processors: For each frame, Calculating a local average value for every pixel; storing, for each row of the frame, the length of the longest line along which the local average value remains above a first threshold; and storing, for each column of said frame, the largest rectangle in which said local mean value remains above a second threshold; The system of claim 17, further configured to:

20. 20. The system of claim 17, wherein the second video is selected from a plurality of candidate videos based on a duration of the candidate video position within the sequence of frames and a duration of the second video.

21. A non-transitory computer-readable storage medium having stored thereon instructions executable by one or more processors to perform a method, the method comprising: identifying candidate frame locations within a sequence of frames in a first moving image; comparing the candidate frame positions among frames in the sequence of frames; selecting candidate video positions based on the comparison, the candidate video positions including frame positions that are consistent among the candidate frame positions in the sequence of frames; selecting a second video based on one or more parameters associated with the candidate video location; inserting the second video within the candidate video location of the first video for simultaneous playback with the first video; A non-transitory computer readable storage medium comprising:

Images