Image space detection suitable for overlaying media content

Abstract

Disclosed herein are a system, method, and computer-readable storage medium for detecting spaces suitable for overlaying media content on an image using (i) a neural network-based method and (ii) a heuristic-based method. The system receives a candidate image, which can be an image or a video frame. In the neural network-based method, the candidate image is then input to a neural network. The neural network can output coordinates and one or more metrics representing one or more bounding boxes for inserting media content into the candidate image. In the heuristic-based method, the candidate image is processed using image processing techniques to automatically suggest spaces, which are further analyzed using a heuristic rule-based method to select an insertion space defined by a bounding box for inserting media content. One or more media content items can then be selected for insertion into the image in the selected bounding box. The system can then cause a display of the image and the selected media content item overlaid on the image within the selected bounding box.

Description

TECHNICAL FIELD

[0001] The disclosure relates generally to the field of detecting spaces within an image or video frame in which content can be inserted, and more particularly to neural network based space detection and heuristic based space detection suitable for overlaying media content. BACKGROUND

[0002] Many images and video frames include areas in which text and other content can be inserted without obscuring one or more important portions of the image or video frame. These areas are sometimes referred to as copy space. For example, an image or video frame can focus on a boat and a person on the boat. However, the image or video frame can include areas that show as sky and / or ocean. These areas can be used to display text or other media content (e.g., images). In one embodiment, these areas can be used to display additional content items to a viewer. In another embodiment, copy space can be used to insert links and other information into the image or video frame.

[0003] There are various systems available today that enable a responsible person to mark copy space on an image. When tens of thousands of video frames of images must be marked, the process is often time consuming and inefficient. Thus, it is desirable to automatically and accurately identify copy space on images and video frames. BRIEF DESCRIPTION OF DRAWINGS

[0004] The disclosed embodiments have other advantages and features that will be apparent from the following detailed description, the appended claims and the accompanying figures (or drawings). A brief introduction of the figures is given below.

[0005] Figure 1 A neural network based system for detecting spaces suitable for overlaying media content on an image is shown in accordance with some embodiments.

[0006] Figure 2 An image with exemplary bounding boxes is shown.

[0007] Figure 3 An embodiment of a data structure with corresponding fields representing the output of a neural network is shown.

[0008] Figure 4 Various media content items overlaid on a candidate image are shown.

[0009] Figure 5 is a flowchart showing the overall process for selecting spaces suitable for overlaying media content on an image in accordance with some embodiments.

[0010] Figure 6 A candidate image with exemplary identified spaces for overlaying media content items is shown in accordance with some embodiments.

[0011] Figures 7A to 7D The results of various image processing techniques on candidate images according to some implementation schemes are shown.

[0012] Figures 8A to 8E The results of various image processing techniques on candidate images according to some implementation schemes are shown.

[0013] Figures 9A to 9C An image showing an exemplary detection space for covering media content items is illustrated according to some implementation schemes.

[0014] Figure 10 A block diagram of a heuristic media content insertion system according to some implementation schemes is shown.

[0015] Figure 11 This is a flowchart illustrating the overall process for detecting a space suitable for overlaying media content on an image, according to some implementation schemes.

[0016] Figure 12 This demonstrates the ability to read instructions from a machine-readable medium and execute those instructions in a processor (or controller) for use in... Figure 1 and Figure 10 The diagram shows a block diagram of the components of an example machine used by the modules of the system shown.

[0017] These figures depict various embodiments for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein can be employed without departing from the principles of the embodiments described herein.

[0018] The accompanying figures use similar reference numerals to identify similar elements. A letter following a reference numeral, such as "115a," indicates that the text specifically refers to the element with that particular reference numeral. Reference numerals without a following letter, such as "115," refer to any or all elements in the figure that bear that reference numeral. Detailed Implementation

[0019] Placing text on images is an important part of producing high quality visual designs. Applications of copy space detection include generating email banners, homepages, and calls to action, among others. These are mostly performed manually, thus making the task of graphic asset development time consuming, requiring designers to plan and manipulate media and vector graphics, build layer stacks, and finally place and format text while balancing style, brand consistency, and tone in the design. Automating this work by choosing appropriate positions and orientations and styles for text elements requires understanding the content of the image on which the text must be placed. Furthermore, establishing the parameters of a copy space that can render media content such as text on an image is different from foreground-background separation methods in image processing, as the resulting separation map can not take into account image parts that can potentially be covered by media content without destroying important visual elements in the image.

[0020] Disclosed herein are methods for detecting spaces (copy spaces) suitable for overlaying media content on images. One method proposed is a neural network-based system, method, and computer-readable storage medium for detecting spaces (copy spaces) suitable for overlaying media content on images. Another method proposed is a heuristic-based system, method, and computer-readable storage medium for detecting spaces (copy spaces) suitable for overlaying media content on images.

[0021] Neural network-based system for copy space detection

[0022] Configuration overview

[0023] Disclosed herein are neural network-based systems, methods, and computer-readable storage media for detecting spaces (copy spaces) suitable for overlaying media content on images. The system receives a candidate image, which can be an image or a video frame. Typically, the image or video frame can have some spaces for inserting media content without covering important parts of the image. The candidate image is then input to a neural network that has been trained with training data comprising a plurality of images and one or more corresponding bounding boxes for each of the plurality of images. The neural network can output coordinates and one or more measures of one or more bounding boxes representing spaces for inserting media content into the candidate image. The one or more bounding boxes can be sent with a request to display a media content item in the bounding box. The request can include the one or more measures of the one or more bounding boxes. The media content item can be received in response to the request, and the candidate image and the media content item can be displayed with the media content item overlaid on the candidate image within the bounding box.

[0024] Media content insertion process

[0025] Figure 1An exemplary system for detecting spaces suitable for overlaying media content on an image is shown. The system may include a media content storage system 100 and a media content insertion system 160. The media content insertion system 160 includes a communication module 170, a bounding box detection module 180, and an overlay generation module 190.

[0026] Communication module 170 may include hardware, software, or a combination of both for communication and may be used by media content insertion system 160 to communicate with other systems (e.g., media content storage systems) and devices. Communication module 170 may receive candidate images for overlaying media content. Communication module 170 may receive candidate images from another device or location within the media content insertion system. Candidate images may be any appropriately formatted image. In some embodiments, candidate images may be frames of video content items. Communication module 170 may pass candidate images to bounding box detection module 180.

[0027] The bounding box detection module 180 can receive and process candidate images from the communication module 170. The bounding box detection module 180 can input the candidate images into a neural network to receive one or more bounding boxes as output. The input operation occurs after the neural network has been trained using training data. In this example, the training data includes images and one or more corresponding bounding boxes for each image. In some embodiments, the neural network can be a convolutional neural network. This network can use image pixels that have been converted into vectors as input. In some embodiments, the neural network can be a multilayer perceptron including an input layer, hidden layers, and an output layer. This type of network can include other layers.

[0028] As described herein, a bounding box refers to the area on an image used to place media content. Although bounding boxes are shown as rectangles, they can be any shape, including but not limited to squares, circles, pentagons, and any other suitable shape. Bounding boxes can be irregular in size and / or include multiple shapes.

[0029] For example, the bounding box can be represented by a rectangle that includes one or more coordinates on the candidate image and one or more measures extending from those coordinates. In some implementations, the bounding box can be represented by coordinates and two measures, the coordinates being counted based on the number of horizontal and vertical pixels starting from the top-left corner of the candidate image. The coordinates can represent the top-left corner of the bounding box. The two measures can include a vertical offset and a horizontal offset extending from the coordinates. Based on the coordinates and offsets, a rectangular bounding box with an area based on the offset and a position based on the coordinates can be generated.

[0030] Figure 2Images with exemplary corresponding bounding boxes are shown. Image 200 shows bounding box 202. However, bounding boxes of different sizes / shapes can be added to image 200. Image 220 is shown as having two corresponding bounding boxes (bounding box 222 and bounding box 224). Although bounding boxes 222 and 224 are shown as rectangles, these bounding boxes can have different shapes and sizes.

[0031] In some embodiments, the media content insertion system 160 may receive from a neural network a first coordinate representing a first offset along a horizontal axis of a candidate image and a second coordinate representing a second offset along a vertical axis of the candidate image. These two coordinates may represent a specific pixel on the candidate image. This pixel may be used as a point associated with a bounding box. For example, as described above, this point may be the center of a circle, a corner of a rectangle, a triangle, a square, or another suitable point. The media content insertion system 160 may receive a first measure extending from the first coordinate along the horizontal axis. For example, the first measure may be the number of units (e.g., pixels) that the media content insertion system 160 can use to calculate the next point (e.g., in the horizontal direction) associated with the bounding box. In some embodiments, the first measure may correspond to the diameter of a circle, the radius of a circle, the side of a rectangle, the side of a square, or the side of a triangle.

[0032] In some embodiments, the media content insertion system 160 may receive a second measure extending from the second coordinate along a vertical axis. The second measure may be a number of units (e.g., pixels) that the media content insertion system 160 can use to calculate the next point (e.g., in the vertical direction) associated with the bounding box. In some embodiments, the second measure may correspond to the radius or diameter of an ellipse, or the second side of a rectangle or triangle. In some embodiments, the received bounding box may have only one measure, and the second measure may not be shown.

[0033] In some implementations, the media content insertion system 160 may receive the probability that a corresponding bounding box lies within a region of the candidate image suitable for inserting media content into the candidate image. For example, the neural network may return multiple possible bounding boxes, each with a probability that it is a valid bounding box for inserting media content into the candidate image. The bounding box detection module 180 may, for each bounding box, determine whether the corresponding probability meets a threshold probability. For example, the threshold probability may be a percentage value, a value between zero and one or another suitable value. For example, the threshold probability may be set to .5, .8, .9, or another suitable value. For example, the bounding box detection module 180 may retrieve the threshold probability from memory. In response to determining that the probability of a particular bounding box does not meet the threshold probability, the bounding box detection module 180 removes the particular bounding box from the request. That is, the bounding boxes received from the neural network can be filtered based on probability.

[0034] Figure 3 An example of a data structure with a corresponding field representing the output of a neural network that represents a bounding box is shown. Figure 3 Data structure 300 may include one or more fields. Field 302 may include one or more coordinates associated with the bounding box. For example, field 302 may include the coordinates of a corner of a rectangular bounding box, the center of a circular bounding box, or another suitable point associated with the bounding box. In some embodiments, field 302 may include a vector representing the points of the bounding box. Field 304 may include a first metric stored as the number of units (e.g., pixels) offset from the point represented by the coordinates in field 302. Field 306 may include a second metric stored as the number of units (e.g., pixels) offset from the point represented by the coordinates in field 302. Other metrics may also be stored in data structure 300. Field 308 stores the probability that the bounding box represented by the particular data structure is a valid bounding box. In some embodiments, each field of data structure 300 may be coordinates in a coordinate vector representing an irregular bounding box.

[0035] As described above, in some embodiments, the bounding box can have different shapes. For example, the bounding box can be a square (e.g., represented by coordinates and an offset), a circle (e.g., represented by coordinates as the center of the circle and a measure representing the radius or diameter of the circle), or an ellipse (e.g., represented by two radii and coordinates at the center). As described above, the bounding box can be any shape and / or can have irregular measures. In some embodiments, the bounding box can be represented by a vector representing the coordinates of points on the candidate image. Vectors and concatenations of these vectors can represent the bounding box.

[0036] In some implementations, a neural network can be trained using images with known bounding boxes (sometimes called training images). A training module (not shown) can be used to train the neural network, which may be part of the media content insertion system 160 or a different system. The training module can receive a large number of training images and corresponding vectors. Each vector may include a set of coordinates and a set of metrics representing a specific bounding box. In some implementations, some training images may include more than one bounding box and therefore may correspond to multiple vectors. The training module can input the training images and corresponding vectors into the neural network to train it.

[0037] Return to reference Figure 1 The bounding box detection module 180 can receive from the neural network coordinates and one or more metrics representing one or more bounding boxes used to insert media content into candidate images. The bounding box detection module 180 can receive, for example... Figure 3The data structure is shown in the figure. The bounding box detection module 180 passes one or more bounding boxes to the overlay generation module 190.

[0038] Overlay generation module 190 may include hardware, software, or a combination of both. Overlay generation module 190 may generate a request to display media content items in bounding boxes received from bounding box detection module 180. The request may include one or more metrics of one or more bounding boxes. Overlay generation module 190 may pass the request to communication module 170, which may send the request to media content storage system 100 (e.g., to communication module 110).

[0039] Communication module 110 can receive the request and pass it to media content selection module 120. Communication module 110 may include hardware, software, or a combination of both. Media content selection module 120 can extract the content associated with each bounding box (e.g., ...). Figure 3 The media content selection module 120 retrieves (e.g., from a database) data associated with one or more data structures 300, and searches for media content whose metrics match those of one or more bounding boxes. The media content selection module 120 can obtain (e.g., from a database) one or more media content items that match the metrics of one or more bounding boxes, and generate a response to the request that includes the one or more media content items. In some embodiments, in addition to each media content item, the response may also include metadata associated with each media content item. The metadata may include the metric of the corresponding media content item and other information (e.g., name, type (image, video, etc.), orientation, and / or other suitable information). The media content selection module 120 may pass the response to the communication module 110. The communication module 110 may send the response to the media content insertion system 160.

[0040] In some implementations, the response may include one or more identifiers for one or more media content items. Each identifier may be part of a data structure that also includes a metric for each media content item. The data structure may include other information (e.g., name, type (image, video, etc.)). This can be advantageous because the media content items may be large, and the media content items may take longer to transmit compared to transmitting only the identifier. In some implementations, the response may include a link to each media content item so that it can be retrieved.

[0041] The media content insertion system 160 can use the communication module 170 to receive a response to a request, which includes one or more media content items. The communication module 170 can then pass the response to the overlay generation module 190. The overlay generation module 190 can include hardware, software, or a combination of both. The overlay generation module 190 can extract the one or more media content items from the response.

[0042] In some implementations, the overlay generation module 190 can receive multiple media content items and select one of the received media content items to be displayed based on the metric of the content item that matches the metric of the bounding box with the highest probability of being a valid bounding box. For example, in response to this request, the media content insertion system 160 can receive multiple media content items corresponding to multiple bounding boxes. The overlay generation module 190 can identify a specific media content item from the multiple media content items that corresponds to the bounding box with the highest probability. The overlay generation module 190 can access the metadata of each received media content item and obtain the metric of each media content item. The overlay generation module 190 can obtain the probability of each detected bounding box and determine which bounding box has the highest probability.

[0043] The overlay generation module 190 can obtain the metrics of the bounding boxes with the highest probability and compare these metrics with the metrics of the media content items. The overlay generation module 190 can select the media content items that best match these metrics. For example, some media content items may be too large, so the overlay generation module 190 can filter out these items. The overlay generation module 190 can select the media content items whose metrics are closest to the metrics of the selected bounding boxes. In some implementations, the overlay generation module 190 can take into account the orientation of the media content items. For example, if a particular media content item should be displayed in a portrait orientation, and the orientation of the bounding box will force the media content item to be lateral, then based on that metric, the overlay generation module 190 can filter (e.g., remove from consideration) that particular media content item. When the overlay generation module 190 selects media content items, it can display candidate images and media content items overlaid on those candidate images within the bounding boxes.

[0044] Figure 4 Various media content items are shown overlaid on candidate images. Image 402 shows a media content item within a bounding box. In this example, the measure of the bounding box is not apparent in the overlay. Image 404 shows an embodiment of a bounding box adapted to the media content item. The bounding box can be indicated by a box surrounding the media content item. In this case, the media content item can be text or an image. In some embodiments, the media content item can be associated with a selectable graphical user element that, when selected, instructs a computing device to open a link to, for example, a webpage associated with the content of the bounding box.

[0045] Image 406 illustrates another media content item that can use the same bounding box. It should be noted that the media content items in images 404 and 406 are different. The media content item in image 404 has a transparent portion, while the media content item in image 406 does not. Images 408, 410, and 412 illustrate other embodiments of the bounding box on the same image.

[0046] The overall process of using neural networks

[0047] Figure 5 This is a flowchart illustrating the overall process of detecting a space suitable for overlaying media content on an image using a neural network-based method (such as performed by a neural network-based media content insertion system 160) according to one embodiment. Various embodiments may be performed in a different order than indicated herein. Figure 6 The steps described herein. Furthermore, other implementations may include steps that differ from and / or are additional to those described herein.

[0048] At 502, the media content insertion system 160 receives candidate images for media content overlay. For example, the media content insertion system 160 can receive the candidate images over a network using a network interface device. The media content insertion system 160 can store the candidate images in memory.

[0049] At 504, the media content insertion system 160 inputs candidate images into a neural network that has been trained using training data, which includes multiple images and one or more corresponding bounding boxes for each of the multiple images. The neural network may be stored in memory. The neural network may include an application programming interface for input images and corresponding metadata. The media content insertion system 160 may use one or more processors to execute the input, and may use those same processors to process the candidate images within the neural network.

[0050] At 506, the media content insertion system 160 receives from the neural network coordinates and one or more metrics representing one or more bounding boxes used to insert media content into candidate images. The media content insertion system 160 may receive the coordinates and metrics and store them in memory.

[0051] At 508, the media content insertion system 160 sends a request to display a media content item within one or more bounding boxes, the request including one or more measures of the one or more bounding boxes. For example, the media content insertion system 160 may use a network interface device to send the request over a network.

[0052] At 510, the media content insertion system 160 responds to the request by receiving a media content item. For example, the media content insertion system 160 may receive the request over a network using a network interface device. At 512, the media content insertion system 160 causes the display of a candidate image and the media content item overlaid on the candidate image within a bounding box. For example, the media content insertion system 160 may send the overlaid candidate image to a client device for display. In some embodiments, the media content insertion system 160 may use a visual interface to cause the display of the overlaid candidate image.

[0053] Insert media content into video content items using neural networks

[0054] In some implementations, the image may be a frame of a video content item. The media content insertion system 160 may process these frames in different ways. For example, the media content insertion system 160 may determine that a candidate image is a video frame associated with a video content item. The overlay generation module 190 may make this determination based on an indicator in the image metadata or based on another signal received along with the candidate image.

[0055] The bounding box detection module 180 can obtain a set of video frames for video content items. For example, the bounding box detection module can request frames using an application programming interface (API). The set of video frames can include video frames that are played after candidate images. For example, a candidate image can be the first frame in a set of consecutive frames to be played back by a video player, and it may be desirable to insert media content items (e.g., advertisements) into at least some of these frames.

[0056] The bounding box detection module 180 can input each video frame in the video frame set into the neural network. For example, the bounding box detection module 180 can input the frames sequentially into the neural network based on the playback order. For each video frame in the video frame set, the bounding box detection module can receive from the neural network the corresponding coordinates and corresponding metrics representing one or more bounding boxes. That is, the bounding box detection module 180 can process each video frame in the same way as candidate images to identify one or more bounding boxes in each frame.

[0057] The bounding box detection module 180 can identify bounding boxes in each video frame of the video frame set that match bounding boxes in every other video frame within the video frame set. For example, the bounding box detection module 180 can determine that the same copy space (e.g., a bounding box) is available in all frames of the frame set or at least a subset of the frame set. The bounding box detection module 180 can then include the bounding box in the request. In some embodiments, the bounding box detection module can detect one bounding box in some frames of the frame set and another bounding box in other frames. Based on this, the bounding box detection module 180 can include both bounding boxes in the request. After receiving a response to the request, the overlay generation module 190 can display the video frame set and media content items overlaid on each of a plurality of subsequent video frames within the bounding boxes. That is, the video can be played back accompanied by one or more bounding boxes included in one or more consecutive frames.

[0058] Heuristic-based system for copy space detection

[0059] This paper discloses a heuristic-based system, method, and computer-readable storage medium for detecting spaces (i.e., copy spaces, insertion spaces) suitable for overlaying media content onto an image. The system receives an image, which may be an image or a video frame. Typically, an image or video frame may have spaces for inserting media content without covering significant portions of the image. Implementations of the system process the image to determine occupied and unoccupied spaces within the image, and then select regions within the unoccupied spaces for overlaying media content onto the image. The image is processed using various image processing techniques to automatically suggest spaces for inserting media content onto candidate images. The suggested spaces can then be further analyzed using a heuristic rule-based approach to select the insertion space for inserting the media content. The selected insertion space can be defined by a bounding box of the system. Subsequently, one or more media content items can be selected to insert onto the selected bounding box of the image. The system can then cause the image and the selected media content items overlaid on the image within the selected bounding box to be displayed.

[0060] Figure 6An exemplary image 600 is shown. As illustrated, a portion 610 of image 600 may have multiple visual elements, while another portion 620 of image 600 may have no visual elements, or may be identified as having visually non-essential elements (e.g., textured background, blank space, clouds, tree canopy, beach sand, etc.). The embodiments described herein are designed to automatically identify insertion spaces, also referred to herein as bounding boxes, which have visually non-essential elements, and media content items can be inserted into them without visually affecting any visually essential elements of the image. For example, an insertion space 630 may be identified by an embodiment such that any media content can overlay on image 600 within the identified insertion space 630.

[0061] Figures 7A to 7D Exemplary image processing that can be performed on an image according to some implementation schemes is shown.

[0062] Figure 7A It shows Figure 6 The grayscale image 600 depicted herein. In the embodiments described herein, when it is necessary to select an interpolation space for a color image, preprocessing can be performed on the color image to first obtain the corresponding grayscale image 600, and then processed as described herein. In some embodiments, the grayscale image 600 may be blurred (e.g., by using a low-pass filter) to remove noise from the image before further processing occurs.

[0063] Figure 7B The result 710 of edge detection performed on image 600 is depicted. The image discontinuities identified in the gradient image 600 are organized into a set of line segments, i.e., edges, by an edge detection filter. The depicted edge detection image 710 is obtained by applying a ShenCasten edge detection filter to the gradient image 600. A similar result can be obtained using a Sobel filter.

[0064] Figure 7C The result 720 of performing saliency filtering on the edge detection image 710 is depicted. Salient filtering is an image segmentation process that assists in locating objects and boundaries. The depicted saliency-filtered image 720 is obtained by applying a saliency filter to the edge detection image 710.

[0065] Figure 7DThe process of further processing grayscale image 600 using a single-lens text detector (SSD) network and standard classifiers, along with object detection, to mask text, faces, and people is depicted as a result 730. This processing identifies regions 732 and 734, which are subsequently masked as occupied space by person and face detection classifiers. Although not depicted herein, any text elements in grayscale image 600 can also be masked by processing image 600 using the SSD network. In embodiments described herein, this masking of text, faces, and people can also be performed as a separate process on grayscale image 600, in addition to the image processing described herein, to provide further indication of occupied and unoccupied space.

[0066] Figures 8A to 8E An exemplary image processing method, according to an embodiment, can be performed on an image to generate a binary matrix.

[0067] Figure 8A The result of performing local adaptive binaryization on the saliency-filtered image 730 is depicted 810.

[0068] Figure 8B A foreground / background separation image 820 is depicted, generated from the original grayscale image 600 and the saliency-filtered image 730, on which adaptive binary conversion has been performed, namely on image 810.

[0069] Figure 8C The results of using a visual entropy filter on the foreground / background separated image 820 to generate a visual entropy filtered image 830 are depicted.

[0070] Figure 8D The diagram depicts the generation of a binary matrix image 840 from a visual entropy-filtered image 830 using downsampling. The binary matrix image 840 has values ​​of 0 and 1 for each pixel (representing black and white values, respectively). Therefore, unoccupied space in the binary matrix image, or the downsampled binary matrix image, can be considered, but is not limited to, spaces composed of pixel values ​​of "0".

[0071] Figure 8E Image 850 is depicted, in which candidate bounding boxes have already been suggested in the binary matrix image 840. A set of candidate rectangles satisfying predefined threshold parameters (e.g., minimum width, minimum height, etc.) can be constructed using the "maximum rectangle in binary matrix" technique. Image 850 depicts two exemplary bounding box candidates, 852 and 854.

[0072] Figure 9AAn embodiment of applying heuristic rules to select the best bounding box is described. The best bounding box candidate can be selected first using a non-maximum suppression technique based on an intersection-over-union (IoU) method incorporating a confidence threshold. In image 600, this results in the selection of bounding box 852 as the best bounding box candidate. A subsequent heuristic rule can be applied to the best bounding box candidate, which involves a grid-capturing method where an imaginary grid is overlaid on the image, and the best bounding box is selected by capturing the best bounding box candidate onto the overlaid grid. Therefore, a grid 910 can be applied to image 600, and the best bounding box 900 can be selected as the insertion space for the media content item through the grid-capturing process, because when grid 910 is captured, bounding box 900 can be the bounding box closest to bounding box 852.

[0073] Figure 9B Another embodiment of applying heuristic rules to select the best bounding box is described. Once the best bounding box candidates are selected (e.g., using non-maximum suppression techniques based on the intersection-over-union (IoU) method), another heuristic applied in some embodiments involves a pyramid column grouping method. In this method, the image can be divided into multiple columns, which are themselves grouped in a pyramid style. For example, the image can be divided into twelve columns, four groups of three columns each, three groups of four columns each, two groups of six columns each, and a last group of all twelve columns. The column group that shows the best (i.e., within a predefined minimum threshold) coverage with the best bounding box candidates is selected as the best bounding box. Therefore, in Figure 9B In the image, 600 is divided into columns 920. The six columns are then grouped into groups 950, which are selected as the coverage groups closest to the best candidate bounding box 852. The insertion space for the media content items is selected as space 950.

[0074] Figure 9C Another image 960 is shown, showing two exemplary corresponding bounding boxes (bounding box 970 and bounding box 980). Figure 9C The diagram illustrates that, according to some implementations, heuristic rules can be applied to define the preferred shape for selecting the best bounding box, and more than one best bounding box can be selected to insert media content items. Therefore, bounding box 970 is a rectangle, while bounding box 980 is an ellipse.

[0075] Figure 10 This is a block diagram illustrating components of a heuristic media content insertion system 1000 according to some embodiments. The heuristic media content insertion system 1000 includes an image receiving module 1010, an image processing module 1020, an insertion space suggestion module 1030, a bounding box selection module 1040, a media content selection module 1050, a media content overlay module 1060, and a data storage module 1070. Other embodiments may include... Figure 10The modules shown may include more or fewer modules. Functions indicated to be performed by a particular module may be performed by modules other than those indicated herein. Furthermore, the steps of any process described herein may be performed in a different order than that shown herein.

[0076] Image receiving module 1010 receives images used to overlay media content items. Image receiving module 1010 can receive images via a network or can receive instructions to retrieve images from image storage. The image can be any appropriately formatted image. In some embodiments, the image can be a frame of a video content item also received by image receiving module 1010. In some embodiments, image receiving module 1010 can receive a video content item and extract image frames from the received video content item for further processing. In some embodiments, image receiving module 1010 can receive an identifier for a specific starting image frame (e.g., a time value associated with the specific frame) such that the display of video content starting from that frame requires the insertion of media content. In these embodiments, the specific starting image frame and image frames in the video content following the specific image frame can be processed by a heuristic-based media content insertion system 1000. In some embodiments, image receiving module 1010 can receive an identifier for a specific ending image frame (e.g., an end time value, a start time value, and a time interval, etc.). In these embodiments, the image frames extracted and stored by the heuristic-based media content insertion system 1000 for processing may include frames appearing from the starting image frame up to and including the ending image frame. For example, the starting image frame may be the first frame in a set of consecutive frames to be played back by a video player, and it may be desirable to insert media content items (e.g., advertisements) into at least some of these frames. The image receiving module 1010 may pass one or more images to the image processing module 1020, or it may store the received images in the data storage 1070 for access by other modules of the heuristic-based media content insertion system 1000.

[0077] Image processing module 1020 processes images using one or more image processing techniques. Image processing module 1020 receives images from image receiving module 1010. In some embodiments, image processing module 1020 may obtain images from data storage 1070. In one embodiment, the image may be a color image. In this case, image processing module 1020 may first obtain a corresponding grayscale image. In some embodiments, grayscale image 100 may be blurred (e.g., by using a low-pass filter) to remove noise from the image before further processing. The grayscale image can then be filtered using a gradient filter. Gradient filtering is performed to perform edge detection in the image, i.e., identifying pixel locations where image brightness changes by at least some predefined threshold. An embodiment of the gradient filter is the Sobel gradient filter. In some embodiments, the image processing module 1020 may use an edge detection filter instead of a gradient filter to filter the grayscale image. Image processing module 1020 may use any suitable edge detection filter, such as a ShenCasten filter, a Sobel filter, or a Canny edge detector. The image processing module 1020 then uses an image saliency filter that performs local image segmentation to process the edge-detected image.

[0078] Some implementations of the image processing module 1020 can use complementary machine learning techniques to identify text, people, and faces in grayscale or RGB images—for example, a single-lens text detector (SSD) network that identifies word locations can be used to locate text in an image, while an object classifier can be used to locate people and faces in an image. The identified text, people, and face regions in the image can then be masked into the occupied space within the image.

[0079] Image processing module 1020 performs adaptive binaryization on the saliency-filtered image, uses the adaptive binaryized image and the original grayscale image to perform foreground / background estimation and generate a foreground / background estimated image. Visual entropy filtering can be performed by image processing module 1020 to generate a binary matrix image. This binary matrix image can then be downsampled. Image processing module 1020 can obtain predefined parameter values ​​from data storage 1070 for performing the various image processing techniques described herein. In some embodiments, image processing module 1020 can store the downsampled binary matrix image in the data storage. In some embodiments, image processing module 1020 can send the downsampled binary matrix to insertion space proposal module 1030.

[0080] The insertion space proposal module 1030 generates one or more proposal spaces in the image for media content item coverage. The insertion space proposal module 1030 receives a downsampled binary matrix image generated by the image processing module 1020. The insertion space proposal module 1030 can obtain the downsampled binary matrix image from the data storage, or it can receive the downsampled binary matrix image from the image processing module 1020. Module 1030 can also obtain predefined minimum threshold parameters (e.g., rectangle width, rectangle height, and rectangle area, etc.) from the data storage 1070.

[0081] A binary matrix image has 0 and 1 values ​​for each pixel (representing black and white values, respectively). Therefore, unoccupied space in a binary matrix image or a downsampled binary matrix image can be considered, but is not limited to, spaces with pixel values ​​of "0". The insertion space proposal module 1030 determines rectangular spaces that satisfy a predefined minimum threshold parameter and are suitable for unoccupied space in the downsampled binary matrix image received from the image processing module 1020. In determining the rectangular spaces, the insertion space proposal module 1030 may use a suitable variant of the "maximum rectangle in binary matrix" technique. The "maximum rectangle in binary matrix" technique determines possible rectangles residing in unoccupied space within the downsampled binary matrix that satisfy a predefined minimum threshold parameter. In some embodiments, all determined rectangular spaces (i.e., rectangles) may be placed in a list sorted by area. In some embodiments, the insertion space proposal module 1030 may store the sorted list of proposed rectangles in the data storage 1070. In some embodiments, the insertion space proposal module 1030 may provide the sorted list of rectangular spaces to the bounding box selection module 1040 as one or more proposed spaces for media content item coverage in the received image. In some implementations, each rectangle in the sorted list can be associated with a confidence level. An exemplary confidence level associated with a rectangle in the sorted list can be based on the space occupied within the rectangle when superimposed on a binary matrix image generated before downsampling.

[0082] Bounding box selection module 1040 applies one or more heuristic rules to automatically select bounding boxes in an image from one or more suggested spaces for media content item coverage. Bounding box selection module 1040 receives a suggested set of rectangles and associated confidence scores from insertion space suggestion module 1030. The suggested set of rectangles may be in the form of a sorted list of rectangles generated by insertion space suggestion module 1030. A bounding box refers to an area on an image selected as the insertion space for placing a media content item. In one embodiment, multiple bounding boxes may be generated based on the number of media content items that may need to be inserted into the image. Information about the number of media content items that may need to be inserted can be obtained by bounding box selection module 1040 from data storage. In the embodiments described herein, the bounding box can be any shape, including but not limited to rectangles, squares, circles, polygons, and any other suitable shape. The bounding box can be an irregular measure and / or a combination of multiple shapes. For example, the bounding box can be a square (e.g., represented by coordinates and an offset), a circle (e.g., represented by coordinates as the center of the circle and a measure representing the radius or diameter of the circle), or an ellipse (e.g., represented by two radii and coordinates at the center). In some implementations, the bounding box can be described by a vector representing the coordinates of points on the candidate image. The vectors and the connections between them can represent the bounding box.

[0083] The bounding box selection module 1040 applies heuristic rules to a sorted list of rectangles to select bounding boxes from the suggested rectangles in the sorted list for a target number of media content items to be covered. The target number of bounding boxes can be based on the target number of media content items that need to be covered on the image. The heuristic rules applied to select the target number of bounding boxes for media content item insertion in the image can include heuristic rules based on core design principles. Such heuristic rules can include rules that enforce similarity between rectangles, enforce proximity between rectangles, enforce closure of rectangle combinations to generate a combined irregular polygonal space, etc. The heuristic rules applied by the bounding box selection module 1040 can have associated predefined priorities that define the order in which the heuristic rules can be applied. The bounding box selection module 1040 can retrieve the heuristic rules and associated predefined threshold parameters for applying the heuristic rules from the data storage 1070.

[0084] In some implementations, the bounding box selection module 1040 can trim the sorted list generated by the insertion space proposal module 1030 by selecting a single rectangle from each subset of rectangles in the list that meets a predefined threshold criterion for similarity (i.e., retaining only dissimilar rectangles). The predefined similarity threshold parameters can be a combination of parameters such as the location, area, width, and height of the proposed rectangles (e.g., a heuristic for similarity could be "two proposed rectangles are similar if their centroids lie on the image at a distance less than a predefined threshold and their aspect ratios differ by less than a predefined threshold," and a heuristic for trimming could be "if a subset of rectangles is marked as similar, then all rectangles are replaced with a single rectangle that is a weighted combination of the rectangles in the subset," etc.). In some implementations, the bounding box selection module 1040 can use a variant of the iterative method of "non-maximum suppression," which uses an intersection-over-union (IoU) ratio associated with the confidence of the proposed rectangle pairs to select one or more rectangles.

[0085] In some implementations, heuristic rules can be applied to modify the selected rectangles to a minimum required extent to ensure that the bounding boxes are appended (i.e., captured) to the resulting rectangles superimposed on the image. In some implementations, another heuristic rule may involve a pyramid column grouping method. In this method, the image can be divided into multiple adjacent columns, which are themselves grouped in a pyramidal pattern. For example, the image can be divided into twelve columns: four groups of three adjacent columns each, three groups of four adjacent columns, two groups of six adjacent columns each, and a final group of all twelve columns. Heuristic rules can be applied to determine the grouping of adjacent columns from the grouping of columns that best overlap with the selected rectangle for each of the selected rectangles. Thus, based on determining the grouping of adjacent columns from multiple groupings of columns superimposed on the image, such that the determined grouping of columns satisfies a predefined minimum threshold of overlap with the selected rectangle, the generated bounding boxes in the image used for media content overlay can be selected.

[0086] In some implementations, heuristic rules can specify predefined target attributes for the generated bounding boxes, such as selecting a predefined number of bounding boxes, target area, target shape, target aspect ratio, and target location of the bounding boxes. For example, heuristic rules can specify a preference for media content overlay, which takes the form of "a single banner centered on the image." In some implementations, if additional information providing estimated depth values ​​to pixel locations in the image is available, these depth estimates can also be used to select bounding boxes.

[0087] In some embodiments, the bounding box selection module 1040 can generate a bounding box as a weighted combination of one or more selected rectangles, with weights based on the confidence level associated with each of the selected rectangles. The bounding box can be described by parameters describing its position in the received image and parameters describing its layout in the received image. For example, the bounding box can be described by a rectangle comprising one or more coordinates on the image and one or more measures extending from those coordinates. The coordinates can represent a specific pixel on the image. In some embodiments, the bounding box can be represented by coordinates and two measures, the coordinates being counted based on the number of horizontal and vertical pixels starting from the top-left corner of the image. The coordinates can represent the top-left corner of the bounding box. The two measures can include a vertical offset and a horizontal offset extending from the coordinates. Based on the coordinates and offsets, a rectangular bounding box with an area based on the offset and a position based on the coordinates can be generated. In the embodiments described herein, other descriptions of the bounding box are possible, such as an ellipse as the circumscribed ellipse of a weighted combination of the closest selected rectangles, etc. In some embodiments, the insertion space selection module 1040 can store the description of the bounding box in data storage 1070. In some implementations, the bounding box selection module 1040 may send a description of the bounding box to the media content overlay module 1060.

[0088] The media content selection module 1050 can receive descriptions of one or more bounding boxes. In some embodiments, the media content selection module 1050 can obtain descriptions of one or more bounding boxes from the data storage 1070. The media content selection module 1050 can extract metrics from the descriptions associated with each bounding box and search the data storage 1070 for media content items with metrics that best match the extracted metrics. The media content selection module 1050 can select media content items from a plurality of media content items based on the metrics of the content items that match the bounding box metrics. The media content selection module 1050 can obtain the selected one or more media content items from the data storage 1070. In some embodiments, in addition to each selected media content item, the media content selection module 1050 can extract metadata associated with each media content item. The metadata may include the metric of the corresponding media content item and other information (e.g., name, type (image, video, etc.)), orientation, and / or other suitable information. The media content selection module 1050 can send the selected media content items and associated metadata to the media content overlay module 1060. In some implementations, the media content selection module 1050 may alternatively send one or more identifiers of the selected media content items to the media content overlay module 1060. Each identifier may be part of a data structure that also includes a metric for each media content item. The data structure may include other information (e.g., name, type (image, video, etc.)). This may be advantageous because the media content items may be large and may take longer to transmit compared to simply transmitting the identifier.

[0089] The media content overlay module 1060 receives descriptions of one or more bounding boxes. In some embodiments, the descriptions may be received from the bounding box selection module 1040. In some embodiments, the media content overlay module 1060 may obtain the descriptions from the data storage 1070. The media content overlay module 1060 may receive media content items to be displayed in the bounding boxes from the media content item selection module 1050. In some embodiments, the media content overlay module 1060 may obtain the media content items from the data storage 1070.

[0090] As previously described, in some implementations, the image may be an image frame of a video content item. The media content overlay module 1060 may store position and layout information of determined bounding boxes for one or more previous frames, and apply heuristic rules to modify any of the measurements, layouts, and positions of the determined bounding boxes received from the bounding box selection module 1040 to ensure a smooth transition of overlaid media content in each video frame image displayed after the display of the received image when displaying the video content item.

[0091] Data storage 1070 receives and stores data for use and easy access by modules of the heuristic-based media content insertion system 1000. Data storage 1070 may store one or more images received from image receiving module 1010. Data storage 1070 may store binary matrices as well as downsampled binary matrices generated by image processing module 1020. Data storage 1070 stores heuristic rules and threshold parameter values ​​used by processes performed by modules of the media content insertion system 1000, such as target bounding box description parameters, the number of target bounding boxes, target area, position, and shape parameters. Data storage 1070 may store intermediate data used by various modules of the heuristic-based media content insertion system 1000, such as a suggested list of rectangles that can be used to select the actual insertion space. In some embodiments, data storage 1070 may also be used to store media content items and associated metadata parameter values. Data storage 1070 is a memory, such as read-only memory (ROM), dynamic random access memory (DRAM), static random access memory (SRAM), or some combination thereof.

[0092] Overall process

[0093] Figure 11 This is a flowchart illustrating the overall process of detecting a space suitable for overlaying media content on an image, as performed by a heuristic media content insertion system 1000 according to one embodiment. Various embodiments may be performed in a different order than indicated herein. Figure 11 The steps described herein. Furthermore, other implementations may include steps that differ from and / or are additional to those described herein.

[0094] A heuristic-based media content insertion system 1000 receives 1110 images for covering media content items. For example, the heuristic-based media content insertion system 1000 can receive the 1110 images via a network, or it can receive instructions for retrieving images from image storage. The heuristic-based media content insertion system 1000 can also receive 1110 video content items and can extract one or more image frames from the video content items.

[0095] The heuristic-based media content insertion system 1000 processes the image received by 1120 using one or more image processing techniques to automatically generate one or more proposal spaces in the image for media content item coverage. The image processing techniques performed by the heuristic-based media content insertion system 1000 include any combination of generating grayscale images, blurring images, performing gradient filtering or edge detection filtering, saliency filtering, using supplementary machine learning methods such as SSD networks and classifiers for text, person, and face detection, foreground / background estimation, visual entropy filtering, generating binary matrices, and downsampling. The heuristic-based media content insertion system 1000 determines rectangular spaces in the binary matrix of the image that satisfy a predefined minimum threshold parameter, and covers text and faces in these rectangular spaces within the generated binary matrix. Subsequently, the heuristic-based system 1000 generates a sorted list of rectangular spaces, where the sorting is performed based on predefined parameters (e.g., area), and provides the rectangular spaces in the generated sorted list as one or more proposal spaces in the image for media content item coverage.

[0096] The heuristic-based media content insertion system 1000 applies 1130 heuristic rules to automatically select one or more bounding boxes from one or more suggestion spaces in an image for media content item coverage. The heuristic rules can be preferentially applied by the heuristic-based media content insertion system 1000. The heuristic rules may include, for example, (i) generating bounding boxes based on modifying one or more suggestion spaces to lie on a grid superimposed on the image, (ii) dividing the image into multiple adjacent columns, grouping the columns in a pyramid manner, and generating bounding boxes by selecting a set of adjacent columns that overlap with the suggestion spaces by at least a predefined threshold (e.g., at least 80% of the selected rectangle overlaps with the selected set of adjacent columns), (iii) generating bounding boxes that are weighted combinations of one or more suggestion spaces, and (iv) generating bounding boxes using one or more suggestion spaces based on predefined target attributes, wherein the predefined target attributes include, for example, the number of bounding boxes, the area, shape, aspect ratio, and position of the bounding boxes, as well as other heuristic rules.

[0097] A heuristic media content insertion system 1000 selects 1140 media content items to overlay corresponding bounding boxes selected in an image. In some embodiments, the heuristic media content insertion system 1000 may receive descriptions of one or more bounding boxes and select media content items from a plurality of media content items based on the measure of the media content item that matches the measure of the received bounding box description.

[0098] The heuristic-based media content insertion system 1000 causes 1150 to display the image and overlay one or more media content items selected within one or more corresponding bounding boxes in the image. For example, the heuristic-based media content insertion system 1000 may transmit candidate images for overlay to a client device for display. In some embodiments, the media content insertion system 1000 may use a visual interface to cause the display of candidate images for overlay.

[0099] Computer architecture

[0100] Figure 1 The media content storage system 100 and the neural network-based media content insertion system 160 depicted in the figure are as follows: Figure 10 Each of the heuristic-based media content insertion systems 1000 described in the text can include Figure 12 One or more components depicted in the text. Figure 12 This is a block diagram illustrating the components of an example machine capable of reading instructions from a machine-readable medium and executing them in a processor (or controller). Specifically, Figure 12 A schematic representation of a machine in an example form of computer system 1200 is shown, wherein program code (e.g., software) can be executed to cause the machine to perform any one or more of the methods discussed herein. The program code may include instructions 1224 executable by one or more processors 1202. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may run as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0101] The machine can be a server computer, client computer, personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular phone, smartphone, network device, network router, switch, or bridge, or any machine capable of executing instructions 1224 (sequential or otherwise) specifying the action to be taken by the machine. Furthermore, although only a single machine is shown, the term "machine" should also be understood to include any collection of machines that individually or jointly execute instructions 1224 to perform any one or more of the methods discussed herein.

[0102] Example computer system 1200 includes a processor 1202 (e.g., a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), one or more application-specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof), main memory 1204, and static memory 1206, configured to communicate with each other via bus 1208. Computer system 1200 may also include a visual display interface 1210. The visual interface may include software drivers capable of displaying a user interface on a screen (or monitor). The visual interface may display the user interface directly (e.g., on a screen) or indirectly (e.g., via a visual projection unit) on a surface, window, etc. For ease of discussion, the visual interface may be described as a screen. Visual interface 1210 may include a touch-enabled screen, or may be interfaced with it. The computer system 1200 may also include an alphanumeric input device 1212 (e.g., a keyboard or touchscreen keyboard), a cursor control device 1214 (e.g., a mouse, trackball, joystick, motion sensor or other pointing instrument), a storage unit 1216, a signal generation device 1218 (e.g., a speaker), and a network interface device 1220, which is also configured to communicate via a bus 1208.

[0103] Storage unit 1216 includes machine-readable medium 1222 thereon storing instructions 1224 (e.g., software) that implement one or more of the methods or functions described herein. During execution of the instructions 1224 (e.g., software) by computer system 1200, the instructions 1224 may also reside wholly or at least partially within main memory 1204 or processing device 1102 (e.g., processor cache memory), which also constitute machine-readable storage media. The instructions 1224 (e.g., software) may be sent or received on network 1226 via network interface device 1220.

[0104] Although machine-readable medium 1222 is shown as a single medium in the example embodiment, the term "machine-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) capable of storing instructions (e.g., instruction 1224). The term "machine-readable medium" should also be understood to include any medium capable of storing instructions (e.g., instruction 1224) for machine execution and causing the machine to perform any one or more methods disclosed herein. The term "machine-readable medium" includes, but is not limited to, data repositories in the form of solid-state memory, optical media, and magnetic media.

[0105] Figure 12The components can be used in systems or processes that use neural network-based methods and heuristic-based methods to detect spaces suitable for overlaying media content on images.

[0106] Other configuration considerations

[0107] Throughout this specification, multiple instances may implement the components, operations, or structures described as single instances. Although individual operations of one or more methods are shown and described as separate operations, one or more of these separate operations may be performed simultaneously, and they need not be performed in the order shown. Structures and functions presented as separate components in the example configuration may be implemented as combined structures or components. Similarly, structures and functions presented as single components may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of this document's subject matter.

[0108] Some implementations herein are described as including logic or multiple components, modules, or mechanisms. Modules can constitute software modules (e.g., code contained on a machine-readable medium or in transmitted signals) or hardware modules. A hardware module is a tangible unit capable of performing a particular operation and can be configured or arranged in a particular manner. In example implementations, one or more computer systems (e.g., standalone, client, or server computer systems) or one or more hardware modules (e.g., processors or processor groups) of a computer system can be configured as hardware modules by software (e.g., an application or application portion) that operates to perform certain operations as described herein.

[0109] In various implementations, the hardware module can be implemented mechanically or electronically. For example, a hardware module may include dedicated circuitry or logic that is permanently configured (e.g., as a dedicated processor, such as a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC)) to perform certain operations. A hardware module may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations (e.g., contained within a general-purpose processor or other programmable processor). It should be understood that the decision to implement the hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., via software configuration) may be driven by cost and time considerations.

[0110] Therefore, the term "hardware module" should be understood to include tangible entities that are physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate or perform the specific operations described herein in a particular manner. As used herein, "hardware implementation module" refers to a hardware module. Considering implementation schemes where hardware modules are temporarily configured (e.g., programmed), each hardware module does not need to be configured or instantiated in a timely manner in any single instance. For example, in the case where hardware modules include a general-purpose processor configured using software, the general-purpose processor can be configured as different hardware modules at different times. The software can configure the processor accordingly, for example, constituting a specific hardware module at one time and different hardware modules at different times.

[0111] Hardware modules can provide information to and receive information from other hardware modules. Therefore, the described hardware modules can be considered communicatively linked. In the presence of multiple such hardware modules, communication can be achieved through signal transmission connecting the hardware modules (e.g., via appropriate circuitry and buses). In implementations where multiple hardware modules are configured or instantiated at different times, communication between these hardware modules can be achieved, for example, by storing and retrieving information in a memory structure accessible to the multiple hardware modules. For example, one hardware module can perform an operation and store the output of that operation in a memory device communicatively linked to it. Another hardware module can then access the memory device at a later time to retrieve and process the stored output. Hardware modules can also initiate communication with input or output devices and can operate on resources (e.g., information sets).

[0112] The various operations of the example methods described herein can be performed at least in part by one or more processors, which are temporarily (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented modules whose operations are to perform one or more operations or functions. In some example implementations, the modules mentioned herein may include processor-implemented modules.

[0113] Similarly, the methods described herein can be implemented at least in part by a processor. For example, at least some operations of the methods can be performed by one or more processors or hardware modules implemented by processors. The execution of certain operations can be distributed among one or more processors, residing not only within a single machine but also deployed across multiple machines. In some example embodiments, one or more processors may reside in a single location (e.g., in a home environment, an office environment, or as a server farm), while in other embodiments, the processors may be distributed across multiple locations.

[0114] One or more processors may also operate to support the execution of related operations in a “cloud computing” environment or as “Software as a Service” (SaaS). For example, at least some operations may be performed by a set of computers (as an embodiment of a machine including processors), and these operations may be accessible via a network (e.g., the Internet) and via one or more suitable interfaces (e.g., application programming interfaces (APIs)).

[0115] The execution of certain operations can be distributed across one or more processors, residing not only within a single machine but also deployed across multiple machines. In some example implementations, one or more processors or processor-implemented modules may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example implementations, one or more processors or processor-implemented modules may be distributed across multiple geographic locations.

[0116] Some portions of this specification are presented based on algorithms or symbolic representations of operations performed on data stored as bit or binary digital signals in machine memory (e.g., computer memory). These algorithms or symbolic representations are embodiments of techniques used by those skilled in the art of data processing to convey the essence of their work to others skilled in the art. As used herein, an "algorithm" is a self-consistent sequence of operations or similar processing that leads to a desired result. In this context, algorithms and operations involve the physical manipulation of physical quantities. Typically, but not necessarily, these quantities may take the form of electrical, magnetic, or optical signals that can be stored, accessed, transmitted, combined, compared, or otherwise manipulated by a machine. Primarily for common reasons, it is sometimes convenient to use terms such as "data," "content," "bit," "value," "element," "symbol," "character," "term," "number," "numerical value," etc., to refer to such signals. However, these terms are merely convenient labels and are associated with appropriate physical quantities.

[0117] Unless otherwise specified, the discussion herein using terms such as “processing,” “computing,” “calculating,” “determining,” “presenting,” “displaying,” etc., may refer to the actions or processes of a machine (e.g., a computer) that operate or transform data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0118] As used herein, any reference to “an embodiment” or “implementation” means that a particular element, feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. The phrase “in an embodiment” appearing in various places in the specification does not necessarily refer to the same embodiment.

[0119] Some embodiments can be described using the terms "connection" and "linkage," as well as their derivatives. It should be understood that these terms are not synonyms. For example, the term "connection" can be used to describe some embodiments to indicate that two or more elements are in direct physical or electrical contact with each other. In another embodiment, the term "linkage" can be used to describe some embodiments to indicate that two or more elements are in direct physical or electrical contact. However, the term "linkage" can also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. Embodiments are not limited to this context.

[0120] As used herein, the terms “comprising,” “including,” “including,” “having,” or any other variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such a process, method, article, or apparatus. Furthermore, unless expressly stated to the contrary, “or” refers to an inclusive or, not an exclusive, or. For example, conditions A or B satisfy any of the following: A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); and both A and B are true (or exist).

[0121] Furthermore, the terms "a" or "an" are used to describe elements and components of the embodiments described herein. This is done merely for convenience and to give the general meaning of the invention. This description should be understood to include one or at least one, and the singular includes the plural, unless it is obvious otherwise.

[0122] Upon reading the disclosed text, those skilled in the art will understand additional alternative structural and functional designs for systems and processes used to insert media content into available copy space on an image. Therefore, although specific embodiments and applications have been illustrated and described, it should be understood that the disclosed embodiments are not limited to the precise structures and components disclosed herein. Various modifications, alterations, and variations that will be apparent to those skilled in the art may be made in the arrangement, operation, and details of the methods and apparatus disclosed herein without departing from the spirit and scope defined by the appended claims.

Claims

1. A computer-implemented method for detecting a space suitable for overlaying media content on an image, the method comprising: The neural network is trained using multiple images, each of which corresponds to one or more vectors, each vector including (1) coordinates of a point associated with a bounding box in the corresponding image and (2) a metric of the bounding box, wherein each bounding box is suitable for media content coverage; The media content insertion system receives candidate images for media content overlay; The candidate image is input into the neural network, wherein; The neural network outputs one or more bounding boxes as one or more data structures, each of which includes at least (1) coordinates of a point associated with the bounding box, (2) a measure of the bounding box, and (3) the probability that the bounding box is located in an area suitable for the coverage of the media content; For at least one of the one or more bounding boxes, determine whether the probability satisfies a threshold probability; In response to determining that the probability satisfies the threshold probability, the bounding box is determined to be a candidate bounding box of the candidate image; The data structure corresponding to the candidate bounding box of the candidate image is received from the neural network; Send a request to display media content items within the candidate bounding box of the candidate image, the request including the measure of the candidate bounding box; In response to the request, the media content item is received; and Display the candidate image and the media content item that is overlaid on the candidate image within the candidate bounding box.

2. The method according to claim 1, further comprising: Receive the multiple images; For each of the plurality of images, one or more vectors are received, each vector including a set of coordinates and a set of metrics, wherein each of the one or more vectors represents a specific bounding box; as well as The neural network is trained using each of the plurality of images and one or more corresponding vectors.

3. The method of claim 1, wherein receiving coordinates and one or more metrics representing the one or more bounding boxes for inserting the media content into the candidate image includes receiving for each bounding box: A first coordinate representing a first offset along the horizontal axis of the candidate image and a second coordinate representing a second offset along the vertical axis of the candidate image. A first measurement extending from the first coordinate along the horizontal axis, and A second measure extending from the second coordinate along the vertical axis.

4. The method according to claim 1, further comprising: In response to the request, receive multiple media content items corresponding to the one or more bounding boxes; Identify the specific media content item corresponding to the bounding box with the highest probability from the plurality of media content items; and Select the specific media content item as the media content item.

5. The method according to claim 1, further comprising: The candidate image is determined to be a video frame associated with a video content item; Obtain the video frame set of the video content item, wherein the video frame set contains video frames played after the candidate image; Each video frame in the video frame set is input into the neural network; For each video frame in the video frame set, the neural network receives the corresponding coordinates and one or more corresponding measures representing one or more bounding boxes. In each video frame of the video frame set, identify a bounding box that matches a bounding box in every other video frame within the video frame set; and The request includes a bounding box that matches the bounding box of each other video frame.

6. The method according to claim 5, further comprising: This causes the display of the set of video frames and the media content items that are overlaid on each of the plurality of subsequent video frames within the bounding box.

7. A system for detecting a space suitable for overlaying media content on an image, the system comprising: A memory containing encoded instructions; and One or more processors, when executing the instructions, cause the one or more processors to perform operations including: The neural network is trained using multiple images, each of which corresponds to one or more vectors, each vector including (1) coordinates of a point associated with a bounding box in the corresponding image and (2) a metric of the bounding box, wherein each bounding box is suitable for media content coverage; The media content insertion system receives candidate images for media content overlay; The candidate image is input into the neural network; The neural network outputs one or more bounding boxes as one or more data structures, each of which includes at least (1) coordinates of a point associated with the bounding box, (2) a measure of the bounding box, and (3) the probability that the bounding box is located in an area suitable for the coverage of the media content; For at least one of the one or more bounding boxes, determine whether the probability satisfies a threshold probability; In response to determining that the probability satisfies the threshold probability, the bounding box is determined to be a candidate bounding box of the candidate image; The data structure corresponding to the candidate bounding box of the candidate image is received from the neural network; Send a request to display media content items within the candidate bounding box of the candidate image, the request including the measure of the candidate bounding box; In response to the request, the media content item is received; and Display the candidate image and the media content item that is overlaid on the candidate image within the candidate bounding box.

8. The system of claim 7, wherein the instructions cause the one or more processors to perform operations including: Receive the multiple images; For each of the plurality of images, one or more vectors are received, each vector including a set of coordinates and a set of metrics, wherein each of the one or more vectors represents a specific bounding box; and The neural network is trained using each of the plurality of images and one or more corresponding vectors.

9. The system of claim 7, wherein receiving coordinates and one or more metrics representing the one or more bounding boxes for inserting the media content into the candidate image includes receiving for each bounding box: A first coordinate representing a first offset along the horizontal axis of the candidate image and a second coordinate representing a second offset along the vertical axis of the candidate image. A first measurement extending from the first coordinate along the horizontal axis, and A second measure extending from the second coordinate along the vertical axis.

10. The system of claim 7, wherein the instructions cause the one or more processors to perform operations including: In response to the request, receive multiple media content items corresponding to the one or more bounding boxes; Identify the specific media content item corresponding to the bounding box with the highest probability from the plurality of media content items; and Select the specific media content item as the media content item.

11. The system of claim 7, wherein the instructions cause the one or more processors to perform operations including: The candidate image is determined to be a video frame associated with a video content item; Obtain the video frame set of the video content item, wherein the video frame set contains video frames played after the candidate image; Each video frame in the video frame set is input into the neural network; For each video frame in the video frame set, the neural network receives the corresponding coordinates and one or more corresponding measures representing one or more bounding boxes. In each video frame of the video frame set, identify a bounding box that matches a bounding box in every other video frame within the video frame set; and The request includes a bounding box that matches the bounding box of each other video frame.

12. The system of claim 11, wherein the instructions cause the one or more processors to perform the operation of displaying the set of video frames and the media content items that are overlaid on each of a plurality of subsequent video frames within the bounding box.

13. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the following operations: The neural network is trained using multiple images, each of which corresponds to one or more vectors, each vector including (1) coordinates of a point associated with a bounding box in the corresponding image and (2) a metric of the bounding box, wherein each bounding box is suitable for media content coverage; The media content insertion system receives candidate images for media content overlay; The candidate image is input into the neural network; The neural network outputs one or more bounding boxes as one or more data structures, each of which includes at least (1) coordinates of a point associated with the bounding box, (2) a measure of the bounding box, and (3) the probability that the bounding box is located in an area suitable for the coverage of the media content; For at least one of the one or more bounding boxes, determine whether the probability satisfies a threshold probability; In response to determining that the probability satisfies the threshold probability, the bounding box is determined to be a candidate bounding box of the candidate image; The data structure corresponding to the candidate bounding box of the candidate image is received from the neural network; Send a request to display media content items within the candidate bounding box of the candidate image, the request including the measure of the candidate bounding box; In response to the request, the media content item is received; and Display the candidate image and the media content item that is overlaid on the candidate image within the candidate bounding box.

14. The non-transitory computer-readable medium of claim 13, further comprising stored instructions that cause the one or more processors to perform the following operations: Receive the multiple images; For each of the plurality of images, one or more vectors are received, each vector including a set of coordinates and a set of metrics, wherein each of the one or more vectors represents a specific bounding box; and The neural network is trained using each of the plurality of images and one or more corresponding vectors.

15. The non-transitory computer-readable medium of claim 13, wherein receiving the coordinates and the one or more measures representing the one or more bounding boxes for inserting the media content into the candidate image includes receiving, for each bounding box: A first coordinate representing a first offset along the horizontal axis of the candidate image and a second coordinate representing a second offset along the vertical axis of the candidate image. A first measurement extending from the first coordinate along the horizontal axis, and A second measure extending from the second coordinate along the vertical axis.

16. The non-transitory computer-readable medium of claim 13, further comprising stored instructions that cause the one or more processors to perform the following operations: In response to the request, receive multiple media content items corresponding to the one or more bounding boxes; Identify the specific media content item corresponding to the bounding box with the highest probability from the plurality of media content items; and Select the specific media content item as the media content item.

17. The non-transitory computer-readable medium of claim 13, further comprising stored instructions that cause the one or more processors to perform the following operations: The candidate image is determined to be a video frame associated with a video content item; Obtain the video frame set of the video content item, wherein the video frame set contains video frames played after the candidate image; Each video frame in the video frame set is input into the neural network; For each video frame in the video frame set, the neural network receives the corresponding coordinates and one or more corresponding measures representing one or more bounding boxes. In each video frame of the video frame set, identify a bounding box that matches a bounding box in every other video frame within the video frame set; and The request includes a bounding box that matches the bounding box of each other video frame.

18. The non-transitory computer-readable medium of claim 17, further comprising stored instructions that cause the one or more processors to perform the following operations: This causes the display of the set of video frames and the media content items that are overlaid on each of the plurality of subsequent video frames within the bounding box.

19. A computer-implemented method for detecting a space suitable for overlaying media content on an image, the method comprising: Receive the image used for media content item overlay; The received image is processed using a group of one or more image processing techniques to automatically generate one or more suggestion spaces in the image for media content item coverage; One or more heuristic rules are applied to automatically select one or more bounding boxes from the one or more suggestion spaces in the image for media content item coverage, wherein applying the one or more heuristic rules includes generating bounding boxes based on grouping of adjacent columns, the grouping being in a pyramid style, the grouping being determined by multiple groups of adjacent columns superimposed on the image, the determined grouping satisfying a predefined minimum threshold of overlap with the suggestion spaces in the one or more suggestion spaces; Select one or more media content items to cover the corresponding one or more bounding boxes selected in the image; and Display the image and overlay one or more selected media content items within the corresponding one or more bounding boxes in the image.

20. The method of claim 19, wherein processing the received image using a group of one or more image processing techniques to automatically generate one or more suggestion spaces for media content item coverage further comprises: Generate a binary matrix representation of the received image; Determine one or more rectangular spaces within the generated binary matrix that satisfy a predefined minimum threshold parameter; Cover the text and face within one or more rectangular spaces defined in the generated binary matrix; Generate a sorted list of the rectangular space, wherein the sorting is performed based on predefined parameters; and The rectangular space in the generated sorted list is provided as one or more suggested spaces in the received image for media content item coverage.

21. The method of claim 19, wherein applying one or more heuristic rules to automatically select one or more bounding boxes from the one or more suggestion spaces in the image for media content item coverage includes one or more of the following: A bounding box is generated by modifying one or more proposed spaces to lie on a grid superimposed on the image; Based on determining the grouping of adjacent columns from multiple groups of adjacent columns superimposed on the image, such that the determined grouping of adjacent columns satisfies a predefined minimum threshold of overlap with the proposal space of the one or more proposal spaces, a bounding box is generated. Generate a bounding box, which is a weighted combination of the one or more proposed spaces; and Using the one or more suggestion spaces, bounding boxes are generated based on predefined target attributes.

22. The method of claim 21, wherein the predefined target attribute includes: Number of bounding boxes; The area of ​​the bounding box; The shape of the bounding box; The aspect ratio of the bounding box; and The position of the bounding box.

23. The method of claim 19, wherein the selected one or more bounding boxes are described by parameters describing the position of the bounding boxes in the received image and parameters describing the layout of the bounding boxes in the received image.

24. The method of claim 19, further comprising: It is determined that the received image is a video frame associated with a video content item; Obtain the video frame set of the video content item, wherein the video frame set includes video frames displayed after the received image is displayed; Each video frame in the acquired video frame set is processed using a group of one or more image processing techniques to automatically generate one or more suggestion spaces for media content item coverage in each video frame; Apply one or more heuristic rules to automatically select one or more bounding boxes from the one or more suggestion spaces in each video frame for media content item coverage; and Display each video frame image and overlay one or more selected media content items within the corresponding one or more bounding boxes in each video frame.

25. The method of claim 24, wherein applying one or more heuristic rules to automatically select one or more bounding boxes from the one or more suggestion spaces in each video frame for media content item coverage further comprises: Ensure that when displaying the video content item, there is a smooth transition of overlaid media content in each of the video frame images displayed after the display of the received image.

26. A non-transitory computer-readable storage medium for storing instructions, which, when executed by a computer processor, cause the computer processor to perform a step of detecting a space suitable for overlaying media content on an image, the step comprising: Receive the image used for media content item overlay; The received image is processed using a group of one or more image processing techniques to automatically generate one or more suggestion spaces in the image for media content item coverage; One or more heuristic rules are applied to automatically select one or more bounding boxes from the one or more suggestion spaces in the image for media content item coverage, wherein applying the one or more heuristic rules includes generating bounding boxes based on grouping of adjacent columns, the grouping being in a pyramid style, the grouping being determined by multiple groups of adjacent columns superimposed on the image, the determined grouping satisfying a predefined minimum threshold of overlap with the suggestion spaces in the one or more suggestion spaces; Select one or more media content items to cover the corresponding one or more bounding boxes selected in the image; and Display the image and overlay one or more selected media content items within the corresponding one or more bounding boxes in the image.

27. The non-transitory computer-readable storage medium of claim 26, wherein processing the received image using one or more image processing techniques of a group to automatically generate one or more suggested spaces for media content item coverage further comprises: Generate a binary matrix representation of the received image; Determine one or more rectangular spaces within the generated binary matrix that satisfy a predefined minimum threshold parameter; Cover text and faces within one or more rectangular spaces defined in the generated binary matrix; Generate a sorted list of the rectangular space, wherein the sorting is performed based on predefined parameters; and The rectangular space in the generated sorted list is provided as one or more suggested spaces in the received image for media content item coverage.

28. The non-transitory computer-readable storage medium of claim 26, wherein applying one or more heuristic rules to automatically select one or more bounding boxes from the one or more suggestion spaces in the image for media content item coverage includes one or more of the following: A bounding box is generated by modifying one or more proposed spaces to lie on a grid superimposed on the image; Based on determining the grouping of adjacent columns from multiple groups of adjacent columns superimposed on the image, such that the determined grouping of adjacent columns satisfies a predefined minimum threshold of overlap with the proposal space of the one or more proposal spaces, a bounding box is generated. Generate a bounding box, which is a weighted combination of the one or more proposed spaces; and The bounding box is generated using one or more suggestion spaces based on predefined target attributes.

29. The non-transitory computer-readable storage medium of claim 28, wherein the predefined target attribute includes: Number of bounding boxes; The area of ​​the bounding box; The shape of the bounding box; The aspect ratio of the bounding box; and The position of the bounding box.

30. The non-transitory computer-readable storage medium of claim 26, wherein the selected one or more bounding boxes are described by parameters describing the position of the bounding boxes in the received image and parameters describing the layout of the bounding boxes in the received image.

31. The non-transitory computer-readable storage medium of claim 26, further comprising: It is determined that the received image is a video frame associated with a video content item; Obtain the video frame set of the video content item, wherein the video frame set includes video frames displayed after the received image is displayed; Each video frame in the acquired video frame set is processed using a group of one or more image processing techniques to automatically generate one or more suggestion spaces for media content item coverage in each video frame; Apply one or more heuristic rules to automatically select one or more bounding boxes from the one or more suggestion spaces in each video frame for media content item coverage; and Display each video frame image and overlay one or more selected media content items within the corresponding one or more bounding boxes in each video frame.

32. The non-transitory computer-readable storage medium of claim 31, wherein applying one or more heuristic rules to automatically select one or more bounding boxes from the one or more suggestion spaces in each video frame for media content item coverage further comprises: Ensure that when displaying the video content item, there is a smooth transition of overlaid media content in each of the video frame images displayed after the display of the received image.

33. A computer system, comprising: Computer processor; and A non-transitory computer-readable storage medium for storing instructions, which, when executed by a computer processor, cause the computer processor to perform steps for detecting a space suitable for overlaying media content on an image, the steps including: Receive the image used for media content item overlay; The received image is processed using a group of one or more image processing techniques to automatically generate one or more suggestion spaces in the image for media content item coverage; One or more heuristic rules are applied to automatically select one or more bounding boxes from the one or more suggestion spaces in the image for media content item coverage, wherein applying the one or more heuristic rules includes generating bounding boxes based on grouping of adjacent columns, the grouping being in a pyramid style, the grouping being determined by multiple groups of adjacent columns superimposed on the image, the determined grouping satisfying a predefined minimum threshold of overlap with the suggestion spaces in the one or more suggestion spaces; Select one or more media content items to overlay the corresponding one or more bounding boxes selected in the image; and Display the image and overlay one or more selected media content items within the corresponding one or more bounding boxes in the image.

34. The computer system of claim 33, wherein processing the received image using a group of one or more image processing techniques to automatically generate one or more suggestion spaces for media content item coverage further comprises: Generate a binary matrix representation of the received image; Determine one or more rectangular spaces within the generated binary matrix that satisfy a predefined minimum threshold parameter; Cover text and faces within one or more rectangular spaces defined in the generated binary matrix; Generate a sorted list of the rectangular space, wherein the sorting is performed based on predefined parameters; and The rectangular space in the generated sorted list is provided as one or more suggested spaces in the received image for media content item coverage.

35. The computer system of claim 33, wherein applying one or more heuristic rules to automatically select one or more bounding boxes from the one or more suggestion spaces in the image for media content item coverage includes one or more of the following: A bounding box is generated by modifying one or more proposed spaces to lie on a grid superimposed on the image; Based on determining the grouping of adjacent columns from multiple groups of adjacent columns superimposed on the image, such that the determined grouping of adjacent columns satisfies a predefined minimum threshold of overlap with the proposal space of the one or more proposal spaces, a bounding box is generated. Generate a bounding box, which is a weighted combination of the one or more proposed spaces; and The bounding box is generated using one or more suggestion spaces based on predefined target attributes.

36. The computer system of claim 35, wherein the predefined target attribute includes: Number of bounding boxes; The area of ​​the bounding box; The shape of the bounding box; The aspect ratio of the bounding box; and The position of the bounding box.

37. The computer system of claim 33, wherein the selected one or more bounding boxes are described by parameters describing the position of the bounding boxes in the received image and parameters describing the layout of the bounding boxes in the received image.

38. The computer system according to claim 33, further comprising: It is determined that the received image is a video frame associated with a video content item; Obtain the video frame set of the video content item, wherein the video frame set includes video frames displayed after the received image is displayed; Each video frame in the acquired video frame set is processed using a group of one or more image processing techniques to automatically generate one or more suggestion spaces for media content item coverage in each video frame; Apply one or more heuristic rules to automatically select one or more bounding boxes from the one or more suggestion spaces in each video frame for media content item coverage; and Display each video frame image and overlay one or more selected media content items within the corresponding one or more bounding boxes in each video frame.

39. The computer system of claim 38, wherein applying one or more heuristic rules to automatically select one or more bounding boxes from the one or more suggestion spaces in each video frame for media content item coverage further comprises: Ensure that when displaying the video content item, there is a smooth transition of overlaid media content in each of the video frame images displayed after the display of the received image.

Images