Using weakly supervised models to detect objects

Abstract

The present disclosure relates to detecting objects using weakly supervised models. The present disclosure relates to systems and methods for detecting objects in an input image based on target object keywords. For example, one or more embodiments described herein generate a heat map of the input image based on the target object keywords and generate various bounding boxes based on pixel analysis of the heat map. One or more embodiments described herein then use the various bounding boxes to determine scores for the generated object location proposals to provide the highest-scoring object location proposals superimposed on the input image.

Description

Technical Field

[0001] The present disclosure relates to the field of digital image processing. Background Art

[0002] Recent years have seen rapid technological advancements in automatic object detection in digital images. Indeed, due to the proliferation of personal computing devices and digital cameras, individuals and businesses now routinely manage large repositories of digital images and videos. Consequently, automatic object detection in digital images has become a common requirement for individuals and businesses in a variety of scenarios, ranging from casual users seeking to locate specific moments in their personal photo collections to professional graphic designers categorizing stock images to enhance creative projects.

[0003] Unfortunately, conventional object detection systems have several shortcomings. For example, most accurate conventional object detection systems involve a form of machine learning. These conventional machine learning methods require supervised learning (human-annotated training data) for training, or they do not provide useful predictions. Generating annotated images for training is time-consuming and expensive.

[0004] Because of the need for supervised data, conventional object detection systems are only able to identify a small number of types of objects. In fact, conventional object detection systems are typically only able to identify 20 types of objects with reasonable accuracy. Some conventional object detection systems can identify as many as 80 or even 120 different types of objects, but at unacceptable levels of accuracy.

[0005] Therefore, there is a need for robust, efficient, and effective detection of objects in large datasets of digital images. Summary of the Invention

[0006] One or more embodiments provide benefits and / or address one or more of the aforementioned or other problems in the art using systems, computer-readable media, and methods that employ a weakly supervised learning model that allows for automatic detection of objects in digital images in a manner that detects a large number of categories in a large data set. Specifically, in one or more embodiments, the disclosed system performs object detection associated with a digital image by generating a heat map of the digital image based on target object keywords. Furthermore, the disclosed system then generates one or more bounding boxes based on pixel analysis associated with the heat map. In one or more embodiments, the disclosed system then scores various object location proposals based on the interaction of each object location proposal with the one or more generated bounding boxes. Finally, the disclosed system outputs a bounding box superimposed on the location or orientation in the digital image corresponding to the object location proposal with the highest score. In this manner, the disclosed system and method quickly and accurately detect objects in digital images without having to train the disclosed system and method using supervised learning.

[0007] Additional features and advantages of one or more embodiments of the present disclosure will be set forth in the description which follows and, in part, will be apparent from the description, or may be learned by practicing such example embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The detailed description is described with reference to the accompanying drawings, in which:

[0009] Figure 1A to Figure 1B illustrates a graphical user interface illustrating object detection functionality of an object detection system according to one or more embodiments;

[0010] Figure 2 illustrates a sequential diagram of steps for detecting an object in an input image according to one or more embodiments;

[0011] Figure 3 illustrates an overview of generating a heat map according to one or more embodiments;

[0012] Figures 4A to 4B illustrates an example bounding box according to one or more embodiments;

[0013] Figures 5A to 5B illustrates an example center frame according to one or more embodiments;

[0014] Figure 6illustrates an overview of a process for detecting objects in an input image using a heat map, a bounding box, and one or more center boxes, according to one or more embodiments;

[0015] Figure 7 illustrates a schematic diagram of an object detection system including an object detection system according to one or more embodiments;

[0016] Figure 8 illustrates an example environment in which an object detection system operates according to one or more embodiments;

[0017] Figure 9 A flowchart illustrating steps for detecting objects in an input image according to one or more embodiments; and

[0018] Figure 10 Illustrated is a block diagram of an exemplary computing device in accordance with one or more embodiments. DETAILED DESCRIPTION

[0019] The present application discloses various embodiments of an object detection system that uses a weakly supervised learning model to perform object detection. According to one or more embodiments, the object detection system detects objects in an input image based on a received target object keyword. Specifically, the object detection system first generates a heat map of the input image based on the target object keyword. The object detection system then generates various bounding boxes based on pixel analysis of the heat map and scores object location proposals based on the interaction between each proposal and the generated bounding box. The object detection system outputs an indication of the object location proposal with the highest score superimposed on the input image to show the location of the detected object associated with the target object keyword in the input image.

[0020] To further illustrate the features and functions of the object detection system, in one or more embodiments, the object detection system receives a target object keyword and an input image (or set of image classes). In response to receiving the input image and the target object keyword, the object detection system generates a heat map associated with the input image and the target object keyword. As used herein, a "heat map" refers to a graphical representation of data that is transmitted based on an image-to-topic embedding method. Specifically, a heat map may include a mapping of an image indicating that the image may include a portion of an object corresponding to the target object keyword. For example, the object detection system may generate a heat map based on the input image and the target object keyword using an embedding network, such that the color in each region (e.g., each pixel) of the heat map conveys the likelihood that a given pixel is part of an object corresponding to the target object keyword.

[0021] For example, in one or more embodiments, the object detection system utilizes a fully convolutional dense labeling network that receives an input image and a target object keyword and outputs a heat map. In one or more embodiments, the fully convolutional dense labeling network (e.g., as described in more detail below) performs an analysis of color, texture, line, etc. in each region or pixel of the input image to determine the likelihood that the region is associated with the target object keyword. In at least one embodiment, the fully convolutional dense labeling network places brighter pixels in regions of the heat map that correspond to likely locations of the target object keyword in the input image.

[0022] The resulting color-coded heatmap indicates the general location in the input image that corresponds to the target object keyword. However, this general location may lack sufficient specificity to allow the object detection system to clearly indicate the location or orientation of the target object. Therefore, in response to generating a heatmap of the input image based on the target object keyword, the object detection system generates a more precise location of the object corresponding to the target object keyword based on the heatmap using bounding boxes and object proposals.

[0023] For example, the object detection system first generates a bounding box based on the heat map. In one or more embodiments, the object detection system generates a bounding box by identifying pixels in the heat map using pixel values greater than a global threshold. After identifying pixels in the heat map using pixel values greater than the global threshold, the object detection system performs various transformations on the identified pixels to generate a fully connected region or shape within the heat map. In at least one embodiment, the object detection system then adapts a rectangle (e.g., a bounding box) to the shape or region to generate a bounding box.

[0024] The object detection system then generates one or more center boxes based on the heat map. In one or more embodiments, the object detection system generates the one or more center boxes by identifying the pixel with the maximum pixel value in each row and column in the heat map. The object detection system then performs various transformations on each region of identified pixels to generate one or more fully connected shapes within the heat map. Finally, the object detection system fits a bounding box to each fully connected shape in the heat map to generate the one or more center boxes.

[0025] After generating the bounding box and one or more center boxes, the object detection system generates one or more object location proposals. As used herein, an "object location proposal" refers to a predicted location or region in the input image that is associated with a target object keyword. For example, in one or more embodiments, the object detection system utilizes various methods to generate one or more object location proposals that include coordinates summarizing a predicted location or region in the input image that includes the object specified by the target object keyword. In at least one embodiment, and as will be further described below, the object detection system utilizes a selective search method (such as the selective search method described in J. Uijlings, K. van de Sande, T. Gevers, and A. Smeulders. Selective Search For Object Recognition, IJCV, 2013, the entire contents of which are incorporated herein by reference) and a YOLO (“You Only Look Once”) network method (such as the YOLO network method described in Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi: You Only Look Once: Unified, Real-Time Object Detection arXiv:1506.02640, May 6, 2016, the entire contents of which are incorporated herein by reference) to generate one or more object location proposals.

[0026] To identify the best object location proposal, the object detection system determines a score for each of the one or more object location proposals based on the generated bounding box and the one or more center boxes. For example, in at least one embodiment, the object detection system determines the score for the object location proposal based on the amount of overlap between the object location proposal and the bounding box and the amount of overlap between the object location proposal and each of the one or more center boxes. After determining the score for each of the one or more object location proposals, the object detection system filters out poor proposals with scores below a predetermined threshold.

[0027] In response to identifying at least one object location proposal with a high score, the object detection system overlays an indication of the object location proposal on the input image at the coordinates specified by the object location proposal. The object detection system then provides the input image with the overlaid indication. For example, the object detection system may provide the input image with the overlaid indication for display on a client device. Alternatively, the object detection system may add the input image with the overlaid indication to a collection of other input images that the object detection system has determined to include an object corresponding to the target object keyword (e.g., as part of a batch process of a large data set including input images).

[0028] Thus, object detection systems offer a computer-based solution to existing problems in automatic object detection. For example, rather than relying on expensive, human-annotated training data, object detection systems utilize weakly supervised learning models. Consequently, training is much faster than with conventional systems because time-consuming image annotation is not required.

[0029] Furthermore, object detection systems are more robust than conventional systems. In fact, instead of being able to identify objects in 20 to 100 object categories, object detection systems can detect all object categories in a dataset (e.g., 40,000 or more categories). Therefore, object detection systems can be used in conjunction with large-scale industry datasets.

[0030] Figure 1A and Figure 1B An example graphical user interface displaying one or more features of an object detection system is illustrated. For example, as shown in FIG1 , the object detection system can provide an object detection graphical user interface 106 via a client computing device 102. In one or more embodiments, when the object detection system is a server-hosted application, the object detection graphical user interface 106 can be provided via a web browser 104. Specifically, a user can identify, upload, or select an input image 108 or a collection of images. In response, the object detection system displays the input image 108 in the object detection graphical user interface 106.

[0031] In addition, the object detection system provides a keyword input box 110 and recommended tags 112 in the object detection graphical user interface 106. In one or more embodiments, the object detection system receives a target object keyword via the keyword input box 110. In response to receiving the target object keyword via the keyword input box 110, the object detection system provides an indication of an area in the input image 108 that corresponds to the target object keyword.

[0032] For example, as in Figure 1BAs shown in FIG, in response to receiving the target object keyword "koala" via the keyword input box 110, the object detection system provides an indication 116 associated with the object location in the input image 108, the object location being associated with the received target object keyword. Figure 1B As shown in , the indication 116 is a bounding box superimposed on the input image 108 such that the detected object (eg, koala) is within the coordinates of the corner points of the bounding box.

[0033] Although the single input image displayed via the object detection graphical user interface 106 is Figure 1A and Figure 1B The object detection features of the object detection system are described in

[0014] , but the object detection system can perform the same object detection as part of a batch process. For example, in at least one embodiment, the object detection system receives a batch command to process a large dataset of input images associated with a target object keyword. In response to receiving the batch command, the object detection system detects an object associated with the target object keyword in each of the input images in the dataset. However, in at least one embodiment, instead of outputting a graphical display of an indication associated with the location of the object in each input image, the object detection system stores coordinates associated with each indication determined for the corresponding input image.

[0034] Figure 2 The diagram shows an overview of a series of actions by which an object detection system detects an object in a digital image based on received target object keywords. Figure 2 After providing an overview, combined with Figures 3 to 6 Provides additional details about the process of object detection performed by the object detection system. Figure 2 As shown in FIG, when the object detection system receives an input image and a target object keyword (202), a series of actions begins. Figure 1A and Figure 1B As described, the object detection system can receive an input image and a target object keyword via the object detection graphical user interface 106. Additionally or alternatively, the object detection system can receive the input image and the target object keyword via a native application interface (e.g., a mobile device app, a desktop program). Additionally or alternatively, the object detection system can receive the input image and the target object keyword as part of an automatic batch process.

[0035] In response to receiving an input image and a target object keyword, the object detection system generates a heat map (206) of the input image based on the target object keyword. For example, the heat map is a graphical representation of the dimensions of the shared input image, the graphical representation indicating one or more regions in the input image that are associated with the target object keyword. In one or more embodiments, the object detection system generates the heat map (206) of the input image by using a fully convolutional dense labeling network. The object detection system provides the input image and the target object keyword as input to the fully convolutional dense labeling network.

[0036] The fully convolutional dense labeling network outputs a generated heat map corresponding to the input image. For example, the fully convolutional dense labeling network outputs a heat map such that the color intensity of each pixel or region in the heat map corresponds to the target object keyword. For example, if the fully convolutional dense labeling network determines that a particular region in the input image is associated with the target object keyword (i.e., the object corresponding to the target object keyword is likely to be located in that region in the input image), the fully convolutional dense labeling network adds brighter pixels to the corresponding region in the heat map. Figure 3 Let’s further discuss the process by which an object detection system generates heatmaps.

[0037] After generating a heat map (206) of the input image, the object detection system generates various bounding boxes based on the generated heat map. In one or more embodiments, the object detection system first generates bounding boxes (208) based on the generated heat map. In at least one embodiment, the object detection system generates bounding boxes (208) by determining a global threshold in the heat map and identifying all pixels in the heat map with pixel values greater than the global threshold.

[0038] The object detection system will then Figure 2 The object detection system then binarizes the image into pixels above a threshold and pixels below a threshold (e.g., converting the heatmap to black and white). The object detection system then identifies the connected domains in the binarized heatmap. Finally, the object detection system computes the morphology convex of the connected domains and fits the minimum bounding rectangle to the computed morphology convex. In one or more embodiments, the minimum bounding rectangle in the binarized heatmap is a bounding box. Figure 4A and Figure 4B The process by which the object detection system generates the bounding box (208) is further described.

[0039] After generating the bounding box (208), the object detection system generates one or more center boxes (210) based on the generated heat map. For example, the object detection system generates a bounding box in step (208) that identifies a region of pixels in the generated heat map where all pixels are above the global threshold. In one or more embodiments, the object detection system generates one or more center boxes (210) to identify one or more regions of pixels in the generated heat map where pixels are above a local threshold.

[0040] To generate one or more center boxes (210), the object detection system identifies local maxima in each row and column of the heat map. For example, the object detection system identifies pixels in each row and column of the heat map that have a color value that is higher (e.g., brighter) than the color values of all other pixels in that row or column. In one or more embodiments, the object detection system then binarizes the heat map based on the identified pixels and identifies the connected domains of each group of white pixels. Finally, the object detection system calculates the morphological convexity of each connected domain and adapts a minimum bounding rectangle to each calculated morphological convexity. In one or more embodiments, the resulting minimum bounding rectangle is one or more center boxes. The process by which the object detection system generates one or more center boxes (210) is further described below with reference to Figure 5.

[0041] In one or more embodiments, the object detection system performs additional processes in parallel or concurrently with the processes described in steps (206) to (210). For example, Figure 2 As shown in FIG, the object detection system generates object location proposals (204) based on the input image in parallel with generating heat maps (206), generating bounding boxes (208), and generating center boxes (210). In at least one embodiment, the object detection system generates object location proposals (204) by combining one or more object detection methods using the input image and target object keywords. Figure 2 Step (204) is illustrated as being concurrent with steps (206) to (210), but in alternative embodiments, the object detection system may perform step (204) before performing steps (206) to (210) or after performing steps (206) to (210).

[0042] In one or more embodiments, the object detection system utilizes at least two different methods to generate object location proposals (204). For example, in one embodiment, the object detection system utilizes a selective search method and a YOLO (“You Only Have to Look Once at the Image”) method to generate object location proposals (204). As an overview, the object detection system proposes regions, locations, or orientations in an input image that may be associated with a target object keyword. The object detection system begins by over-segmenting the input image based on the color intensity of pixels in the input image. The object detection system then iterates the following steps: 1) adding all bounding boxes corresponding to the segmented portions of the input image to a list of object location proposals, 2) grouping adjacent segments based on similarity. The object detection system iterates steps 1 and 2 until the entire input image is included in a single bounding box. The object detection system measures the similarity between adjacent segments based on color, texture, size, and shape compatibility. The selective search method used by the object detection system typically results in a large number of object location proposals.

[0043] As just mentioned, the object detection system can also utilize the YOLO method to generate object location proposals. As an overview, the YOLO ("You Only Look Once at an Image") method applies a single neural network to an input image. Specifically, using the YOLO neural network, the object detection system divides the input image into regions and predicts bounding boxes within the input image and the probability associated with each region. The object detection system weights each predicted bounding box based on the probability that the object associated with the target object keyword is located in the same region as the predicted bounding box. The object detection system outputs the predicted bounding box with the heaviest weight as the object location proposal.

[0044] In one or more embodiments, the object detection system may use additional object detection methods to generate object location proposals (204). Alternatively, the object detection system may only utilize a single object detection method to generate object location proposals (204). In at least one embodiment, the object detection system adds additional weight to object location proposals generated by a particular method. For example, the object detection system may add additional weight to object location proposals generated by the YOLO method while not weighting object location proposals generated by the selective search method.

[0045] Reference Figure 2 At the end of steps (204) and (210), the object detection system scores the object proposal based on the bounding box and the center box (212). In one or more embodiments, the object detection system scores the object proposal primarily based on the amount of overlap between the object proposal and the bounding box, and the amount of overlap between the object proposal and at least one center box.

[0046] After determining the score for each object location proposal, the object detection system filters out erroneous object location proposals (214). In one or more embodiments, the object detection system filters out erroneous object location proposals (214) based on the scores determined in step (212). For example, in at least one embodiment, the object detection system identifies the maximum score among the scores determined for the object location proposals and filters out object location proposals with scores that are less than a threshold amount (e.g., 70%) of the maximum score. In addition, the object detection system also filters out the remaining object location proposals by filtering out object location proposals that are at the same location (e.g., object location proposals with an overlap of >= 0.8). The object detection system can also further filter the remaining object location proposals by identifying any remaining object location proposals that are not well covered by the bounding box (e.g., overlap <= 0.8) and by identifying any remaining object location proposals that fail to well cover at least one center box.

[0047] In at least one embodiment, the filtering process in step (214) causes the object detection system to filter out all object location proposals from the generated object location proposals. If no object location proposals remain after the filtering process, the object detection system can rescale at least one of the one or more center boxes (216) and provide the rescaled at least one center box as an indication of the object location associated with the target object keyword. If all generated object location proposals are filtered out, the object detection system rescales the at least one center box to enlarge the at least one center box. For example, the object detection system enlarges the largest center box of the one or more center boxes by recalculating the corner coordinates of the largest center box so that the resulting rescaled center box is larger by a threshold percentage (e.g., 50% larger).

[0048] After determining the highest-scoring object location proposal or at least one rescaled center box, the object detection system provides an input image superimposed with an indication of the object location (218). For example, in response to determining the highest-scoring object location proposal or the rescaled center box, the object detection system superimposes a bounding box on the input image having corners at the same coordinates as the corners of the highest-scoring object location proposal or the rescaled center box. In one or more embodiments, the object detection system adds a color or line pattern to the superimposed bounding box to make the superimposed bounding box easier to see with respect to the input image.

[0049] The object detection system may provide the input image (218) with the indication of the object location superimposed thereon to the client device (e.g., via a webpage, via a native application display). Alternatively, the object detection system may provide the input image (218) with the indication of the object location superimposed thereon to a repository of other input images with the indication of the object location superimposed thereon as part of a batch process involving a large number of additional input images. Alternatively, the object detection system may construct a database including unique identifiers associated with the input images and corner coordinates of the indication of the object location as part of a batch process involving a large number of additional input images.

[0050] Figure 3 The process by which an object detection system generates a heat map based on an input image and target object keywords is illustrated. For example, as mentioned above and as Figure 3 As shown in , the object detection system can generate a heat map 310a by first transforming the embedding neural network 302 into a fully convolutional dense labeling neural network 304.

[0051] As an initial step, the object detection system uses a training image set to train the embedding neural network 302. The training image set includes images and associated labels. The object detection system can use word vector representations to exploit label correlations. The embedding neural network 302 is learned to map images to an embedding space, such as e=embed_net(I), where With dimension D. The object detection system then measures the similarity score between the image I and the word vector representation of the label w in this space by the following equation:

[0052]

[0053] Object detection systems use pointwise mutual information wordvectors as label representations. Pointwise mutual information is a measure of association given by:

[0054] The size of the matrix M is V×V, where V is the size of the label dictionary W, and M takes into account the co-occurrence of labels in the training corpus. The object detection system applies eigenvector decomposition to decompose the matrix M into M=USU T . In W=US -1 / 2 In the case of , each row of the column-truncated submatrix is then used as the word vector representation for all V labels. In practice, object detection systems set D to 4096.

[0055] After obtaining the word vector representation for each label, the object detection system uses the encoding scheme on the image label provided by the user. The object detection system forms a soft topic t as a weighted average of the bag of word vectors:

[0056]

[0057] This encoding scheme is a "soft topic" because t is a rough summary of the label list and resides in a continuous word vector space. Object detection systems can use soft topics to discover the dominant concepts in the label list, making them robust to "noisy" annotations. In addition, while each image in the dataset is labeled with many labels, the list labels are annotated relative to the entire image, and there are no bounding box annotations available for each label. Therefore, label-specific embedding methods are not directly implemented.

[0058] A neural network is used to map the image and label list into a common embedding space ε. More specifically, the object detection system subjects each image I to a Resnet. The object detection system extracts the penultimate layer and uses it as the visual feature v. To train the embedding neural network 302, the object detection system uses a cosine similarity loss to compare the visual embedding e and the soft topic t. For example, the object detection system may use the following cosine similarity loss:

[0059]

[0060] Once the embedding neural network 302 is trained, it maps the image and the associated label list into a common embedding space. Accordingly, the embedding neural network 302 is a three-layer fully connected neural network, each fully connected layer is followed by a batch normalization layer and a rectified linear function ("ReLU") nonlinear activation layer.

[0061] To transform the embedding neural network 302 into the fully convolutional densely labeled neural network 304, the object detection system utilizes the fully connected weights in the embedding neural network 302 as 1x1 convolution kernels in the fully convolutional densely labeled neural network 304. The object detection system also utilizes the batch normalization layer in the embedding neural network 302 as a spatial batch normalization layer in the fully convolutional densely labeled neural network 304. To further illustrate, in one or more embodiments, a fully connected layer, such as in the embedding neural network 302, comprises a linear operation in which each input of the fully connected layer is connected to each output from the fully connected layer via a weight. In one or more embodiments, a convolutional layer, such as in the fully convolutional densely labeled neural network 304, utilizes a subset of the weights of a dense layer, where nearby inputs are connected to nearby outputs, and the weights of the convolution at each location are shared.

[0062] After this transformation, as in Figure 3 As shown in FIG, in response to receiving an input image 306 and a target object keyword 308, the fully convolutional dense labeling neural network 304 outputs a heat map 310a.

[0063] In one or more embodiments, the heat map 310a comprises the same dimensions as the input image 306. In addition, the heat map 310a comprises a matrix of pixels in which each pixel is colored to represent specific data. In at least one embodiment, the data represented by the color of the pixel in the heat map 310a is the likelihood of whether the area in the input image corresponding to the location of the pixel in the heat map is associated with the target object keyword 308. Figure 3 As shown in FIG, the fully convolutional dense labeling neural network 304 colors the pixels in the heat map 310a so that the color of the pixel increases in intensity (e.g., becomes brighter) as the likelihood increases. As a result, the brighter areas in the heat map 310a are more likely to correspond to areas in the input image 306 that are associated with the target object keyword 308.

[0064] In one or more embodiments, the above Figure 3 The discussed acts and algorithms may include corresponding structure for performing the steps of generating a heat map of an input image based on target object keywords.

[0065] Figure 4A The diagram illustrates how the object detection system generates a bounding box based on the heat map 310a. In one or more embodiments, the object detection system first determines a global threshold based on the entire heat map 310a. In at least one embodiment, the object detection system uses a triangulation method to determine the global threshold. For example, the object detection system generates a histogram representing the brightness levels of the pixels in the heat map 310a, with the number of pixels along the y-axis and the brightness (e.g., RGB values) along the x-axis. The object detection system then adds a line to the histogram between the maximum value at the brightest point of the histogram and the minimum value at the darkest point of the histogram. After this, for all brightness values along the x-axis, the object detection system identifies the distance between the added line and the histogram. The object detection system identifies the global threshold as the brightness value on the x-axis at which the distance between the histogram and the line is the largest. In additional or alternative embodiments, the object detection system can use other methods to identify the global threshold (e.g., an isodata algorithm, a background symmetry algorithm). Alternatively, the object detection system can use a predefined static global threshold.

[0066] In one or more embodiments, the global threshold value represents a specific pixel color value. For example, as described above, the pixels in the heat map 310 each include RGB color values (e.g., "red, green, blue" color values), where the color absolute black is represented as "(0,0,0)", the color absolute white is represented as "(255,255,255)", and all other colors are represented by RGB values between absolute black and absolute white. Therefore, the global threshold value represents a color with a specific brightness level between absolute black and absolute white.

[0067] To identify all pixels in the heat map that have pixel color values greater than the identified global threshold, the object detection system creates a copy 310a' of the heat map and compares the color value of each pixel in the heat map copy 310a' to the identified global threshold. If the color value of the pixel is at least as bright as the global threshold (e.g., the RGB value of the pixel is at least as large as the RGB value of the global threshold), the object detection system converts the color value of the pixel in the heat map copy 310a' to absolute white (e.g., "(255, 255, 255)"). If the color value of the pixel is darker than the global threshold (e.g., the RGB value of the pixel is less than the RGB value of the global threshold), the object detection system converts the color value of the pixel in the heat map copy 310a' to absolute black (e.g., "(0, 0, 0)"). The object detection system continues this process until the copy 310a' of the heat map is binarized to black and white and each pixel is converted to absolute white or absolute black. As in Figure 4A As shown in FIG, the resulting binarized image 312a in the heat map copy 310a' is black and white, and the white areas of the binarized image 312a represent areas of the heat map 310a that include pixels having color values equal to or higher than the global threshold value.

[0068] In one or more embodiments, the process of creating the binarized image 312a does not result in connected domains. For example, a connected domain is a closed curve that can be continuously contracted into a point in a set. In other words, a connected domain is a closed shape that does not include any holes. In at least one embodiment, and in response to determining that a white area of the binarized image 312a is not a connected domain, the object detection system may convert the white area into a connected domain. For example, the object detection system may convert black pixels in the binarized image 312a into white pixels to close the curve along the white area. Additionally, the object detection system may convert isolated black pixels within the white area of the binarized image 312a to close any existing holes within the binarized image 312a.

[0069] In addition to transforming the binarized image 312a into a connected domain, the object detection system can also perform additional morphological functions in conjunction with the binarized image 312a to generate an optimal bounding box. For example, the object detection system can calculate the morphological convexity of the connected domain. In one or more embodiments, the convexity of a shape (e.g., the white area of the binarized image 312a) refers to: for each pair of points in the shape, how well each point on the straight line segment connecting the point pairs remains within the shape. In other words, a shape with poor convexity does not have good roundness. In at least one embodiment, the object detection system can calculate the morphological convexity of the connected domain to include a peripheral shape with better roundness (e.g., a convex "hull") by transforming the area of the white area of the binarized image 312a.

[0070] After calculating the morphological convexity of the connected domain in the binarized image 312a, the object detection system generates a bounding box. For example, in one or more embodiments, the object detection system generates the bounding box 314a by fitting a minimum bounding rectangle to the transformed white area (e.g., the calculated morphological convexity) in the binarized image 312a. In at least one embodiment, the object detection system stores the generated bounding box 314a as a set of corner coordinates (e.g., "(x, y, width, height)") relative to the heat map copy 310a'.

[0071] Figure 4A The object detection system is shown generating a bounding box 314a in conjunction with a relatively basic heat map (e.g., heat map 310a). Figure 4A As shown in , heat map 310a includes a single "hotspot" that indicates a single location in the corresponding input image that is associated with the target object keyword. In additional embodiments, the object detection system can generate a bounding box in conjunction with a more complex heat map.

[0072] For example, Figure 4B The object detection system is shown generating a bounding box 314b based on the heat map 310b. Figure 4B , heat map 310b includes a plurality of "hot spots" indicating a plurality of locations in the corresponding input image that are associated with the target object keyword. For illustration, the corresponding input image may be a photo of a plurality of family members sitting and standing loosely in a group, and the target object keyword may be "person."

[0073] exist Figure 4B In the embodiment illustrated in FIG, the object detection system can be implemented by utilizing the above relative Figure 4AThe global threshold value of the heat map 310b can be determined by any of the methods described in the preceding claims. Similarly, the object detection system can binarize the heat map copy 310b' by comparing the color value of each pixel in the heat map copy 310b' to the identified global threshold value. Due to the multiple "hot spots" in the heat map 310b, the resulting binarized image 312b may include non-connected regions. Therefore, as described above and as shown in Figure 4B As shown in , the object detection system can transform the white areas of heat map copy 310b' into connected domains. Furthermore, the object detection system can calculate the morphological protrusions of the connected domains in binarized image 312b to achieve a circular perimeter shape. Finally, as described above, the object detection system fits the minimum bounding box to the white areas of binarized image 312b to generate bounding box 314b.

[0074] Figure 5A The process by which the object detection system generates one or more center boxes based on the heat map 310A is illustrated. In one or more embodiments, the object detection system begins by determining a local threshold within the second heat map copy 310a". For example, as in Figure 5A As shown in , the object detection system determines the local threshold by identifying the brightest pixel (e.g., the pixel with the brightest RGB color value) in each row and each column of the second heat map copy 310a". In response to identifying the brightest pixel in each row and each column of the second heat map copy 310a", the object detection system converts the pixel to absolute white and converts the remaining pixels in the row or column to absolute black.

[0075] As in Figure 5A As shown in FIG, the resulting binarized image 312a includes white areas and black areas. In one or more embodiments, the local threshold is more accurate than the global threshold discussed above. Therefore, the white areas generated in response to the local threshold are generally smaller than the white areas generated in response to the global threshold.

[0076] After generating the binarized image 312a from the second heat map copy 310a", as described above with reference to Figures 4A to 4B As described above, the object detection system transforms each white area in the binary image 312a into a connected domain. Figures 4A to 4B In the same process described above, the object detection system calculates the morphological protrusion of each connected domain in the binarized image 312a. Finally, the object detection system fits a minimum bounding rectangle to each transformed white region in the second heat map copy 310a" to generate a center box 316a.

[0077] As mentioned above, the object detection system can generate two or more center boxes based on the heat map. For example, as discussed above with reference to heat map 310b, the input image can include several objects corresponding to the target object keyword. Therefore, the heat map associated with the input image can include several "hot spots" or areas associated with the target object keyword.

[0078] Figure 5B Two or more center boxes are illustrated. For example, the object detection system determines a local threshold by identifying the brightest pixel in each row and each column in the second heat map copy 310b". In response to identifying the brightest pixel in each row and each column of the second heat map copy 310b", the object detection system converts the pixel to absolute white and converts the remaining pixels in the row or column to absolute black. After generating a binarized image 312b from the second heat map copy 310b", the object detection system transforms each white area in the binarized image 312b into a connected domain, and then calculates the morphological protrusion of each connected domain in the binarized image 312b. Finally, the object detection system fits a minimum bounding rectangle to each transformed white area in the second heat map copy 310b" to generate center boxes 316b, 310c and 310d.

[0079] After generating the bounding box and one or more center boxes associated with the heatmap, as shown above Figures 4A to 5B As described, the object detection system uses the generated bounding boxes and center boxes to score and rank object location proposals. Figure 6 The object detection system is shown scoring object location proposals based on the heat map 310a (e.g., by Figure 4A and Figure 5A ). Figure 6 As shown in FIG, the object detection system scores and selects an object location proposal based on the bounding box 314a, the heat map 310a, and the center box 316a. As discussed above, the digital media management scores the object location proposal primarily based on the amount of overlap between the object location proposal and each of the bounding box and one or more center boxes.

[0080] For example, in at least one embodiment, the object detection system determines scores for object location proposals based on the following proposal scoring algorithm:

[0081] s p =w a *s a +w b *s b +w c *s c

[0082] Where: sa is the average pixel color value in the object location proposal (e.g., the average pixel color value indicates the relevance of the object location proposal); s b is a score based on the amount of overlap between the object location proposal and the generated bounding box; c is the correlation between an object location proposal and at least one center box; and w a 、w b and w c are assigned to s by the object detection system. a 、s b and s c The weight of .

[0083] For example, after receiving object location proposals 318a, 318b, 318c (e.g., via the YOLO network or selective search described above), the object detection system can iteratively score each object location proposal 318a to 318c by utilizing a proposal scoring algorithm. For example, in order for the object detection system to calculate the proposal scoring algorithm for object location proposal 318c, the object detection system can determine the average pixel color value (e.g., s ) of object location proposal 318c. a ) and the amount of overlap between the object location proposal and the bounding box 314a (e.g., s b ).

[0084] In one or more embodiments, the object detection system determines s based on the number of overlapping pixels between the object location proposal 318c and the generated bounding box 314a. b (e.g., a score based on the amount of overlap between the object location proposal and the generated bounding box. In this embodiment, the score may be the number of overlapping pixels. Alternatively, the object detection system may calculate s by determining the percentage of the generated bounding box 314a that is superimposed with the object location proposal 318c. b .

[0085] To determine the relevance of the object location proposal 318c and at least one center box (e.g., s c ), the object detection system utilizes the following dependencies of the current proposed algorithm for center box 316a:

[0086] s c =w iou *s iou +w overlap(c,p) *s overlap(c,p) +w overlap(p,c) *s overlap(p,c) Where: s iou is a score based on the intersection-over-union between the center box 316a and the object location proposal 318c; overlap(c,p)is a score based on the amount of overlap between the center box 316a and the object location proposal 318c; overlap(p,c) is a score based on the amount of overlap between the object location proposal 318c and the center box 316a; and w iou 、w overlap(c,p) and w overlap(p,c) are assigned to s by the object detection system. iou 、s overlap(c,p) and s overlap(p,c) The weight of .

[0087] If there are two or more center boxes (such as above Figure 5B The object detection system can then determine the relevance of the object location proposal 318c to each of the two or more center boxes. The object detection system then applies a proposal scoring heuristic to each generated center box to obtain the highest score s for the object location proposal. p The object detection system then takes the highest score s p Assigned to the object location proposal. If there are multiple object location proposals (e.g., object location proposals 318a, 318b), the object detection system repeats the utilization of the proposal scoring heuristic with each additional object location proposal. In one or more embodiments, the algorithm discussed above may include corresponding structure for performing the following steps: determining a score for each of the one or more determined object location proposals in the input image.

[0088] After determining the score (e.g., s p ), the object detection system filters out erroneous object location proposals as discussed above. In one or more embodiments, the object detection system outputs one or more object location proposals (e.g., the highest-scoring object location proposals remaining after the filtering process or at least one rescaled center box) as a list of coordinates. For example, the object detection system may output each object location proposal in the following format:

[0089]

[0090] After determining the highest-scoring object location proposal or at least one rescaled center box, the object detection system provides an input image superimposed with an indication of the object location. For example, in response to determining the highest-scoring object location proposal or the rescaled center box, the object detection system superimposes the input image with a bounding box having corner points at the same coordinates as the corner coordinates of the highest-scoring object location proposal or the rescaled center box. In one or more embodiments, the object detection system adds a color or line pattern to the superimposed bounding box to make the superimposed bounding box easier to see with respect to the input image.

[0091] Specifically, refer to Figure 6 In response to identifying the object location proposal with the highest score (e.g., object location proposal 318c), the object detection system provides the input image 306 superimposed with an indication 318c' of the object location associated with the target object keyword. For example, as mentioned above, the heat map (e.g., heat map 310a) and the heat map copies (e.g., copies 310a', 310a") share the same dimensions as the input image 306. Therefore, any object location proposals, bounding boxes, and center boxes generated by the object detection system based on the input image or heat map are associated with corner coordinates that refer to the same area on the input image and heat map. Therefore, as shown in Figure 6 As shown in FIG, the indication 318′ of the object location associated with the target object keyword is a bounding box that shares coordinates with the highest-scoring object location proposal (e.g., object location proposal 318c). In additional or alternative embodiments, the object detection system can generate the indication 318c′ to include various colors or line patterns. Furthermore, in at least one embodiment, the object detection system can analyze the input image 306 to identify a contrasting color and can apply the contrasting color to the indication 318c′ to make the indication 318c′ stand out against the input image 306.

[0092] Now refer to Figure 7 , will provide additional details about the capabilities and components of an object detection system according to one or more embodiments. Specifically, Figure 7 A schematic diagram illustrating an example architecture of an object detection system 702 as part of a digital media management system 701 and implemented on a computing device 700. Object detection system 702 may represent one or more embodiments of the object detection systems previously described.

[0093] As shown, object detection system 702 is located on or implemented by computing device 700. In general, computing device 700 can represent various types of client devices. For example, in some embodiments, computing device 700 is a mobile device, such as a laptop, tablet, mobile phone, smart phone, etc. In other embodiments, computing device 700 is a non-mobile device, such as a desktop or server, or another type of computing device. Figure 10 Additional details regarding computing device 700 are discussed.

[0094] Generally, the digital media management system 701 facilitates the creation, modification, sharing, and / or deletion of digital images within electronic documents and / or system applications. For example, the digital media management system 701 provides various tools related to image creation and editing (e.g., photo editing) on the computing device 700. In addition, the digital media management system 701 can access additional local or remotely stored image repositories. For example, in one or more embodiments, the digital media management system 701 integrates with digital applications (such as CREATIVE “ADOBE” and “CREATIVE CLOUD” are registered trademarks or trademarks of Adobe Inc. in the United States and / or other countries.

[0095] As in Figure 7 As shown in , object detection system 702 includes a digital media manager 704 , a heatmap generator 706 , a bounding box manager 708 , an object location proposal generator 710 , a proposal selection manager 712 , and a data storage device 714 including digital media data 716 and object detection data 718 .

[0096] In one or more embodiments, the components of object detection system 702 may include software, hardware, or both. For example, the components of object detection system 702 may include computer instructions stored on a non-transitory computer-readable storage medium and executable by at least one processor of computing device 700. When the computer-executable instructions are executed by the at least one processor, the computer-executable instructions cause computing device 700 to perform the methods and processes described herein. Alternatively, the components of the object detection system may include hardware, such as a dedicated processing device for performing a specific function or group of functions. Furthermore, the components of the object detection system may include a combination of computer-executable instructions and hardware.

[0097] As in Figure 7As shown in FIG, object detection system 702 includes a digital media manager 704. In at least one embodiment, digital media manager 704 provides access to digital media items (e.g., digital photos, digital videos) via a cloud storage device, via a social media system, or via a local storage device. Further, digital media manager 704 enables a user of computing device 700 to view and edit digital media items. Furthermore, in at least one embodiment, digital media manager 704 enables a user of computing device 700 to participate in social media activities with respect to one or more digital media items (e.g., commenting on a post that includes a digital photo). Furthermore, in one or more embodiments, digital media manager 704 sends and receives digital media items (e.g., input images, target object keywords), information associated with the digital media items (e.g., editing information, social media activity information), and other information utilized by object detection system 702 (e.g., an indication of the location of an object in the input image).

[0098] Moreover, as in Figure 7 As shown in FIG, object detection system 702 includes heat map generator 706. In one or more embodiments, as described above, heat map generator 706 transforms the embedding neural network into a fully convolutional dense labeling neural network. Furthermore, as also described above, heat map generator 706 provides an input image and a target object keyword as input to the fully convolutional dense labeling neural network and receives a heat map as output based on the input image and the target object keyword.

[0099] In one or more embodiments, the heat map generator 706 includes various functions for handling exceptions. For example, in at least one embodiment, a fully convolutional dense labeling neural network is inaccurate when detecting objects associated with the human category (for example, because all humans share the same basic body shape, facial components). Therefore, in order to improve the accuracy of object detection associated with the target object keywords "man", "woman" and "child", etc., the heat map generator 706 can temporarily convert any of these target object keywords into the target object keyword "person". The heat map generator 706 then generates a heat map of the input image based on the target object keyword "person". As described below, the heat map generator 706 can also provide the target object keyword "person" to the object location proposal generator 710 so that the generated object location proposal is associated with "person".

[0100] As in Figure 7As further shown in , the object detection system 702 includes a bounding box manager 708. In one or more embodiments, the bounding box manager 708 generates a bounding box based on the heat map generated by the heat map generator 706. For example, as discussed above, the bounding box manager 708 generates a bounding box based on the heat map by determining a global threshold for the heat map, binarizing the heat map based on the global threshold, obtaining a connected domain of the binarized heat map, calculating a morphological convexity of the connected domain, and fitting a minimum bounding rectangle to the morphological convexity. In at least one embodiment, the bounding box manager 708 stores the resulting bounding box as a set of coordinates relative to the heat map and the corresponding input image.

[0101] In addition, the bounding box manager 708 generates one or more center boxes based on the heat map generated by the heat map generator 706. For example, as discussed above, the bounding box manager 708 generates the one or more center boxes by identifying the local maximum pixel color value in each row and column in the heat map, binarizing the heat map based on the identified local maximum pixel color value, obtaining a connected domain of the binarized image, calculating a morphological convexity of the connected domain, and then fitting a minimum bounding rectangle to each calculated morphological convexity. In at least one embodiment, the bounding box manager 708 stores the resulting one or more center boxes as a set of coordinates relative to the heat map and the corresponding input image.

[0102] Moreover, as in Figure 7 As shown in FIG, the object detection system 702 includes an object location proposal generator 710. For example, as discussed above, the object location proposal generator 710 generates object location proposals based on an input image and a target object keyword. Furthermore, as discussed above, the object location proposal generator 710 generates object location proposals by utilizing various methods such as a selective search method and a YOLO network method.

[0103] In addition, as in Figure 7 As shown in FIG, the object detection system 702 includes a proposal selection manager 712. In one or more embodiments, the proposal selection manager 712 determines a score for each of the object location proposals generated by the object location proposal generator 710 based on the bounding box and the one or more center boxes. For example, the proposal selection manager 712 determines the score for each of the object location proposals by utilizing Equations 1 and 2 discussed above.

[0104] In one or more embodiments, the proposal selection manager 712 filters out poor object location proposals based on the determined scores. For example, in at least one embodiment, the proposal selection manager 712 filters out object location proposals with scores below a threshold percentage (e.g., 70%) of a maximum score. Furthermore, in at least one embodiment, the proposal selection manager 712 also filters out object location proposals that overlap with higher-scoring object location proposals by more than a threshold amount. Furthermore, in at least one embodiment, the proposal selection manager 712 also filters out object location proposals that are not well covered by a bounding box and at least one of the one or more center boxes.

[0105] In one or more embodiments, the proposal selection manager 712 further determines whether any object location proposals remain after the filtering process. In at least one embodiment, and in response to determining that no object location proposals remain after the filtering process, the proposal selection manager 712 resizes one of the at least one center box. For example, the proposal selection manager 712 resizes the largest center box in the one or more center boxes by a threshold percentage (e.g., enlarging the largest center box by 50%).

[0106] In addition, in one or more embodiments, the proposal selection manager 712 provides an input image with an indication of the object location associated with the target object keyword superimposed thereon. For example, the proposal selection manager 712 generates the indication as a rectangle having a size corresponding to the size of the highest-scoring object location proposal. The proposal selection manager 712 then superimposes the generated indication on the corner coordinates of the input image corresponding to the corner coordinates of the highest-scoring object location proposal.

[0107] In at least one embodiment, the proposal selection manager 712 clusters the input images based on the locations of detected objects associated with the target object keyword. For example, if the object detection system 702 is operating in batch mode and detecting object locations on a large dataset of input images, the object detection system 702 may detect the corresponding object for each of the input images, regardless of where the object is located within the input image. Therefore, the proposal selection manager 712 may analyze the size and location of the detected objects within each input image to cluster the input images based on the detected objects. Thus, for example, the proposal selection manager 712 may only provide input images in which an object is detected in the lower left corner of the input image.

[0108] As in Figure 7As further illustrated in FIG, object detection system 702 includes a data storage device 714 that includes digital media data 716 and object detection data 718. In one or more embodiments, digital media data 716 includes a repository of digital media information, such as input images. In one or more embodiments, object detection data 718 includes object detection information, such as an embedding neural network, a fully convolutional dense labeling neural network, generated heat maps, bounding boxes, center boxes, and indications of object locations.

[0109] Figure 8 An example environment 800 is shown in which the object detection system 702 may operate. Figure 8 , environment 800 includes a client computing device 802 having a client application 806. Furthermore, in one or more embodiments, environment 800 also includes a network 808 that connects the client computing device 802 and server(s) 804 to each other and provides for data exchange between the client computing device 802 and server(s) 804. Figure 8 Also shown is that server(s) 804 can host the digital media management system 701 and the object detection system 702. In this configuration, a client application 806 can include an application that communicates with the object detection system 702 hosted by the server(s) 804. For example, the client application 806 can include a web browser or a native software application. The client computing device 802 can send a request to the object detection system 702 hosted by the server(s) 804 to identify objects corresponding to target object keywords in an image or set of images. The object detection system 702 can then identify objects corresponding to the target object keywords in the image(s) as described above. Then, in response to the request, the object detection system 702 can download the images and the identified objects to the client computing device 802.

[0110] Alternatively, object detection system 702 may include some components operating on client computing device 802 and other components operating on server(s) 804. Still further, object detection system 702 may be installed on client computing device 802 as a native application, a web browser plug-in, or other type of application plug-in (e.g., a social media application plug-in).

[0111] The client computing device 802 may include a computing device such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smartphone, other suitable electronic devices, or any suitable combination thereof. The client computing device 802 may enable a user of the client computing device 802 to access a network 808. The client computing device 802 may enable its user to communicate with other users at other client computing devices.

[0112] In one or more embodiments, server(s) 804 may be capable of linking various entities. As an example and not by way of limitation, server(s) 804 may enable users to interact with each other and receive content from third-party systems or other entities, or allow users to interact with these entities through an application programming interface (API) or other communication channel.

[0113] In one or more embodiments, server 804 may include various servers, subsystems, programs, modules, logs, and data storage devices. In a particular embodiment, server(s) 804 may include one or more of the following: a network server, an action recorder, an API request server, a relevance and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object exposure log, an inference module, an authorization or privacy server, a search module, an advertisement targeting module, a user interface module, a user profile storage device, a connection storage device, a third-party content storage device, or a location storage device. Server(s) 804 may also include suitable components such as network interfaces, security mechanisms, load balancers, failover servers, management and network operations consoles, other suitable components, or any suitable combination thereof.

[0114] Now turn Figure 9 , which illustrates a flow chart of a series of actions 900 for detecting an object in an input image, according to one or more embodiments described herein. Figure 9 The actions according to one embodiment are illustrated, but alternative embodiments may omit the Figure 9 Any action shown in the action, to Figure 9 Add any of the actions shown in the Figure 9 Any of the actions shown in the sequence, and / or modify the Figure 9 Any of the actions shown in . Figure 9 Alternatively, the non-transitory computer readable medium may include instructions that, when executed by one or more processors, cause the computing device to perform Figure 9In a further embodiment, the system may perform Figure 9 action.

[0115] As in Figure 9 As shown in FIG, a series of actions 900 includes an action 910 of generating a heat map. For example, action 910 can involve generating a heat map of an input image based on target object keywords. In one or more embodiments, generating a heat map of the input image based on the target object keywords includes providing the input image and the target object keywords to a fully convolutional network trained for dense image labeling.

[0116] Moreover, as in Figure 9 As shown in FIG, a series of actions 900 includes an action 920 of generating a bounding box. For example, action 920 can involve generating a bounding box in the heat map based on a global threshold. In one or more embodiments, generating a bounding box in the heat map based on the global threshold includes: identifying all pixels in the heat map with pixel values greater than the global threshold, determining a connected shape based on the identified pixels, and adapting the bounding box to the connected shape in the heat map.

[0117] Furthermore, the series of actions 900 includes an action 930 of generating one or more center boxes. For example, action 930 can involve generating one or more center boxes in the heat map based on one or more local thresholds. In one or more embodiments, generating one or more center boxes in the heat map based on the one or more local thresholds includes identifying pixels associated with local maxima in each row and column in the heat map, determining a connected shape for each region of the identified pixels, and fitting the center box to each connected shape.

[0118] The series of actions 900 also includes an action 940 of determining one or more object location proposals. For example, action 940 can involve determining one or more object location proposals in the input image based on the target object keyword. In one or more embodiments, determining the one or more object location proposals in the input image based on the target object keyword includes determining the one or more object location proposals in the input image via one or more methods selected from a selective search method or a YOLO network method.

[0119] In addition, as in Figure 9As shown in FIG, a series of actions 900 includes an action 950 of determining a score for each of the object location proposals. For example, action 950 may involve determining a score for each of the one or more object location proposals determined in the input image based on an overlap between the object location proposal and the bounding box and an overlap between the object location proposal and the one or more center boxes. In one or more embodiments, determining a score for each of the one or more object location proposals determined in the input image includes: determining a score based on an amount of overlap between the object location proposal and the bounding box; determining a correlation of overlap between the object location proposal and each of the one or more center boxes; determining an average pixel value of the object location proposal; and determining a score for the object location proposal based on the determined score based on the amount of overlap between the object location proposal and the bounding box, the correlation of overlap between the object location proposal and each of the one or more center boxes, and the determined average pixel value of the object location proposal.

[0120] In at least one embodiment, determining the correlation of the overlap between the object location proposal and each of the one or more center boxes includes: determining an intersection-over-union (IoU) ratio between the center box and the object location proposal; determining a score based on the amount of overlap between the center box and the object location proposal; determining a score based on the amount of overlap between the object location proposal and the center box; and determining the correlation of the overlap between the object location proposal and the center box based on the determined IoU ratio, the determined score based on the amount of overlap between the center box and the object location proposal, and the determined score based on the amount of overlap between the object location proposal and the center box.

[0121] In addition, in one or more embodiments, the series of actions 900 includes an action of filtering out object location proposals having a determined score below a predetermined threshold. In addition, in at least one embodiment, the series of actions 900 includes an action of filtering out object location proposals in the heat map that are not covered by a bounding box and filtering out object location proposals in the heat map that are not covered by at least one center box in the one or more center boxes. In addition, in one or more embodiments, the series of actions 900 includes an action of determining whether all object location proposals in the one or more object location proposals have been filtered out. For example, if all object location proposals in the one or more object location proposals have been filtered out, the action includes resizing one of the one or more center boxes. In this embodiment, an input image is provided with an indication of the object location associated with the target object keyword superimposed thereon based on the resized center box.

[0122] In addition, as in Figure 9As shown in FIG, a series of actions 900 includes an action 960 of providing an indication of a highest-scoring object location proposal for an input image. For example, action 960 can involve providing an input image with an indication of an object location associated with a target object keyword superimposed thereon based on the highest-scoring object location proposal. In one or more embodiments, the indication of the object location is a bounding box having corner coordinates that match the corner coordinates of the highest-scoring object location proposal.

[0123] As used herein, the term "digital environment" generally refers to an environment that is implemented, for example, as a standalone application (e.g., a personal computer or mobile application running on a computing device), as an element of an application, as a plug-in to an application, as one or more library functions, as a computing device, and / or as a cloud computing system.

[0124] As discussed in more detail below, embodiments of the present disclosure may include or utilize a special-purpose or general-purpose computer that includes computer hardware (such as, for example, one or more processors and system memory). Embodiments within the scope of the present disclosure also include physical media and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Specifically, one or more processes described herein may be implemented at least in part as instructions embodied as non-transitory computer-readable media and executable by one or more computing devices (e.g., any of the media content access devices described herein). Typically, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., a memory, etc.) and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.

[0125] Computer-readable media can be any available medium that can be accessed by a general-purpose or special-purpose computer system. A computer-readable medium that stores computer-executable instructions is a non-transitory computer-readable storage medium (device). A computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of the present disclosure may include at least two distinct types of computer-readable media: a non-transitory computer-readable storage medium (device) and a transmission medium.

[0126] Non-transitory computer-readable storage media (devices) include: RAM, ROM, EEPROM, CD-ROM, solid-state drives ("SSD") (e.g., RAM-based solid-state drives), flash memory, phase-change memory ("PCM"), other types of memory, other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store computer-executable instructions or desired program code devices in the form of data structures and that can be accessed by a general or special purpose computer.

[0127] Furthermore, after the program code device in the form of computer-executable instructions or data structures reaches each computer system component, the program code can be automatically transferred to a non-transitory computer-readable storage medium (device) (or vice versa) via a transmission medium. For example, computer-executable instructions or data structures received via a network or data link can be buffered in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to the computer system RAM and / or a less volatile computer storage medium (device) at the computer system. Therefore, it should be understood that a non-transitory computer-readable storage medium (device) can be included in a computer system component that also (or even primarily) utilizes a transmission medium.

[0128] Computer executable instructions include, for example, instructions and data that, when executed at a processor, cause a general-purpose computer, a special-purpose computer, or a dedicated processing device to perform a specific function or group of functions. In some embodiments, computer executable instructions are executed on a general-purpose computer to convert the general-purpose computer into a special-purpose computer that implements the elements of the present disclosure. Computer executable instructions can be, for example, binary files, intermediate format instructions (such as assembly language), or even source code. Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. On the contrary, the described features and actions are disclosed as example forms of implementing the claims.

[0129] Those skilled in the art will appreciate that the present disclosure can be implemented in a network computing environment having many types of computer system configurations, including: personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablets, pagers, routers, switches, etc. The present disclosure can also be practiced in a distributed system environment in which local computer systems and remote computer systems linked (by hardwired data links, wireless data links, or a combination of hardwired data links and wireless data links) over a network all perform tasks. In a distributed system environment, program modules can be located in local memory storage devices and remote memory storage devices.

[0130] Embodiments of the present disclosure may also be implemented in a cloud computing environment. In this specification, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be employed in a market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly provisioned via virtualization, can be released with minimal management effort or service provider interaction, and can then be scaled accordingly.

[0131] Cloud computing models can include various features such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, and metered call services. Cloud computing models can also expose various service models such as, for example, Software as a Service ("SaaS"), Platform as a Service ("PaaS"), and Infrastructure as a Service ("IaaS"). Cloud computing models can also be deployed using different deployment models, such as private clouds, community clouds, public clouds, hybrid clouds, and the like. Throughout this specification and claims, a "cloud computing environment" is an environment that employs cloud computing.

[0132] Figure 10 An exemplary computing device 1000 that can be configured to perform one or more of the processes described above is illustrated in block diagram form. It is understood that the object detection system 702 can be implemented on an embodiment of the computing device 1000. Figure 10 As shown, the computing device may include a processor 1002, a memory 1004, a storage device 1006, an I / O interface 1008, and a communication interface 1010. In some embodiments, the computing device 1000 may include a processor 1002, a memory 1004, a storage device 1006, an I / O interface 1008, and a communication interface 1010. Figure 10 The output of the components in the will now be described in more detail Figure 10 Components of computing device 1000 are shown in FIG.

[0133] In particular embodiments, processor(s) 1002 include hardware for executing instructions, such as those comprising a computer program. As an example and not by way of limitation, to execute instructions, processor(s) 1002 may retrieve (or fetch) instructions from internal registers, internal cache, memory 1004, or storage device 1006, decode the instructions, and execute the instructions.

[0134] The computing device 1000 includes a memory 1004 coupled to the processor(s) 1002. The memory 1004 may be used to store data, metadata, and programs for execution by the processor(s). The memory 1004 may include one or more of a volatile memory and a non-volatile memory, such as a random access memory ("RAM"), a read-only memory ("ROM"), a solid-state drive ("SSD"), flash memory, a phase-change memory ("PCM"), or other types of data storage devices. The memory 1004 may be an internal memory or a distributed memory. In one or more embodiments, the memory 1004 stores or includes data storage.

[0135] The computing device 1000 includes a storage device 1006 that includes storage for storing data or instructions. As an example and not by way of limitation, the storage device 1006 may include the non-transitory storage medium described above. The storage device 1006 may include a hard disk drive (HDD), a flash memory, a universal serial bus (USB) drive, or a combination of these or other storage devices. In one or more embodiments, the storage device 1006 stores or includes data storage.

[0136] The computing device 1000 also includes one or more input or output ("I / O") devices / interfaces 1008 that are provided to allow a user to provide input (such as user strokes) to the computing device 1000, receive output from the computing device 1000, and otherwise transfer data to and from the computing device 1000. These I / O devices / interfaces 1008 may include a mouse, a keypad or keyboard, a touch screen, a camera, an optical scanner, a network interface, a modem, other known I / O devices, or a combination of such I / O devices / interfaces 1008. A touch screen may be activated with a stylus or a finger.

[0137] I / O devices / interfaces 1008 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, devices / interfaces 1008 are configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular embodiment.

[0138] The computing device 1000 may also include a communication interface 1010. The communication interface 1010 may include hardware, software, or both. The communication interface 1010 may provide one or more interfaces for communication (such as, for example, packet-based communication) between the computing device and one or more other computing devices 1000 or one or more networks. As an example and not by way of limitation, the communication interface 1010 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network (such as WI-FI). The computing device 1000 may also include a bus 1012. The bus 1012 may include hardware, software, or both that couples the components of the computing device 1000 to each other.

[0139] In the foregoing description, the present disclosure has been described with reference to specific exemplary embodiments thereof. Various embodiments and aspects of the present disclosure are described with reference to the details discussed herein, and the accompanying drawings illustrate various embodiments. The foregoing description and drawings are illustrative of the present disclosure and should not be construed as limiting the present disclosure. Numerous specific details are described to provide a thorough understanding of the various embodiments of the present disclosure.

[0140] Without departing from the spirit or key features of the present invention, the present disclosure may be embodied in other specific forms. The described embodiments are to be considered in all respects as illustrative and non-restrictive. For example, the methods described herein may be performed with fewer or more steps or actions, or the steps or actions may be performed in a different order. In addition, the steps or actions described herein may be repeated or performed in parallel with each other, or repeated or performed in parallel with different instances of the same or similar steps or actions. Therefore, the scope of the present application is indicated by the appended claims rather than by the preceding description. All changes within the meaning and scope of the equivalents of the claims are to be included within the scope of the claims.

Claims

1. A non-transitory computer-readable storage medium in a digital environment for detecting a location of an object in a digital image, the non-transitory computer-readable storage medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by a processor, causing a computing device to: Generate a heatmap of the input image based on the target object keywords; Generate a bounding box in the heatmap based on a global threshold by: Identifying pixels in the heat map using pixel values greater than the global threshold; determining a connection shape based on the identified pixels in the heat map; as well as adapting the bounding box to the connected shapes in the heatmap; generating one or more center boxes in the heat map based on one or more local thresholds; determining one or more object location proposals in the input image based on the target object keyword; determining a score for each of the one or more determined object position proposals in the input image based on an overlap between the object position proposal and the bounding box and based on an overlap between the object position proposal and the one or more center boxes; as well as Based on the highest scoring object location proposal, the input image is provided with an indication of the object location associated with the target object keyword superimposed thereon.

2. The non-transitory computer-readable storage medium of claim 1 , wherein the computer-executable instructions cause the computing device to generate the one or more center boxes in the heat map by: identifying pixels associated with local maxima in rows and columns in the heat map; determining a connected shape for each region of identified pixels; as well as Fits the center box to each connected shape.

3. The non-transitory computer-readable storage medium of claim 1 , wherein the computer-executable instructions cause the computing device to determine one or more object location proposals in the input image by identifying coordinates that summarize a predicted location or region in the input image that includes an object corresponding to the target object keyword.

4. The non-transitory computer-readable storage medium of claim 1 , wherein the computer-executable instructions cause the computing device to determine the score for each of the one or more object location proposals in the input image by determining, for each of the one or more object location proposals: a score based on an amount of overlap between the object location proposal and the bounding box; a correlation of overlap between the object location proposal and each of the one or more center boxes; an average pixel value of the object location proposal; and the score of the object position proposal based on the amount of overlap between the object position proposal and the bounding box, based on the correlation of the overlap between the object position proposal and each of the one or more center boxes, and based on the determined average pixel value of the object position proposal.

5. The non-transitory computer-readable storage medium of claim 4 , wherein determining, for each of the one or more object location proposals, the correlation of the overlap between the object location proposal and each of the one or more center boxes comprises: determining an intersection-over-union (IoU) between the center box and the object location proposal; determining a first score based on an amount of overlap between the center box and the object location proposal; determining a second score based on an amount of overlap between the object location proposal and the center box; And determining the correlation of the overlap between the object position proposal and the center box based on the determined intersection-over-union ratio, based on the determined first score based on the amount of overlap between the center box and the object position proposal, and based on the determined second score based on the amount of overlap between the object position proposal and the center box.

6. The non-transitory computer-readable storage medium of claim 1 , further storing computer-executable instructions that, when executed by the processor, cause the computing device to filter out object location proposals having a determined score below a predetermined threshold.

7. The non-transitory computer-readable storage medium of claim 6 , further storing computer-executable instructions that, when executed by the processor, cause the computing device to: Filtering out object location proposals in the heat map that are not covered by the bounding box; and Object position proposals in the heat map that are not covered by at least one center box of the one or more center boxes are filtered out.

8. The non-transitory computer-readable storage medium of claim 7, further storing computer-executable instructions that, when executed by the processor, cause the computing device to: determining whether all object location proposals in the one or more object location proposals have been filtered out; resizing a center box of the one or more center boxes if all of the one or more object location proposals have been filtered out; and The resized center box is used as the object location associated with the target object keyword.

9. A system in a digital environment for detecting a position of an object in a digital image, comprising: Memory, including: input image; and Fully convolutional networks trained for dense image labeling; at least one processor; and at least one non-transitory computer-readable storage medium having stored thereon instructions that, when executed by the at least one processor, cause the system to: generating a heat map of the input image based on a target object keyword by providing the input image and the target object keyword to the fully convolutional network trained for dense image labeling; Generate a bounding box in the heatmap based on a global threshold by: Identifying pixels in the heat map using pixel values greater than the global threshold; determining a connection shape based on the identified pixels in the heat map; and adapting the bounding box to the connected shapes in the heatmap; generating one or more center boxes in the heat map based on one or more local thresholds; determining one or more object location proposals in the input image based on the target object keyword; determining a score for each of the determined one or more object position proposals in the input image based on an overlap between the object position proposal and the bounding box and based on an overlap between the object position proposal and the one or more center boxes; and Based on the highest scoring object location proposal, the input image is provided with an indication of an object location associated with the target object keyword superimposed thereon.

10. The system of claim 9, wherein the instructions, when executed by the at least one processor, cause the system to generate the bounding box in the heat map based on the global threshold by: Identifying all pixels in the heat map with pixel values greater than a global threshold; and The bounding box is fit within a rectangle to the identified pixels in the heat map.

11. The system of claim 10, wherein the instructions, when executed by the at least one processor, cause the system to generate one or more center boxes in the heat map based on one or more local thresholds by: identifying pixels associated with local maxima in each row and column of the heat map; determining a connected shape for each region of identified pixels; as well as Fits the center box to each connected shape.

12. A system according to claim 11, wherein the instructions, when executed by the at least one processor, cause the system to determine one or more object location proposals in the input image by identifying coordinates that summarize the predicted location or area of the input image that includes an object corresponding to the target object keyword.

13. The system of claim 12 , wherein the instructions, when executed by the at least one processor, cause the system to determine, for each of the one or more object location proposals, a score for each of the determined one or more object location proposals in the input image by: calculating a first score based on an amount of overlap between the object location proposal and the bounding box; determining a correlation of overlap between the object location proposal and each of the one or more center boxes; determining an average pixel value of the object location proposals; as well as The score of the object position proposal is determined based on the determined amount of overlap between the object position proposal and the bounding box, the correlation of the overlap between the object position proposal and each of the one or more center boxes, and the determined average pixel value of the object position proposal.

14. The system of claim 13 , wherein determining the correlation of the overlap between the object location proposal and each of the one or more center boxes comprises: determining an intersection-over-union (IoU) between the center box and the object location proposal; determining a second score based on an amount of overlap between the center box and the object location proposal; determining a third score based on an amount of overlap between the object location proposal and the center box; as well as The correlation of the overlap between the object position proposal and the center box is determined based on the determined intersection-over-union ratio, based on the determined second score based on the amount of overlap between the center box and the object position proposal, and based on the determined third score based on the amount of overlap between the object position proposal and the center box.

15. The system of claim 14, further having stored thereon instructions that, when executed by the at least one processor, cause the system to filter out object location proposals having a determined score below a predetermined threshold.

16. The system of claim 15, further having stored thereon instructions that, when executed by the at least one processor, cause the system to: Filtering out object location proposals in the heat map that are not covered by the bounding box; and Object position proposals in the heat map that are not covered by at least one center box of the one or more center boxes are filtered out.

17. The system of claim 16, further having stored thereon instructions that, when executed by the at least one processor, cause the system to: determining whether all object location proposals in the one or more object location proposals have been filtered out; resizing a center box of the one or more center boxes if all of the one or more object location proposals have been filtered out; and The resized center box is used as the object location associated with the target object keyword.

18. A method for detecting one or more objects in a digital image, the method being used in a digital environment for analyzing digital images and comprising: performing the step of generating a heat map of the input image based on the target object keyword; Generate bounding boxes in the heatmap based on a global threshold in the following way: Identifying pixels in the heat map using pixel values greater than the global threshold; determining a connection shape based on the identified pixels in the heat map; as well as adapting the bounding box to the connected shapes in the heatmap; generating one or more center boxes in the heat map based on one or more local thresholds; determining one or more object location proposals in the input image based on the target object keyword; performing a step for determining a score for each of the one or more determined object location proposals in the input image; as well as Based on the highest scoring object location proposal, the input image is provided with an indication of the object location associated with the target object keyword superimposed thereon.

19. The method according to claim 18, wherein: Generating the bounding box in the heat map based on the global threshold comprises: Identify all pixels in the heatmap with pixel values greater than a global threshold, and fitting the bounding box in a rectangle to the identified pixels in the heat map; and Generating one or more center boxes in the heat map based on one or more local thresholds includes: identifying pixels associated with local maxima in each row and column of the heat map; determining a connected shape for each region of identified pixels; and Fits the center box to each connected shape.

Images