A pipeline for identifying supplementary content items related to objects in an image
By using a deep neural network on a client computing device to identify objects in web page images and generate descriptive text, the problem of unable to automatically identify objects in the prior art is solved, and the ability to automatically display supplementary content items related to images is realized.
Patent Information
- Application Number
- CN202080038741.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-05-25
- Filing Date
- 2020-03-25
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-03-25
AI Technical Summary
The prior art is difficult to automatically identify objects in images on web pages, resulting in the inability to effectively display supplementary content items related to images, especially if the objects in the image are not described in accompanying text.
Detect web pages through the interface module on the client computing device. The computing system parses the web page text and uses a deep neural network (DNN) to identify objects in the image, generate descriptive text, and find supplementary content items related to the object based on the text, displaying On the web page.
It realizes the ability to automatically identify image objects in web pages, avoids the dependence of manual annotation, and improves the display efficiency and accuracy of image-related content.
Smart Images

Figure CN113874878B_ABST
Abstract
Description
Background Art
[0001] Supplemental content items are often provided on a web page to generate revenue for the publisher of the web page. For example, a company may pay a publisher to display a supplemental content item on a web page, where the web page is published by the publisher, and further, where the supplemental content item describes a product or service that the company can use to obtain.
[0002] Typically, a company wishes to show a supplementary content item on a web page where the content of the web page is related to the content of the supplementary content item. For example, a company that sells electronic products may wish to display a supplementary content item on a web page that includes information about television programs (including information about televisions that can be purchased from the company).
[0003] A web page may include an image, where the image captures an object. For an exemplary web page that includes an image, the text accompanying the image on the web page may not be descriptive of the content of the image. For example, a web page may include an image of a person wearing a hat, and the text accompanying the image may reference the person. However, the text accompanying the image does not mention the hat worn by the person in the image. Therefore, even though a company may want to present supplemental content items related to hats, there is generally no suitable computer-implemented method to automatically identify supplemental content items to display with an image on a web page. Summary of the invention
[0004] The following is a brief summary of subject matter that is described in greater detail herein. This summary is not intended to limit the scope of the claims.
[0005] Described herein are various technologies for identifying supplementary content items for presentation on a web page based on an image included in a web page. An interface module on a client computing device detects that the client computing device has loaded a web page, and in response to detecting that the client computing device has loaded a web page, transmits an identifier (e.g., URL) of the web page to a computing system. The computing system receives the identifier of the web page from the client computing device and retrieves the text of the web page based on the identifier of the web page. In addition, in response to receiving the identifier of the web page, the computing system further ascertains that the web page includes an image. The computing system can parse the text of the web page to ascertain whether to include supplementary content items on the web page while the web page is being viewed at the client computing device. For example, when the text of the web page discusses controversial topics, when the sentiment of the text in the web page is negative, when the text in the web page includes one or more "stop" words or phrases (such as profanity and / or other words or phrases associated with controversy), etc., the company may not want to show the supplementary content items corresponding to it on the web page. When the computing system determines not to present the supplementary content item on the web page, the computing system terminates processing of the web page.
[0006] When the computing system determines that it is appropriate to include a supplemental content item on the web page, the computing system retrieves the image in the web page. For example, upon detecting that the client computing device has loaded the web page, the interface module may transmit a uniform resource locator (URL) of the image to the computing system, and the computing system then retrieves the image based on the URL. In an exemplary embodiment, once the computing system has retrieved the image, the computing system may provide the image as an input to a first deep neural network (DNN), which is configured to identify objects of predefined types in the image. In a non-limiting example, the first DNN may be configured to identify clothing in the image, such as shirts, pants, hats, scarves, shoes, etc. In another example, the first DNN may be configured to identify furniture in the image, such as sofas, beds, wardrobes, tables, chairs, etc. In addition, the first DNN is configured to define a bounding box that approximately corresponds to the boundaries of the identified object in the image, wherein a portion of the image may be extracted based on the identified bounding box (e.g., such that a portion of the image extracted from the image includes an identified object isolated from other objects captured in the image). Thus, in one example, when an image provided to the first DNN includes a person wearing a shirt, the first DNN may identify the shirt and form a bounding box around the shirt. The portion of the image defined by the bounding box is extracted from the image.
[0007] In response to the computing system extracting the portion of the image including the object from the image, the computing system is configured to generate text describing the object based on the portion of the image. In an exemplary embodiment, the portion of the image is provided to a plurality of DNNs, each of which is configured to output a word or phrase representing a value of a corresponding attribute of the object. In one example, the plurality of DNNs may include a second DNN, a third DNN, and a fourth DNN. The second DNN is configured to receive the portion of the image and output a word or phrase representing a value of a first attribute of the object, the third DNN is configured to receive the portion of the image and output a word or phrase representing a value of a second attribute of the object, and the fourth DNN is configured to receive the portion of the image and output a word or phrase representing a value of a third attribute of the object.
[0008] According to one example, the object may be a black shirt for men, the first attribute may be "sex / age", the second attribute may be "color", and the third attribute may be "type". Thus, the second DNN may output the word "men's", the third DNN may output the word "black", and the fourth DNN may output the word "shirt". Other exemplary attributes will be readily considered.
[0009] Once multiple DNNs have output words or phrases representing the values of the attributes of the object captured in the portion of the image, the output words or phrases can be arranged in a semantically correct sequence. For example, the words output by the second DNN can be placed in the first position in the sequence, the words output by the third DNN can be placed in the second position in the sequence, and the words output by the fourth DNN can be placed in the third position in the sequence. The computing system can then transmit the sequence of words and / or phrases to the search system, whereupon the search system identifies one or more supplementary content items based on the sequence of words and / or phrases. The search system can cause the supplementary content items to be transmitted to the client computing device, and then the interface module can cause the supplementary content items to be presented on a web page of the client computing device together with the image. In another example, the interface module can cause a selectable graphical icon to be presented on a web page, wherein when the selectable graphical icon is selected, the supplementary content item identified by the search system is presented on the web page. Therefore, unlike traditional methods that typically involve manual annotation by a human annotator, the technology described herein enables supplementary content items related to an object captured in an image to be automatically identified, even if the text corresponding to the image does not describe the object in the image.
[0010] The above summary presents a simplified overview in order to provide a basic understanding of some aspects of the systems and / or methods discussed herein. This summary is not an extensive overview of the systems and / or methods discussed herein. It is not intended to identify key / essential elements or to define the scope of such systems and / or methods. Its sole purpose is to present some concepts in a simplified form as a prelude to a more detailed description presented later. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 is a functional block diagram of an exemplary system that supports identification of supplemental content items related to an object captured in an image.
[0012] Figure 2 An exemplary image including several objects is shown.
[0013] Figure 3 Bounding boxes around objects captured in exemplary images are shown.
[0014] Figure 4 is a functional block diagram of an exemplary text generator system configured to generate text describing an object captured in an image.
[0015] Figure 5 is a functional block diagram of an exemplary text generator system that includes a hierarchical arrangement of deep neural networks.
[0016] Figure 6 An exemplary web page having an image included therein is depicted, wherein supplemental content items are displayed on the web page.
[0017] Figure 7 Depicted is a web page including an image in which a selectable graphical icon is presented on the image to indicate that supplemental content items related to an object captured in the image are available.
[0018] Figure 8 A web page including a supplemental content item displayed on an image is depicted.
[0019] Fig. 9 is a flow chart illustrating an exemplary method for generating text describing an object captured in an image, where the image is included in a web page viewed at a client computing device.
[0020] Fig.10 is a flow chart illustrating an exemplary method for outputting words representing values of attributes of an object captured in an image included in a web page.
[0021] Fig.11 is an exemplary computing system. DETAILED DESCRIPTION
[0022] Various techniques for causing supplementary content items associated with images in a web page to be presented on the web page when the web page is viewed at a client computing device are now described with reference to the accompanying drawings, wherein similar reference numerals are used to reference similar elements throughout. In the following description, for the purpose of explanation, many specific details are set forth to provide a thorough understanding of one or more aspects. However, it is apparent that these aspects can be practiced without these specific details. In other cases, well-known structures and devices are shown in block diagram form to facilitate description of one or more aspects. In addition, it should be understood that functions described as being performed by certain system components can be performed by multiple components. Similarly, for example, a component can be configured to perform functions described as being performed by multiple components.
[0023] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless specified otherwise or clear from the context, the phrase "X employs A or B" is intended to mean any of the natural inclusive arrangements. That is, the phrase "X employs A or B" is satisfied by any of the following: X employs A; X employs B; or X employs both A and B. Furthermore, the articles "a" and "an" as used in this application and the appended claims should generally be construed to mean "one or more" unless specified otherwise or clear from the context to direct to a singular form.
[0024] Furthermore, as used herein, the terms "component," "system," and "module" are intended to encompass computer-readable data storage configured with computer-executable instructions that, when executed by a processor, cause a certain function to be performed. Computer-executable instructions may include routines, functions, etc. It should also be understood that a component or system may be located on a single device or distributed across multiple devices. Furthermore, as used herein, the term "exemplary" is intended to mean serving as an illustration or example of something and is not intended to express a preference.
[0025] Various technologies that support identifying supplementary content items related to objects in an image are described herein. More specifically, a processing pipeline for presenting supplementary content items on a web page is described herein. First, an image on a web page viewed at a client computing device is identified, and it is determined that the image includes an object of one type from a plurality of predefined types. A portion of the image including the object is extracted from the image, and text describing the object is generated based on the portion of the image. Supplementary content items related to the object captured in the image are identified based on the generated text. In response to the supplementary content item being identified, the supplementary content item can be caused to be displayed on the web page viewed at the client computing device.
[0026] Reference now Figure 1 , a functional block diagram of an exemplary system 100 is shown, wherein the system 100 supports identification of supplemental content items related to an object in an image. The system 100 includes a client computing device 102, which can be any suitable type of client computing device, including but not limited to a desktop computing device, a laptop computing device, a tablet computing device, a video game console, a mobile phone, a wearable computing device, etc. The client computing device 102 includes a display 104 or is in communication with the display 104. In addition, the client computing device 102 has an interface module 105 executed thereon, and the interface module 105 is configured to interface the client computing device 102 with other computing systems. The interface module 105 is described in more detail below.
[0027] The system 100 further includes a computing system 106 in network communication with the client computing device 102 via the interface module 105, wherein the computing system 106 is configured to generate text describing an object in an image. The computing system 106 includes a processor 108 and a memory 110, the memory 110 including instructions executed by the processor 108. More specifically, the memory 110 includes a sentiment analysis module 111, the sentiment analysis module 111 is configured to determine whether it is appropriate to display supplemental content on a web page. The memory 110 further includes an object identifier system 112 configured to identify the presence of a predefined type of object in the image. The object identifier system 112 is also configured to define a bounding box in the image corresponding to the boundaries of the object in the image. The object identifier system 112 is also configured to extract a portion of the image from the image, wherein the portion extracted from the image is a portion including the object. In one example, the object identifier system 112 may include a deep neural network (DNN) that is trained to identify predefined types of objects in the image and is further trained to define a bounding box corresponding to the boundaries of these objects in the image. In a non-limiting example, the DNN of the object identifier system 112 may be configured to identify 100 or more types of objects (e.g., the DNN may include at least 100 output nodes). The DNN may be trained using labeled training data, wherein the later labeled training data includes images having bounding boxes surrounding the images, and further, wherein labels are assigned to the bounding boxes to indicate the type of objects included within the bounding boxes. The training data may also optionally include images included in electronic advertisements, wherein such electronic advertisements include labels identifying the types of objects in the images.
[0028] The memory 110 also includes a text generator system 114, which is configured to receive a portion of an image output by the object identifier system 112, and is also configured to generate text describing an object included in the portion of the image. According to one example, the text generator system 114 includes a plurality of DNNs, wherein each of the plurality of DNNs is configured to output a word or phrase representing a value of a corresponding attribute of the object. For example, the text generator system 114 may include a first DNN, which upon receiving a portion of an image including an object, is configured to output a word or phrase representing the color of the object. In another example, the text generator system 114 may include a second DNN, which upon receiving the portion of the image, is configured to output a word or phrase representing a gender (and age range) associated with the object. In yet another example, the text generator system 114 may include a third DNN, which upon receiving the portion of the image, is configured to output a word or phrase representing the type (name) of the object. The number of DNNs in the text generator system 114 and the attributes for which the DNNs are trained may be determined depending on the type of object for which the object identifier system 112 is trained to identify. For example, when the object identifier system 112 is trained to identify clothing, the text generator system 114 may include a DNN configured to output words or phrases representing the corresponding attributes "color", "gender", and "name". In another example, when the object identifier system 112 is trained to identify furniture, the text generator system 114 may include a DNN configured to output words or phrases representing the corresponding attributes "material", "color", and "name".
[0029] The system 100 additionally includes a data repository 116 in network communication with the computing system 106. The data repository 116 may include images, metadata such as detected bounding boxes, generated words, sentiment analysis results, and the computing system 106 may retrieve one or more of the data types from the data repository 116. Although shown as being separate from the computing system 106, it should be understood that the computing system 106 may include all or part of the data repository 116.
[0030] The system 100 also includes a search computing system 118, wherein the search computing system 118 is in network communication with the computing system 106. The search computing system 118 is configured to receive the text generated by the text generator system 114 and is further configured to search for supplementary content items based on the received text. The search computing system 118 includes a processor 120 and a memory 122, wherein the memory 122 includes a search module 124. The memory 122 also includes a supplementary content item index 126 that can be searched by the search module 124. The search module 124 receives the text describing the object (generated by the text generator system 114) and searches the supplementary content item index 126 for supplementary content items related to the object based on the text. The search computing system 118 is in network communication with the computing system 106, wherein the computing system 106 receives the supplementary content items from the search system 118. The computing system 106 can transmit the supplementary content items related to the object to the client computing device 102, whereupon the client computing device 102 can display the supplementary content items to the user.
[0031] An exemplary operation of the system 100 is now described. In an exemplary embodiment, a user of a client computing device 102 causes a web browser executing thereon to load a page 128 (where the web page has a URL assigned thereto). When the web browser loads the page 128, the page 128 is presented on the display 104 of the client computing device 102. The page 128 includes an image 130, where the image 130 may be a still image or a frame of a video.
[0032] Brief reference Figure 2 , presents an exemplary depiction of image 130. Image 130 includes several objects. Specifically, image 130 includes trees 202 and 204, dog 206, person 208, shirt 210 worn by person 208, and pants 212 worn by person 208.
[0033] return Figure 1 , the page 128 may also include or refer to the interface module 105. In an exemplary embodiment, the interface module 105 may be or include a script placed in a web page. The interface module 105, when executed by a web browser, causes the client computing device 102 to communicate with the computing system 106 (e.g., send data to and receive data from the computing system 106). More specifically, when executed by a web browser, the interface module 105 causes the client computing device 102 to transmit to the computing system 106 an indication that the page 128 is being presented on the display 104 of the client computing device 102. For example, the indication may include the URL of the page 128, the URL of the image 130 on the page 128, the image 130 itself, the text of the page 128, etc.
[0034] When computing system 106 receives an indication that page 128 is being viewed at client computing device 102, sentiment analysis module 111 may parse the text of the page to ascertain the sentiment of page 128 and / or determine whether any words on the blacklist are included in the text on page 128. For example, when sentiment analysis module 111 determines that the sentiment of the page is highly negative, sentiment analysis module 111 may output an indication that the supplemental content item should not be displayed on page 128. In another example, when sentiment analysis module 111 determines that the text of page 128 includes one or more words or phrases on the blacklist (such as profanity or words and / or phrases related to controversial topics), sentiment analysis module 111 may output an indication that the supplemental content item should not be displayed on page 128. When sentiment analysis module 111 determines that the content of sentiment page 128 is not highly negative and / or when sentiment analysis module 111 determines that the text of page 128 does not include one or more words on the blacklist, computing system 106 may perform actions in support of identifying supplemental content items to be displayed on page 128.
[0035] In an exemplary embodiment, the indication received from the client computing device 102 that the page 128 is being displayed on the display 104 may include a URL of the image 130. In response to receiving the indication from the client computing device 102, the computing system 106 may retrieve the image 130 from the data repository 116 based on the URL of the image 130. The computing system 106 provides the image 130 as an input to the object identifier system 112, wherein the object identifier system 112 identifies one or more objects in the image 130, wherein the objects are of a type from among a plurality of predefined types that the object identifier system 112 is trained to identify. In addition, the object identifier system 112 may define a bounding box corresponding to the boundaries of the identified object in the image 130, and may extract from the image 130 a portion of the image 130 defined by the bounding box. Thus, the object identifier system 112 extracts a portion of the image 130 that isolates the identified object from other objects in the image 130.
[0036] Brief reference Figure 3, image 130 is provided as input to object identifier system 112, wherein (in this example) the DNN of object identifier system 112 is configured to identify clothing items in the image. In exemplary image 130, two items of clothing are captured: 1) shirt 210; and 2) pants 212 worn by person 208. Further, object identifier system 112 may define a first bounding box 302 that includes shirt 210 (but not pants 212) and may further identify a second bounding box 304 that includes pants 212 (but not shirt 210). Object identifier system 112 may extract two portions 306 and 308 of image 130 corresponding to bounding boxes 302 and 304, respectively, and may provide such portions 306 and 308 to text generator system 114. For purposes of explanation, provision of portion 306 of image 130 to text generator system 114 is described; however, it should be understood that text generator system 114 performs similar actions with respect to portion 308 of image 130.
[0037] Return to Figure 1 , the text generator system 114 receives the portion 306 of the image 130 defined by the bounding box 302 output by the object identifier system 112. Figure 4 , a functional block diagram of a text generator system 114 is shown. The text generator system 114 includes a plurality of DNNs 402-404, wherein each of the DNNs 402-404 is configured to output a word or phrase representing an attribute of an object included in a portion 306 of an image 130. In one example, the plurality of DNNs 402-404 may include a first DNN 402, a second DNN, and a third DNN. Upon receiving a portion of an image including an object, the first DNN 402 is configured to output a value representing the attribute "color" of the object; upon receiving the portion of the image, the second DNN is configured to output a value representing the attribute "gender and age" of the object, and upon receiving the portion of the image, the third DNN is configured to output a value representing the attribute "type" of the object.
[0038] Thus, portion 306 of image 130 is provided to first DNN 402, and first DNN 402 is configured to output a word or phrase representing the color of shirt 210 (e.g., "black"). Portion 306 of image 130 is also provided to a second DNN, and the second DNN is configured to output a word or phrase representing the gender / age associated with shirt 210 (e.g., "men's"). The portion of image 130 is also provided to a third DNN, and the third DNN is configured to output a word or phrase representing the type of clothing item (e.g., "shirt"). DNNs 402-404 can be executed in parallel. In another example, the input of one DNN can be based on the output of another DNN.
[0039] The text generator system 114 can then be configured to arrange the words and / or phrases in a semantically and syntactically correct sequence. Thus, the text generator system 114 can position the word output by the second DNN at a first position in the sequence, can position the word output by the first DNN 402 at a second position in the sequence, and can position the word output by the third DNN at a third position in the sequence. Thus, the text generator system 114 can output the phrase "Men's black shirt".
[0040] It should be understood that the text generator system 114 may include any suitable number of DNNs that are trained to output words and / or phrases that represent the value of an attribute, wherein the number of DNNs in the text generator system 114 may be a function of the type of object identified by the object identifier system 112. In addition, the DNNs of the text generator system 114 are trained based on labeled training data, wherein the labeled training data may be obtained through crowdsourcing efforts, from electronic advertisements, etc. For example, a conventional electronic advertisement includes an image of an object and a description of the object captured in the image. Therefore, the images in the electronic advertisement and their labels may be used to train one or more of the DNNs of the text generator system 114.
[0041] Return to Figure 1In response to the text generator system 114 outputting a sequence of words and / or phrases describing the shirt 210 included in the portion 306 of the image 130, the computing system 106 may transmit the sequence of words and / or phrases (hereinafter referred to as text) to the search computing system 118. The search module 124 receives the text and searches the supplemental content item index 126 based on the text. Thus, the search module 124 performs a search of the supplemental content item index 126 using the query "men's black shirts". In response to identifying the supplemental content item, the search module 124 transmits the supplemental content item (or a URL pointing to the supplemental content item) to the computing system 106, which in turn transmits the supplemental content item (or a URL pointing to the supplemental content item) to the client computing device 102 (e.g., to the interface module 105). The interface module 105 may then cause the supplemental content item to be presented on the page 128 in association with the image 130. In another example, upon receiving the supplemental content item, the interface module 105 may cause a graphical icon to be presented on the page 128, whereupon in response to the user of the client computing device 102 selecting the selectable graphical icon, the supplemental content item identified by the search module 124 is presented on the page 128. The supplemental content item may be any suitable electronic item, and may include, but is not limited to, an image, a video text, etc. According to a specific example, the supplemental content item is an electronic advertisement, wherein upon selection of the electronic advertisement, the client computing device 102 is configured to load a page in which the product or service promoted by the electronic advertisement may be purchased. In this example, the supplemental content item may be an electronic advertisement for a men's black shirt available for purchase from a company.
[0042] As can be determined from the foregoing, the client computing device 102, computing system 106, and search system 118 can operate in conjunction with identifying supplemental content items related to objects in images viewed on web pages of the client computing device 102. In contrast to conventional approaches, identification of such supplemental content items does not rely on tags manually assigned by human taggers to images in the page and / or text accompanying such images. Referring again to Figure 2 , image 130 may be a landscape, and text corresponding to such an image on page 128 may describe the identity of person 208, the identification of the location where image 130 was captured, etc. However, the text accompanying image 130 on page 128 may not relate to shirt 210 worn by person 208. Nevertheless, object identifier system 112 may identify that image 130 includes shirt 210, and text generator system 114 may generate text describing shirt 210. Search computing system 118 may then identify supplemental content items related to shirt 210, even though the text of page 128 does not relate to shirt 210.
[0043] Reference now Figure 5, another exemplary functional block diagram of at least a portion of the text generator system 114 is shown. The text generator system 114 includes an Nth DNN 404 and a subclass 502 DNN, wherein the Nth DNN 404 and the subclass DNN 502 are arranged hierarchically. In this example, the Nth DNN 404 can be configured to output words and / or phrases representing the attribute "type" of a clothing item. To this end, the Nth DNN 404 can include a plurality of output nodes 504-506 representing clothing categories; thus, the first output node 504 can represent the category "shirt", the second output node can represent the category "pants", the third output node can represent the category "shoes", and so on. When training the Nth DNN 404, the amount of training data across all desired categories may be unbalanced. For example, the desired output category may be "scarf"; however, there may be relatively few training examples including scarves. In contrast, there may be a very large number of training examples including shirts and a very large number of training examples including pants. Therefore, the amount of training data across the categories "shirt", "pants", and "scarf" may be unbalanced.
[0044] To address the unbalanced amount of training examples across desired categories, it is contemplated that the output nodes in the Nth DNN 404 may represent categories that are supersets of subcategories of such categories. Thus, for example, the Mth node 506 may represent a category that includes several subcategories such as "scarf," "hat," "earmuffs," and "sunglasses." The number of subcategories represented by the Mth node 506 in the Nth DNN 404 may be a function of the amount of training data across categories. In other words, the amount of training data for the category represented by the Mth node 506 may be approximately equal to the amount of training data for the first category represented by the first node 504.
[0045] As described above, the subclass DNN 502 is hierarchically arranged with the Nth DNN 404. The subclass DNN 502 includes nodes 508-510, which represent subclasses of the category represented by the Mth node 506 of the Nth DNN 404. Continuing with the example described above, the first output node 508 represents the subclass "scarf", the second output node represents the subclass "hat", the third output node represents the subclass "earmuffs", and the fourth output node represents the subclass "sunglasses".
[0046] Since the Nth DNN 404 is a probabilistic model, when it is determined that the object in the portion 306 of the image belongs to the category represented by the Mth node 506 of the Nth DNN 404 (e.g., when the probability corresponding to the Mth node 506 is the highest among all the probabilities of the output nodes 504-506), the portion 306 of the image can be provided to the subclass DNN 502. The text generator system 114 can output words and / or phrases based on the output of the subclass DNN 502.
[0047] Reference now Figure 6 , presents an exemplary depiction of page 128. Page 128 includes image 130 and text area 602 corresponding to image 130. Figure 6 In the example shown, the page 128 also includes a supplemental content item 604 identified by the search module 124 as being related to the object in the image 130. For example, the supplemental content item 604 may be an electronic advertisement related to the object 210 captured in the image 130. The supplemental content item 604 may be or include a hyperlink, wherein when a user selects the hyperlink, a web page related to the object captured in the image may be loaded by a web browser executing on the client computing device 102.
[0048] Reference now Figure 7 , another exemplary depiction of page 128 is shown. Page 128 includes image 130 and area 702 including text. When search module 124 ascertains that a supplemental content item is related to shirt 210 in image 130, search module 124 may transmit the supplemental content item (or a URL pointing to such an item) to computing system 106, which transmits the supplemental content item or URL to interface module 105 executing on client computing device 102. Interface module 105 may cause client computing device 102 to present a selectable graphical icon 704 on image 130 (e.g., on or near shirt 210). When a user selects selectable graphical icon 704, interface module 105 may cause one or more supplemental content items related to the object (shirt 210) in image 130 to be presented on page 128.
[0049] Now turn to Figure 8 , another depiction of page 128 is shown. When graphical icon 704 is selected by a user of client computing device 102, interface module 105 may cause multiple supplemental content items 802-812 to be presented on page 128 (e.g., overlaid on image 130). In one example, each of supplemental content items 802-812 may be selectable and may direct the user to a corresponding page corresponding to the object (shirt) captured in image 130.
[0050] Fig. 9 and 10 Exemplary methods related to identifying supplementary content items related to objects captured in images on web pages are shown. Although these methods are shown and described as a series of actions performed in sequence, it should be understood and appreciated that these methods are not limited by the order of the sequence. For example, some actions may occur in an order different from the order described herein. In addition, an action may occur simultaneously with another action. In addition, in some cases, all actions may not be required to implement the methods described herein.
[0051] In addition, the actions described herein may be computer-executable instructions that may be implemented by one or more processors and / or stored on a computer-readable medium or medium. Computer-executable instructions may include routines, subroutines, programs, execution threads, etc. Further, the results of the actions of the method may be stored in a computer-readable medium, displayed on a display device, etc.
[0052] Now turn to Fig. 9 , a flow chart of an exemplary method 900 is shown that supports identifying supplemental content items based on images. In one example, the method can be performed by computing system 106. Method 900 begins at 902, and at 904 an indication that a web browser executing on a client computing device has loaded a web page is received, where the web page includes an image. For example, the web page can have a script therein, where the script is executable by the client computing device. When the client computing device executes the script, the client computing device transmits a message to computing system 106, where the message includes, for example, a URL of the web page and / or a URL of the image.
[0053] At 906, in response to receipt of the indication received at 904, the image is retrieved (through use of the URL of the image). At 908, a portion of the image including the object is identified, where the object is a type from among a plurality of predefined types. For example, the image can be provided to a DNN including a number of output nodes, where the output nodes correspond to the types of the objects. The output of such a DNN can be a probability distribution of the object types.
[0054] The portion of the image including the object is provided to a text generator system at 910. The text generator system may include a plurality of DNNs, wherein each DNN is configured to receive the portion of the image and output a value representing a value of a corresponding attribute of the object included in the portion of the image.
[0055] At 912, a word sequence describing the object is generated through the use of a text generator system. At 914, the word sequence is transmitted to a search system, wherein the search system identifies supplemental content items related to the object based on the word sequence. The supplemental content items are then caused to be displayed on a client computing device that has loaded the web page, so that the supplemental content items can be displayed together with an image including the object. Method 900 is completed at 916.
[0056] Reference now Fig.10, a flow chart of an exemplary method 1000 is shown that supports identifying supplemental content items related to an object in an image presented on a display of a client computing device. The method 1000 begins at 1002, and at 1004 a portion of an image including an object is received. At 1006, a first word representing a value of a first attribute is output through use of a first DNN. For example, the attribute may be "color" and the value of the attribute may be a value of the color of the object included in the portion of the image.
[0057] At 1008, through the use of the second DNN, a second word representing the value of the second attribute of the object is output. For example, the second attribute can be gender, age, etc. At 1010, through the use of the third DNN, a third word representing the type of the object is output. At 1012, the first word, the second word, and the third word are placed in a semantically correct sequence. Therefore, the first word can be placed in the first position in the sequence, the second word can be placed in the second position in the sequence, and the third word can be placed in the third position in the sequence. At 1014, the sequence is transmitted to a search system, wherein the search system is configured to identify supplementary content items related to the object. Method 1000 is completed at 1016.
[0058] Reference now Fig.11 , shows a high-level diagram of an exemplary computing device 1100 that can be used in accordance with the systems and methods disclosed herein. For example, the computing device 1100 can be used in a system configured to generate text describing an object captured in an image. As another example, the computing device 1100 can be used in a system configured to search for supplemental content items based on text describing an object captured in an image. The computing device 1100 includes at least one processor 1102 that executes instructions stored in a memory 1104. For example, the instruction can be an instruction for implementing the functions described as being performed by one or more of the above components or an instruction for implementing one or more of the above methods. The processor 1102 can access the memory 1104 via a system bus 1106. In addition to storing executable instructions, the memory 1104 can also store supplemental content items, images, web pages, etc.
[0059] The computing device 1100 further includes a data repository 1108 accessible by the processor 1102 via the system bus 1106. The data repository 1108 may include executable instructions, images, etc. The computing device 1100 also includes an input interface 1110 that allows external devices to communicate with the computing device 1100. For example, the input interface 1110 may be used to receive instructions from an external computer device, a user, etc. The computing device 1100 also includes an output interface 1112 that interfaces the computing device 1100 with one or more external devices. For example, the computing device 1100 may display text, images, etc. through the output interface 1112.
[0060] It is contemplated that external devices that communicate with the computing device 1100 via the input interface 1110 and the output interface 1112 can be included in an environment that provides substantially any type of user interface with which a user can interact. Examples of user interface types include graphical user interfaces, natural user interfaces, and the like. For example, a graphical user interface can accept input from a user using (multiple) input devices such as a keyboard, a mouse, a remote control, and provide output on an output device such as a display. In addition, a natural user interface can enable a user to interact with the computing device 1100 in a manner that is not constrained by input devices such as a keyboard, a mouse, a remote control, and the like. In contrast, a natural user interface can rely on speech recognition, touch and stylus recognition, gesture recognition on and near the screen, air gestures, head and eye tracking, sound and speech, vision, touch, gestures, machine intelligence, and the like.
[0061] Furthermore, although shown as a single system, it should be understood that the computing device 1100 may be a distributed system. Thus, for example, several devices may communicate via a network connection and may jointly perform the tasks described as being performed by the computing device 1100.
[0062] The various functions described herein can be implemented in hardware, software or any combination thereof. If implemented in software, these functions can be stored on a computer-readable medium or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include computer-readable storage media. Computer-readable storage media can be any available storage medium that a computer can access. As an example and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Disks and optical disks used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks and blue-ray disks (BDs), wherein disks usually reproduce data magnetically, and optical disks usually reproduce data optically with lasers. In addition, propagated signals are not included in the scope of computer-readable storage media. Computer-readable media also include communication media, including any medium that supports transmitting a computer program from one place to another. For example, a connection can be a communication medium. For example, if the software is transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio and microwave), then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio and microwave) are included in the definition of communication media. Combinations of the above should also be included within the scope of computer-readable media.
[0063] Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, and not limitation, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.
[0064] What has been described above includes examples of one or more embodiments. Of course, it is not possible to describe every possible modification and variation of the above-described apparatus or method for the purpose of describing the above-described aspects, but one of ordinary skill in the art will recognize that many other modifications and permutations of various aspects are possible. Therefore, the described aspects are intended to include all such changes, modifications and variations that fall within the spirit and scope of the appended claims. In addition, to the extent the term "includes" is used in the detailed description or claims, the term is intended to be inclusive in a manner similar to the term "comprising" because "comprising" is interpreted as a transitional word when used in a claim.
Claims
1. A method executed by a processor of a computing system, the method comprising: In response to receiving an indication that the client computing device has loaded an electronic page having an image therein: providing the images to a deep neural network (DNN) configured to identify a plurality of objects of a plurality of types in a plurality of images; identifying, using the DNN, that a portion of the image includes an object of a type from among a plurality of predefined types; providing the portion of the image to a second DNN when the portion of the image is identified by the DNN, wherein the second DNN is configured to, upon receiving multiple portions of multiple images including multiple objects of multiple types, identify multiple values for a first attribute, and further wherein the second DNN outputs a value of the first attribute for the object based on the portion of the image provided to the second DNN; After identifying that the portion of the image includes the object of the type, generating text describing the object, wherein the text includes a first word and a second word, the first word identifying the value of the first attribute for the object, and the second word identifying the value of a second attribute of the object; In response to generating the text, transmitting the text to a search system, wherein the search system identifies supplemental content items related to the object based on the text, and further, wherein a graphical indicator is caused to be presented on the page at the client computing device to indicate that the supplemental content items related to the object have been identified. 2 . The method of claim 1 , wherein the electronic page is a web page loaded by a web browser executing on the client computing device.
3. The method according to claim 1, wherein generating the text further comprises: providing the portion of the image to a third DNN, wherein the third DNN is configured to identify a value for the second attribute upon receiving a plurality of portions of a plurality of images including the plurality of objects of the plurality of types; as well as The second word identifying the value of the second attribute of the object is generated by using the third DNN.
4. The method according to claim 3, wherein generating the text further comprises: providing the portion of the image to a fourth DNN, wherein the fourth DNN is configured to identify types of the plurality of objects upon receiving the plurality of portions of the plurality of images including the plurality of objects of the plurality of types; as well as By using the fourth DNN, a third word identifying the type of the object is generated, wherein the generated text additionally includes the third word.
5. The method according to claim 4, wherein generating the text further comprises: The first word, the second word, and the third word are arranged in a sequence, wherein the first word is first in the sequence, the second word is second in the sequence, and the third word is third in the sequence. 6 . The method of claim 1 , wherein the supplemental content item is caused to be presented on the electronic page with the image in response to the search system identifying the supplemental content item.
7. The method of claim 1, wherein the first attribute is color and the second attribute is a gender associated with the object.
8. The method of claim 1, wherein the first attribute is color and the second attribute is an age associated with the object.
9. A computing system comprising: processor; as well as A memory storing instructions, wherein when the instructions are executed by the processor, the processor performs actions, the actions comprising: receiving an image, wherein the image is on a page that is displayed on a client computing device in network communication with the computing system, and further wherein the image includes an object; extracting a portion of the image from the image, wherein the portion of the image includes the object; providing the portion of the image to a first deep neural network (DNN), wherein the first DNN is configured to output words representing a plurality of values of a first property of a plurality of objects upon receiving a plurality of portions of a plurality of images including a plurality of objects, wherein the first DNN outputs a first word in response to receiving the portion of the image, and further wherein the first word represents a value of the first property of the object; providing the portion of the images to a second DNN, wherein the second DNN is configured to output a word representing a plurality of values of a second attribute of the plurality of objects upon receiving the plurality of portions of the plurality of images including the plurality of objects, wherein the second DNN outputs a second word in response to receiving the portion of the images, and further wherein the second word represents a value of the second attribute of the objects; and transmitting the first term and the second term to a search system, wherein the search system is configured to identify a supplemental content item based on the first term and the second term, wherein the supplemental content item is related to the object in the image, and further wherein a graphical indicator is caused to be presented on the page at the client computing device to indicate that the supplemental content item related to the object has been identified.
10. The computing system of claim 9, wherein the first attribute is color and the second attribute is a gender associated with the object.
11. The computing system of claim 9, wherein the first attribute is color and the second attribute is an age associated with the object.
12. The computing system of claim 9, wherein the portion of the image is extracted by a third DNN, wherein the third DNN is configured to identify a plurality of objects in a plurality of images provided to the third DNN, and further wherein the third DNN is configured to define bounding boxes around the plurality of objects in the plurality of images provided to the third DNN.
13. The computing system of claim 9, the actions further comprising: The portion of the image is provided to a third DNN, wherein the third DNN is configured to output words representing types of the multiple objects when receiving the multiple portions of the multiple images including the multiple objects, wherein the third DNN outputs a third word in response to receiving the portion of the image, wherein the third word represents the type of the object in the image, and further, wherein the third word is transmitted to the search system together with the first word and the second word.
14. The computing system of claim 13, wherein the actions further comprise: The first word, the second word, and the third word are arranged in a sequence before transmitting the first word, the second word, and the third word to the search system, wherein the first word is first in the sequence, the second word is second in the sequence, and the third word is third in the sequence.
15. The computing system of claim 9, wherein the object is an article of clothing worn by a person in the image.
16. A computer-readable storage medium comprising instructions which, when executed by a processor, cause the processor to perform actions comprising: In response to receiving an indication that the client computing device has loaded an electronic page having an image therein, providing the image to a deep neural network (DNN) configured to identify a plurality of objects of a plurality of types in a plurality of images; identifying, using the DNN, that a portion of the image includes an object of a type from among a plurality of predefined types; providing the portion of the image to a second DNN, wherein the second DNN is configured to identify a plurality of values for a first attribute upon receiving a plurality of portions of a plurality of images including a plurality of objects of a plurality of types, and further wherein the second DNN outputs a value of the first attribute for the object; generating text describing the object after identifying that the portion of the image includes the object of the type and further after the second DNN outputs the value for the first attribute, wherein the text includes a first word and a second word, the first word identifying the value for the first attribute of the object and the second word identifying the value of a second attribute of the object, wherein supplemental content items related to the object are identified based on the generated text; transmitting the first term and the second term to a search system, wherein the search system is configured to identify supplemental content items based on a query including the first term and the second term, wherein the supplemental content items are related to the object in the image, and further wherein a graphical indicator is caused to be presented on the electronic page at the client computing device to indicate that the supplemental content items related to the object have been identified by the search system.
17. The computer-readable storage medium of claim 16, wherein generating the text further comprises: providing the portion of the image to a third DNN, wherein the third DNN is configured to identify a value for the second attribute upon receiving a plurality of portions of a plurality of images including the plurality of objects of the plurality of types; as well as The second word identifying the value of the second attribute of the object is generated by using the third DNN.
18. The computer-readable storage medium of claim 17, wherein generating the text further comprises: providing the portion of the image to a fourth DNN, wherein the fourth DNN is configured to, upon receiving the plurality of portions of the plurality of images including the plurality of objects of the plurality of types, identify types of the plurality of objects; as well as A third word identifying the type of the object is generated by using the fourth DNN, wherein the third word is transmitted to the search system together with the first word and the second word.
19. The computer-readable storage medium of claim 18, the actions further comprising: Prior to transmitting the first word, the second word, and the third word to the search system, the first word, the second word, and the third word are arranged in a predefined sequence.
20. The computer-readable storage medium of claim 16, wherein the graphical indicator is a link that, when selected, causes a web browser executing on the client computing device to load a web page in which the object having the value of the first attribute and the value for the second attribute is presented for purchase.
Citation Information
Patent Citations
Presenting advertisements based on web-page interaction
CN102067161A
System and method for enabling image recognition and searching of images
US20080152231A1
