Digital supplemental association and retrieval for visual search

By generating digital supplementary indexes and multi-data structure storage related to visual content, the problem of digital supplementary and visual content in visual search is solved, efficient retrieval and augmented reality content presentation on mobile devices, and improved user experience and information Retrieval efficiency.

CN119025689APending Publication Date: 2024-11-26GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410775019.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-06-21
Filing Date
2019-06-21
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The prior art is difficult to effectively solve the problem of correlation and retrieval between digital supplements and visual content in visual search, especially when providing augmented reality (AR) content or experience on mobile devices, and lacks efficient data structures and indexing mechanisms.

Method used

By generating digital supplementary indexes related to visual content and utilizing database storage of multiple data structure instances, digital supplementation is allowed to be triggered through images, and the retrieval and presentation of responses to visual content queries and digital supplementation are achieved. The system includes client computing devices, search servers and digital supplement servers, uses technologies such as convolutional neural networks to identify supplementary anchors, and provides relevant digital supplements based on metadata.

Benefits of technology

It realizes efficient retrieval and present digital supplements related to visual content on mobile devices, enhances the user experience, and provides more effective information retrieval and augmented reality content presentation capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025689A_ABST
    Figure CN119025689A_ABST
Patent Text Reader

Abstract

The invention discloses digital supplemental association and retrieval for visual search. Systems and methods for identification and retrieval of content for visual search are provided. An example method includes receiving data specifying a digital supplement. The data may identify a digital complement and a complement anchor for associating the digital complement with the visual content. The method may also include generating a data structure instance specifying the digital supplementation and the supplementation anchor, and after generating the data structure instance, enabling the digital supplementation to be triggered by the image based at least on storing the data structure instance in a database including a plurality of other data structure instances. Other data structure instances may each specify a digital supplement and one or more supplement anchors.
Need to check novelty before this filing date? Find Prior Art

Description

Description of the case

[0001] This application is a divisional application of Chinese invention patent application 201980022269.0, with a filing date of June 21, 2019. CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation of and claims priority to U.S. non-provisional patent application No. 16 / 014,520, filed on June 21, 2018, entitled “DIGITAL SUPPLEMENT ASSOCIATION AND RETRIEVAL FOR VISUAL SEARCH,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This application relates to digital supplemental association and retrieval for visual search. Background Art

[0004] Mobile computing devices, such as smartphones, often include cameras. These cameras can be used to capture images of entities in the environment surrounding the computing device. Various types of content or experiences related to those entities can be made available to users via the mobile computing device. Summary of the invention

[0005] The present disclosure describes systems and methods for digital supplement association and retrieval for visual search. For example, the systems and techniques described herein can be used to provide digital supplements, such as augmented reality (AR) content or experiences, in response to visual search. The visual search can be based on an image or an entity identified within an image, for example. The digital supplement can include, for example, providing information or functionality associated with the image.

[0006] One aspect is a computer-implemented method comprising: receiving data specifying a digital supplement, the data identifying the digital supplement and a supplement anchor for associating the digital supplement with visual content. The method also includes generating a data structure instance specifying the digital supplement and the supplement anchor. The method further includes, after generating the data structure instance, enabling triggering of the digital supplement by an image based at least on storing the data structure instance in a database including a plurality of other data structure instances. Each of the other data structure instances specifies the digital supplement and one or more supplement anchors.

[0007] In another aspect, a computing device includes at least one processor and a memory storing instructions. When executed by the at least one processor, the instructions cause the computing device to receive data specifying a digital supplement, the data identifying the digital supplement, a supplement anchor for associating the digital supplement with visual content, and contextual information. The instructions also cause the computing device to generate a data structure instance specifying the digital supplement, the supplement anchor, and the contextual information. The instructions further cause the computing device, after generating the data structure instance, to enable triggering of the digital supplement by an image based at least on storing the data structure instance in a database including a plurality of other data structure instances. Each of the other data structure instances specifies the digital supplement and one or more supplement anchors.

[0008] Yet another aspect is a computer-implemented method that includes receiving a visual content query from a computing device and identifying supplemental anchors based on the visual content query. The method also includes generating a ranked list of digital supplements based on the identified supplemental anchors and transmitting the ranked list to a client computing device.

[0009] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram illustrating a system according to an example implementation.

[0011] Figure 2 is the third-person perspective of the example physical space, where Figure 1 One embodiment of a client computing device is accessing a digital supplement.

[0012] Figure 3 is a diagram of an example method for enabling triggering of digital supplementation according to implementations described herein.

[0013] Figure 4 is a diagram of an example method for enabling triggering of digital supplementation according to implementations described herein.

[0014] Figure 5 is a diagram of an example method of searching and presenting digital supplements according to implementations described herein.

[0015] Figure 6 is a diagram of an example method for image-based recognition and presentation of a digital supplement according to an implementation described herein.

[0016] Figures 7A-7C Is Figure 1 A schematic diagram of a user interface screen displayed while performing a visual content search and displaying a digital supplement on an embodiment of a client computing device.

[0017] Figures 8A-8C Is Figure 1 A schematic diagram of a user interface screen displayed while performing a visual content search and displaying a digital supplement on an embodiment of a client computing device.

[0018] Fig. 9A and 9B Is Figure 1 A schematic diagram of a user interface screen displayed while performing a visual content search and displaying a digital supplement on an embodiment of a client computing device.

[0019] Figures 10A-10C Is Figure 1 A schematic diagram of a user interface screen displayed while performing a visual content search and displaying a digital supplement on an embodiment of a client computing device.

[0020] Figures 11A-11C Is Figure 1 Schematic diagram of a user interface screen displayed when an embodiment of a client computing device performs a search for various visual content within a store.

[0021] Figures 12A-12C is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 11 during various visual content searches.

[0022] Fig.13 is a schematic diagram of an example of a computer device and a mobile computer device that can be used to implement the techniques described herein.

[0023] Reference will now be made in detail to non-limiting examples of the present disclosure, examples of which are illustrated in the accompanying drawings. The examples are described below with reference to the accompanying drawings, wherein the same reference numerals refer to the same elements. When the same reference numerals are shown, the corresponding description will not be repeated and the interested reader can refer to the previously discussed drawings for a description of the same elements. DETAILED DESCRIPTION

[0024] The present disclosure describes technical improvements that simplify the identification and presentation of digital supplements based on visual content. Some implementations of the technology described herein generate an index of digital supplements related to a particular type of visual content and provide those digital supplements in response to a visual content query received from a client computing device. The index can allow a user to access relevant digital supplements provided by network-accessible resources (e.g., web pages) located throughout a global setting. This can provide a functional data structure that allows more efficient retrieval of information.

[0025] For example, a client computing device such as a smart phone can capture an image of a supplemental anchor such as an entity. The client computing device can then transmit a visual content query to a server computing device based on the image to retrieve a digital supplement associated with the identified supplemental anchor. In some implementations, the supplemental anchor is based on the physical environment surrounding the client computing device, and the digital supplement is virtual content that can supplement the user's experience in the physical environment.

[0026] The visual content query may include an image or data determined from the image (e.g., such as an indicator of an identified supplemental anchor). An example of data determined from an image is text extracted from the image using, for example, optical character recognition. Other examples of data extracted from an image include a value read from a barcode, QR code, etc. in the image, an identifier or description of an entity, product, or entity type identified in the image.

[0027] For example, a neural network system such as a convolutional neural network system can be used to identify entities, products, or entity types in an image. The identifier or description of the entity, product, or entity type can include metadata or a reference to a record in a database related to the entity, product, or entity type. Non-limiting examples of entities include buildings, artwork, products, books, posters, photographs, catalogs, signs, documents (e.g., business cards, receipts, coupons, catalogs), people, and body parts.

[0028] Various types of digital supplements associated with supplement anchors may be available. Digital supplements may be provided through web-accessible resources, such as web pages available on the Internet. A way to locate and provide these digital supplements in response to visual content queries is needed. Some embodiments generate and maintain an index of digital supplements associated with an entity for use in responding to visual content queries. For example, the index may be populated by crawling web-accessible resources to determine whether the web-accessible resources include or provide any digital supplements, and to determine supplement anchors associated with those digital supplements.

[0029] For example, the web-accessible resource may include metadata identifying a supplement anchor (e.g., text, code, entity, or entity type) associated with the digital supplement. The web-accessible resource may include the metadata in response to a hypertext transfer protocol (HTTP) request. The metadata may be provided in various formats, such as extensible markup language (XML), JavaScript object notation (JSON), or another format.

[0030] Metadata for a digital supplement may include one or more of the following: a type indicator, an anchor indicator, a name, a description, a content snippet (i.e., an excerpt or preview of a portion of the content), an associated image, a link such as a URL to the digital supplement, and an identifier of an application associated with the digital supplement. The metadata may also include information about the publisher of the digital supplement. For example, the metadata may include one or more of a publisher name, a publisher description, and an image or icon associated with the publisher. In some embodiments, the metadata includes contextual information related to providing the digital supplement. For example, the metadata may also include conditions associated with providing or accessing the digital supplement (e.g., geographic conditions, required applications).

[0031] The identified digital supplement can be added to an index stored in a memory. In at least some embodiments, the associated supplement anchor for the digital supplement is used as a key to the index. The digital supplement can also be associated with various scores. For example, the digital supplement can be associated with a reputation score based on how many other links are found (e.g., when crawling accessible resources) and the reputation of the network-accessible resources that provide these links, which reference the digital supplement or the network-accessible resources associated with the digital supplement. As another example, the digital supplement can be associated with one or more relevance scores corresponding to the relevance of the digital supplement (or the associated network-accessible resource) to a specific anchor. The relevance score can also be associated with a keyword or theme. The relevance score can be determined based on one or more of the content of the digital supplement, the content of the network-accessible resource, the content of the site linked to the network-accessible resource, and the content (e.g., text) of the link to the network-accessible resource.

[0032] Figure 1 is a block diagram illustrating a system 100 according to an example implementation. The system 100 may associate a digital supplement with an entity or entity type, and may retrieve the digital supplement in response to a visual search. A visual search is a search based on visual content. For example, a visual search may be performed based on a visual content query. A visual content query is a query based on an image or other visual content. For example, a visual content query may include an image. In some implementations, a visual content query may include text or data based on an image. For example, the text or data may be generated by identifying one or more entities in an image. Some visual content queries do not include images (e.g., a visual content query may only include data or text generated from an image). In some implementations, the system 100 includes a client computing device 102, a search server 152, and a digital supplement server 172. A network 190 is also shown, through which the client computing device 102, the search server 152, and the digital supplement server 172 may communicate.

[0033] The client computing device 102 may include a processor component 104, a communication module 106, a sensor system 110, and a memory 120. The sensor system 110 may include various sensors, such as a camera component 112, an inertial motion unit (IMU) 114, and a global positioning system (GPS) receiver 116. Implementations of the sensor system 110 may also include other sensors, including, for example, light sensors, audio sensors, image sensors, distance and / or proximity sensors, contact sensors such as capacitive sensors, timers, and / or other sensors and / or different combinations of sensors. In some implementations, the client computing device 102 is a mobile device (e.g., a smart phone).

[0034] The camera component 112 captures images or videos of the physical space surrounding the client computing device 102. The camera component 112 may include one or more cameras. The camera component 112 may also include an infrared camera. Images captured with the camera component 112 may be used to identify supplemental anchors and form visual content queries.

[0035] In some embodiments, images captured with camera assembly 112 may also be used to determine the position and orientation of client computing device 102 within a physical space, such as an interior space, based on a representation of the physical space received from memory 120 or an external computing device. In some embodiments, the representation of the physical space may include visual features of the physical space (e.g., features extracted from an image of the physical space). The representation may also include position determination data associated with those features that may be used by a visual positioning system to determine a position and / or location within the physical space based on one or more images of the physical space. The representation may also include a three-dimensional model of at least some structures within the physical space. In some embodiments, the representation does not include a three-dimensional model of the physical space.

[0036] The IMU 114 can detect motion, movement, and / or acceleration of the client computing device. The IMU 114 can include a variety of different types of sensors, such as, for example, accelerometers, gyroscopes, magnetometers, and other such sensors. The orientation of the client computing device 102 can be detected and tracked based on data provided by the IMU 114 or the GPS receiver 116.

[0037] The GPS receiver 116 can receive signals transmitted by GPS satellites. The signals include the time and location of the satellites. Based on the signals received from several satellites (e.g., at least four), the GPS receiver 116 can determine the global location of the client computing device 102.

[0038] Memory 120 may include application 122, other applications 140, and device location system 142. Other applications 140 include any other applications installed on client computing device 102 or otherwise available for execution on client computing device 102. Application 122 may cause one of other applications 140 to be launched to provide a digital supplement. In some implementations, some digital supplements are only available if other applications 140 include a specific application associated with or required to provide the digital supplement.

[0039] Device positioning system 142 determines the location of client computing device 102. Device positioning system 142 may use sensor system 110 to determine the location and orientation of client computing device 102 within a global or physical space. In some implementations, device positioning system 142 determines the location of client computing device 102 based on, for example, cellular triangulation.

[0040] In some implementations, the client computing device 102 may include a vision positioning system that compares images captured by the camera assembly 112 (or features extracted from those images) to a known arrangement of features within a representation of a physical space to determine a six degree-of-freedom pose (e.g., position and orientation) of the client computing device 102 within the physical space.

[0041] The application 122 may include a supplemental anchor identification engine 124, a digital supplemental retrieval engine 126, a digital supplemental presentation engine 128, and a user interface engine 130. Some implementations of the application 122 may include fewer, more, or other components.

[0042] The supplemental anchor recognition engine 124 recognizes supplemental anchors based on images captured, for example, by the camera assembly 112. In some embodiments, the supplemental anchor recognition engine 124 analyzes the image to recognize text. The text can then be used to recognize an anchor. For example, the text can be mapped to a node in a knowledge graph. For example, the text can be recognized as the name of an entity such as a person, place, product, building, artwork, movie, or other type of entity. In some embodiments, the text can be recognized as a phrase that is generally associated with a specific entity or as a phrase that describes a specific entity. For example, the text can then be recognized as an anchor associated with a specific entity.

[0043] In some implementations, the supplemental anchor recognition engine 124 recognizes one or more codes within the image, such as a barcode, a QR code, or another type of code. The codes may then be mapped to supplemental anchors.

[0044] The supplemental anchor recognition engine 124 may include a machine learning module that can recognize at least some types of entities within an image. For example, the machine learning module may include a neural network system. A neural network is a computational model for machine learning and consists of nodes organized in layers with weighted connections. A training neural network uses training examples, each of which is an input and a desired output, to determine, through a series of iterative rounds, weight values ​​for the connections between layers that increase the likelihood that the neural network will provide the desired output for a given input. In each training round, the weights are adjusted to account for erroneous output values. After training, the neural network can be used to predict outputs based on the inputs provided.

[0045] In some embodiments, the neural network system includes a convolutional neural network (CNN). A convolutional neural network (CNN) is a neural network in which at least one layer of the neural network is a convolutional layer. A convolutional layer is a layer in which the values ​​of the layer are calculated based on applying a kernel function to a subset of the values ​​of the previous layer. Training a neural network can involve adjusting the weights of the kernel function based on training examples. Typically, the same kernel function is used to calculate each value in the convolutional layer. Therefore, far fewer weights must be learned while training a convolutional layer than a fully connected layer in a neural network (e.g., a layer in which each value in a layer is calculated as a weighted combination of independent adjustments to each value in the previous layer). Because there are typically fewer weights in a convolutional layer, training and using a convolutional layer may require less memory, processor cycles, and time than an equivalent fully connected layer.

[0046] After the supplemental anchor recognition engine 124 identifies an entity or entity type in an image, a textual description of the entity or entity type may be generated. Additionally, an entity or entity type may be mapped to a supplemental anchor. In some embodiments, a supplemental anchor is associated with one or more digital supplements.

[0047] In some implementations, the supplemental anchor identification engine 124 determines a confidence score for the identified anchor. A higher confidence score may indicate that the content from the image (e.g., image, extracted text, barcode, QR code) is more likely to be associated with the determined anchor than determining a lower confidence score.

[0048] although Figure 1 While the example shows the supplemental anchor identification engine 124 as a component of the application 122 on the client computing device 102, some embodiments include the supplemental anchor identification engine on the search server 152. For example, the client computing device 102 may send an image captured by the camera component 112 to the search server 152, which may then identify supplemental anchors within the image.

[0049] In some embodiments, the supplemental anchor identification engine 124 identifies potential supplemental anchors. For example, the supplemental anchor identification engine 124 can identify various entities (recognized) within the image. The identifiers of the recognized entities can then be transmitted to the search server 152, which can determine whether any entity is associated with any supplemental anchor. In some embodiments, the search server 152 can use the recognized entity as contextual information even if the recognized entity is not a supplemental anchor.

[0050] The digital supplement retrieval engine 126 retrieves the digital supplements. For example, the digital supplement retrieval engine 126 may retrieve the digital supplements associated with the supplement anchors identified by the supplement anchor identification engine 124. In some implementations, the digital supplement retrieval engine 126 retrieves the digital supplements from the search server 152 or the digital supplement server 172.

[0051] For example, after identifying a supplemental anchor, the digital supplement retrieval engine 126 may retrieve one or more digital supplements associated with the identified supplemental anchor. The digital supplement retrieval engine 126 may generate a visual content query including an image (or an identifier of a supplemental anchor or entity within an image) and transmit the visual content query to the search server 152. The visual content query may also include contextual information, such as the location of the client computing device 102. In some embodiments, data related to the digital supplement, such as a name, image, or description, is retrieved and presented to the user (e.g., via the user interface engine 130). If multiple digital supplements are presented, the user may select one of the digital supplements via a user interface generated by the user interface engine 130.

[0052] The digital supplement presentation engine 128 presents or causes the digital supplement to be presented on the client computing device 102. In some embodiments, the digital supplement presentation engine 128 causes the client computing device to launch one of the other applications 140. In some embodiments, the digital supplement presentation engine 128 causes information or content to be displayed. For example, the digital supplement presentation engine 128 can cause the user interface engine 130 to generate a user interface that includes information or content from the digital supplement to be displayed by the client computing device 102. In some embodiments, the digital supplement presentation engine 128 is triggered by the digital supplement retrieval engine 126 that retrieves the digital supplement. The digital supplement presentation engine 128 can then trigger the display device 108 to display content associated with the digital supplement. In some embodiments, the digital supplement presentation engine 128 causes the digital supplement to be displayed at a different time than when the digital supplement retrieval engine 126 retrieves the digital supplement. For example, the digital supplement can be retrieved in response to a visual content query at a first time, and the digital supplement can be presented at a second time. For example, at a first time (e.g., when a user is browsing a catalog or is in a store), the digital supplement can be retrieved in response to a visual content query based on an image of home decor or furniture from a catalog or store. The digital supplement of the AR content including the home decor or furniture may be presented at a second time (e.g., when the user is in a room in which the home decor or furniture may be placed).

[0053] The user interface engine 130 generates a user interface. The user interface engine 130 may also cause the client computing device 102 to display the generated user interface. The generated user interface may, for example, display information or content from the digital supplement. In some embodiments, the user interface engine 130 generates a user interface that includes a plurality of user-actuatable controls, each of which is associated with a digital supplement. For example, a user may actuate one of the user-actuatable controls (e.g., by touching the control on a touch screen, clicking the control using a mouse or another input device, or otherwise actuating the control).

[0054] The search server 152 is a computing device. The search server 152 can respond to search requests such as visual content queries. The response can include one or more digital supplements that are potentially relevant to the visual content query. In some embodiments, the search server 152 includes a memory 160, a processor component 154, and a communication module 156. The memory 160 can include a content crawler 162, a digital supplement search engine 164, and a digital supplement data store 166.

[0055] The content crawler 162 can crawl network-accessible resources to identify digital supplements. For example, the content crawler 162 can access web pages accessible via the Internet, such as web pages provided by the digital supplement server 172. Crawling the network-accessible resources can include requesting resources from a web server and parsing at least a portion of the resources. The digital supplement can be identified based on metadata provided by the network-accessible resources, such as XML or JSON data that provides information about the digital supplement. In some embodiments, the crawler identifies the network-accessible resources based on extracting links from previously crawled network-accessible resources. The content crawler 162 can also identify the network-accessible resources to be crawled based on receiving input submitted by a user. For example, a user can submit a URL (or other information) to a network-accessible resource including a digital supplement through a web form or an application programming interface (API). In some embodiments, the content crawler 162 generates an index of the identified digital supplements. The content crawler 162 can also generate a score associated with the digital supplement, such as a relevance score or a popularity (reputation) score.

[0056] The digital supplement search engine 164 receives a search query and generates a response that may include one or more potentially relevant digital supplements. For example, the digital supplement search engine 164 may receive a visual content query from the client computing device 102. The visual content query may include an image. The digital supplement search engine 164 may identify a supplement anchor in the image, and based on the identified supplement anchor, identify relevant or potentially relevant digital supplements. The digital supplement search engine 164 may transmit a response including a digital supplement or information that can be used to access the digital supplement to the client computing device 102. In some embodiments, the digital supplement search engine 164 may return information associated with multiple digital supplements. For example, a list of digital supplements may be included in the response to the query. The list may be sorted based on relevance to the supplement anchor, popularity, or other properties of the digital supplement.

[0057] The visual content query may include, for example, an image captured by the camera assembly 112 or text or other data associated with an image captured by the camera assembly 112. The visual content query may also include other information, such as a location of the client computing device 102 or an identifier of a user of the client computing device 102. In some implementations, the search server 152 may determine a possible location of the client computing device 102 based on the user identifier (e.g., if the user has enabled location services on the client computing device 102 that associate information about the user's location with a user account).

[0058] The digital supplement data store 166 stores information about the digital supplements. In some implementations, the digital supplement data store 166 includes an index of the digital supplements. For example, the index may be generated by the content crawler 162. The digital supplement search engine 164 may use the index to respond to search queries.

[0059] The digital supplement server 172 is a computing device. The digital supplement server 172 provides the digital supplement. In some embodiments, the digital supplement server 172 includes a memory 180, a processor component 174, and a communication module 176. The memory 180 may include the digital supplement 182 and metadata 184. In some embodiments, the memory 180 may also include other network-accessible resources, such as web pages that are not necessarily digital supplements. For example, the memory 180 may store web pages that include metadata to provide details about one or more digital supplements and how to access those digital supplements. In addition, the memory 180 may include a resource service engine such as a web server that, for example, utilizes network-accessible resources such as web pages and digital supplements to respond to requests such as HTTP requests.

[0060] The digital supplement 182 is any type of content that can be provided as a supplement to things in the physical environment around the user. The digital supplement 182 can also include any type of content that can supplement the stored image (e.g., an image of a previous physical environment around the user). For example, the digital supplement can be associated with a supplement anchor, such as an image, an object or product or location identified in the image. The digital supplement 182 can include one or more images, audio content, text data, videos, games, data files, applications, or structured text documents. Examples of structured text documents include Hypertext Markup Language (HTML) documents, XML documents, and other types of structured text documents.

[0061] The digital supplement 182 may cause an application to be launched and may define parameters for the application. The digital supplement 182 may also cause a request to be transmitted to a server (e.g., an HTTP request) and may define parameters for the request. In some implementations, the digital supplement 182 is initiated as a workflow for completing an activity, such as a workflow for completing a purchase. For example, the digital supplement 182 may transmit an HTTP request to a server to add a particular product to a user's shopping cart, add a coupon code, and retrieve a purchase confirmation page.

[0062] Metadata 184 is data describing a digital supplement. Metadata 184 may describe one or more digital supplements provided by the digital supplement server 172 or provided elsewhere. Metadata 184 for a digital supplement may include one or more of the following: a type indicator, an anchor indicator, a name, a description, a preview snippet or excerpt, an associated image, such as a link to the URL of the digital supplement, and an identifier of an application associated with the digital supplement. Metadata may also include information about the publisher of the digital supplement, such as the publisher name, the publisher description, and an image or icon associated with the publisher. In some embodiments, metadata also includes contextual information about the digital supplement or contextual information that must be met to provide the digital supplement. For example, metadata may include conditions that must be met to access the digital supplement (e.g., geographic conditions, client computing device requirements, required applications). Exemplary contextual information includes location, entities identified within an image, or multiple entities identified within an image (e.g., some digital supplements may require a combination of entities to be identified within an image). The identified entity may be a supplement anchor. In some embodiments, the identified entity is not a supplement anchor, but provides contextual information. The metadata 184 may also include a supplement anchor (eg, text, code, entity, or entity type) associated with the digital supplement.

[0063] The metadata 184 can be stored in a variety of formats. In some implementations, the metadata 184 is stored in a database. The metadata 184 can also be stored as an XML file, a JSON file, or another format file. In some implementations, the digital supplement server 172 retrieves the metadata 184 from the database and formats the metadata 184 as XML, JSON, or otherwise to provide a response to a request from a client or search server 152. For example, the search server 152 can access the metadata 184 to generate data stored in the digital supplement data store 166 and used to respond to a search request from a client computing device 102.

[0064] The communication module 106 includes one or more devices for communicating with other computing devices, such as the search server 152 or the digital supplement server 172. The communication module 106 may communicate via a wireless or wired network, such as the network 190. The communication module 156 of the search server 152 and the communication module 176 of the digital supplement server 172 may be similar to the communication module 106.

[0065] The display device 108 may, for example, include an LCD (liquid crystal display) screen, an LED (light emitting diode) screen, an OLED (organic light emitting diode) screen, a touch screen, or any other screen or display for displaying images or information to a user. In some embodiments, the display device 108 includes a light projector arranged to project light onto a portion of the user's eye.

[0066] Memory 120 may include one or more non-transitory computer-readable storage media. Memory 120 may store instructions and data that may be used by client computing device 102 to implement the techniques described herein, such as generating a visual content query based on a captured image, transmitting the visual content query, receiving a response to the visual content query, and presenting a digital supplement identified in response to the visual content query. Memory 160 of search server 152 and memory 180 of digital supplement server 172 may be similar to memory 120 and may store data instructions that may be used to implement the techniques of search server 152 and digital supplement server 172, respectively.

[0067] The processor component 104 includes one or more devices that are capable of executing instructions, such as instructions stored by the memory 120, to perform various tasks associated with associating and retrieving digital supplements for visual search. For example, the processor component 104 may include a central processing unit (CPU) and / or a graphics processor unit (GPU). For example, if a GPU is present, some image / video rendering tasks (such as generating and displaying portions of a user interface or displaying a digital supplement) may be offloaded from the CPU to the GPU. In some embodiments, some image recognition tasks may also be offloaded from the CPU to the GPU.

[0068] although Figure 1 Not shown, but some embodiments include a head mounted display device (HMD). The HMD can be a separate device from the client computing device 102, or the client computing device 102 can include the HMD. In some embodiments, the client computing device 102 communicates with the HMD via a cable. For example, the client computing device 102 can transmit video signals and / or audio signals to the HMD for display to the user, and the HMD can transmit motion, position, and / or orientation information to the client computing device 102.

[0069] The client computing device 102 may also include various user input components (not shown), such as a controller that communicates with the client computing device 102 using a wireless communication protocol. In some implementations, the client computing device 102 may communicate with an HMD (not shown) via a wired connection (e.g., a Universal Serial Bus (USB) cable) or via a wireless communication protocol (e.g., any WiFi protocol, any Bluetooth protocol, ZigBee, etc.). In some implementations, the client computing device 102 is a component of the HMD and may be contained within a housing of the HMD.

[0070] Network 190 may be the Internet, a local area network (LAN), a wireless local area network (WLAN), and / or any other network. Client computing device 102 may, for example, receive audio / video signals via the network, which may be provided as part of a digital supplement in the illustrative example implementation.

[0071] Figure 2 2 is a third person perspective of an example physical space 200 in which an embodiment of a client computing device 102 is accessing a digital supplement. In this example, the physical space 200 includes an object 222. Here, the object 222 is a piece of art on a wall of the physical space 200. The object 222 is contained within the field of view 204 of the camera assembly 112 of the client computing device 102.

[0072] Also shown is an example user interface screen 206. The user interface screen 206 may be generated, for example, by the user interface engine 130 of the client computing device 102. The user interface screen 206 includes an image display panel 208 and a digital supplement selection panel 210. The image display panel 208 shows an image. For example, the image display panel 208 may show an image corresponding to a live feed from the camera assembly 112 of the client computing device 102. In some implementations, the image display panel 208 shows a previously captured image or information that has been retrieved from the memory 120 of the client computing device 102.

[0073] In some implementations, the user interface screen 206 is displayed to the user on a display device of the client computing device 102. In some implementations, the user interface screen 206 can be overlaid on an image of the physical space (or a video feed captured by a camera of the computing device). In addition, the user interface screen 206 can be displayed as AR content in the user's field of view using an HMD worn by the user.

[0074] The image display panel 208 may also include annotations or user interface elements that may be related to the image. For example, the image display panel 208 may include an indicator that an object (e.g., object 222) in the image has been identified as a supplemental anchor. The indicator may include a user-actuable control to access or view information about the digital supplement associated with the identified supplemental anchor. In some cases, the image displayed in the image display panel 208 may include multiple objects that are identified as supplemental anchors, and the image display panel 208 may include multiple annotations overlaying the image to identify those supplemental anchors.

[0075] The supplemental anchors may be identified by a supplemental anchor recognition engine of the client computing device 102. In some implementations, the supplemental anchors are identified by transmitting the image to the search server 152. The search server 152 may then analyze the image and identify the supplemental anchors in the image. In some implementations, the search server 152 may transmit one or more of the location (e.g., image coordinates) or size of any identified object associated with the supplemental anchor to the client computing device 102. The client computing device 102 may then update the user interface screen to show annotations identifying the supplemental anchors (or associated objects) in the image. In some implementations, the client computing device 102 may track the locations of the supplemental anchors (or associated objects) in the video stream (e.g., images captured in a sequential order) captured by the camera assembly 112 (e.g., the supplemental anchor recognition engine 124 may track the supplemental anchors identified by the search server 152).

[0076] The digital supplement selection panel 210 allows the user to select the digital supplement for presentation. For example, the digital supplement selection panel 210 may include a menu including user-actuated controls associated with the digital supplement. In this example, the digital supplement selection panel 210 includes user-actuated controls 212 and user-actuated controls 214, each of which includes information about the associated digital supplement. For example, the user-actuated controls may display one or more of a name (or title), a brief description, and an image associated with the digital supplement that may be received from the search server 152. After actuation of the user-actuated controls 212 or the user-actuated controls 214, the content of the associated digital supplement may be presented to the user. Presenting the digital supplement to the user may include causing the client computing device 102 to display a user interface screen including images, videos, text, other content, or a combination thereof from the digital supplement. In some embodiments, the digital supplement content is displayed as an overlay on an image or camera feed on the image display panel 208. The digital supplement content may be a three-dimensional augmented reality content.

[0077] In some embodiments, presenting the digital supplement includes activating an application (e.g., one of the other applications 140) installed on the client computing device 102. Presenting the digital supplement may also include transmitting a request to a URL associated with the digital supplement. The request may include parameters associated with the digital supplement, such as an identifier of a product or object identified within the image. In some embodiments, an image (or other content) from a visual content query is passed the parameters as well as the request. The image may also be provided via an API associated with the digital supplement server 172. In some embodiments, the client computing device 102 transmits the image to the digital supplement server 172. In some embodiments, the search server 152 may transmit the image to the digital supplement server 172. For example, in response to a user selecting a digital supplement, the client computing device 102 may transmit an indicator of the selection to the search server 152, and the search server 152 may then transmit the image to the corresponding digital supplement server. The client computing device 102 may also transmit a URL to a location on the search server 152 that the digital supplement server 172 may use to access the image. Advantageously, these embodiments may reduce the amount of data that the client computing device needs to transmit.

[0078] The digital supplement associated with user-actuatable control 212 may cause information about object 222, such as information from a museum, to be displayed. The digital supplement associated with user-actuatable control 214 may cause information related to a museum tour to be displayed. For example, presentation of the digital supplement may cause a stop on a museum tour to be marked as completed, and information about the next stop to be displayed.

[0079] Figure 3 is a diagram of an example method 300 for enabling triggering of digital supplements according to implementations described herein. The method 300 may be performed, for example, by the content crawler 162 of the search server 152 to allow a user to access digital supplements based on a visual content query.

[0080] At operation 302, data specifying a digital supplement is received. The data may identify a digital supplement and the circumstances under which the digital supplement should be provided. The data specifying the digital supplement may be received in a variety of ways. For example, the data specifying the digital supplement may be received from a network accessible resource such as a web page that includes metadata about the digital supplement. The data specifying the digital supplement may also be received via an API or form provided by, for example, the search server 152. The data specifying the digital supplement may also be received from a memory location or data store.

[0081] The data about the digital supplement may include access data that can be used by the client computing device to access the digital supplement. For example, the access data may include the URL of the digital supplement and parameters passed to the URL. The access data may also include an application identifier and parameters of the application. The data about the digital supplement may also include descriptive data about the digital supplement. The client computing device may use the descriptive data to present information about the digital supplement to the user (e.g., on a menu where the user can select the digital supplement). The descriptive data may include, for example, a name (or title, description, publisher name, and image). The data about the digital supplement may also include an identifier of a supplement anchor.

[0082] At operation 304, a data structure instance based on the received data is generated. The data structure may be, for example, a record in a database. The database may be a relational database, and the data structure instance may be linked (eg, via a foreign key) to one or more records associated with the supplemental anchor.

[0083] At operation 306, after the data structure instance is generated, the digital supplement is enabled to be retrieved by a visual content query. For example, a database field associated with the data structure instance can be set to active so that the digital supplement search engine 164 can access and return the associated digital supplement. In some embodiments, the triggering of the digital supplement can include saving or recording a database record. In some embodiments, enabling the retrieval of the digital supplement includes enabling the digital supplement to be triggered by the client computing device. For example, after the instance is generated, the digital supplement can be returned to the client computing device in response to a search and activated or presented by the client computing device.

[0084] Figure 4 is a diagram of an example method 400 for enabling triggering of digital supplements according to implementations described herein. The method 400 may be performed, for example, by the content crawler 162 of the search server 152 to allow a user to access digital supplements based on a visual content query.

[0085] At operation 402, web accessible resources are analyzed. In some embodiments, the web accessible resources are web pages served by the digital supplement server 172, for example. In some embodiments, a set of web accessible resources are analyzed. The set of web accessible resources can be generated based on a submission via a form or API. In some embodiments, the set of web accessible resources can be generated by crawling other web accessible resources to identify URLs. The crawling process can be performed recursively.

[0086] At operation 404, metadata associated with the digital supplement within the network-accessible resource is identified. In some implementations, the network-accessible resource may include an indicator of the metadata associated with the digital supplement. For example, the network-accessible resource may include a tag that identifies a portion of the network-accessible resource that includes the metadata. The tag may be an XML tag with a particular type or attribute. The tag may be an HTML tag, such as a script tag, that includes a JSON data structure that includes the metadata.

[0087] At operation 406 , a digital supplemental data structure instance based on the metadata is generated. Operation 406 may be similar to operation 304 .

[0088] At operation 408, a visual content query is received. The visual content query may be sent, for example, by a client computing device such as client computing device 102. In some embodiments, the visual content query includes an image. The visual content query may also include textual data describing the image. For example, the textual data may include an identifier of a supplemental anchor within an image captured by a camera component of the client computing device. In some embodiments, the visual content query also includes other information, such as a location of the client computing device or an identifier of a user account associated with the client computing device.

[0089] At operation 410, a plurality of digital supplemental data structure instances are identified based on a visual content query. In some embodiments, supplemental anchors are identified within an image provided in the visual content query. The supplemental anchors may then be used to query an index or database for relevant digital supplements. In some embodiments, other data provided with the query may also be used to identify the digital supplements, such as the location of the client computing device or information associated with a user account. In some embodiments, a plurality of supplemental anchors are used to identify relevant supplemental anchors.

[0090] At operation 412, a ranking of the plurality of digital supplemental data structure instances is determined. The ranking may be based on various scores associated with the digital supplement or the relevance of the digital supplement to the visual content query. In some embodiments, the relevance score corresponding to the relevance of the digital supplement to the visual content query is used to rank the plurality of digital supplemental data structure instances.

[0091] The relevance score may be determined from one or more of a number of factors, such as the content of the digital supplement, the content of a web-accessible resource linking to the digital supplement (or a web-accessible resource associated with the digital supplement), link text or content near links to the digital supplement information on other web-accessible resources.

[0092] The score may also be based on a popularity metric. A reputation metric is an example of a popularity metric. The reputation metric may be based on a combination of how many other web resources link to the digital supplemental content and the reputation scores of these other web accessible resources. In some embodiments, the popularity score may be based on how often the digital resource is or has been selected. In some embodiments, the popularity score may correspond to how often the digital resource is selected for a visual content query.

[0093] The score may be determined or may be retrieved from a data store or API. In some embodiments, the API is accessed to retrieve the score of the digital supplement. For example, the score may be retrieved from a search engine that has determined the relevance and / or popularity of the digital resource with respect to a search term based on the supplement anchor.

[0094] The plurality of digital supplemental data structures may also be ordered based on a frequency of use by a particular user (eg, a user of a client computing device) or recency of use by a particular user. In some implementations, the plurality of digital supplemental data structures are randomly ordered.

[0095] At operation 414, the visual content query is responded to based on the multiple digital supplemental data structure instances. For example, information associated with the multiple digital supplemental data structure instances can be transmitted to the client computing device in the order determined at operation 412. In some embodiments, the information includes descriptive data that can be displayed in a menu or another type of user interface configured to receive user selections of digital supplements. The information can also include access data that can be used by the client computing device to access or present the digital supplement.

[0096] Figure 5 is a diagram of an example method 500 for searching and presenting digital supplements according to implementations described herein. The method 500 may be performed, for example, by an application 122 of a client computing device 102 to identify and access digital supplements based on a visual content query.

[0097] At operation 502, an image-based visual content query is transmitted to a server computing device (e.g., search server 152). For example, an image may be captured by camera component 112 of client computing device 102. The image may also be a stored image, such as an image previously captured by camera component 112. In some embodiments, the visual content query contains only the image. In some embodiments, the visual content query includes additional information. For example, the visual content query may include information such as the location of client computing device 102 or an identifier of an account associated with a user of client computing device 102. Application 122 may also identify anchors in the image (e.g., using supplemental anchor recognition engine 124). The visual content query may include an identifier (e.g., text, number, or other type of identifier) ​​of the identified anchor. In at least some embodiments, the visual content query does not include an image.

[0098] In some embodiments, transmitting the visual content query to the server includes calling an API. In some embodiments, transmitting the visual content query to the server includes calling an API provided by the server. In some embodiments, transmitting the visual content query to the server includes submitting a form using the HTTP protocol (e.g., submitting a GET or POST request).

[0099] At operation 504, a response to the visual content query identifying the digital supplement is received. The response can be received from the search server 152 via the network 190. The response may include one or more digital supplements identified by the search server 152 based on the visual content query. For example, the response may include an array of data associated with the digital supplement. In some embodiments, the data associated with the digital supplement may include descriptive data that may be used to present the digital supplement options for user selection. For example, the descriptive data may include a name, a short description, a publisher name, and an image. The data may also include access data, such as a URL and parameters or an application name and related parameters included with the request via the URL. The data may also include the position, coordinates, or size of the supplementary anchor in the image transmitted with the visual content query (e.g., if the supplementary anchor is identified by the search server 152).

[0100] At operation 506, a user interface screen including information associated with the digital supplement is displayed. In some embodiments, the user interface screen includes annotations (e.g., based on the provided coordinates) overlaying the identified supplement anchor. The annotations can provide information about objects in the image associated with the identified supplement anchor. The annotations can include user-actuable controls that can be actuated to present or activate the digital supplement. The user interface screen can also include a digital supplement selection panel that can be used to select from a plurality of digital supplements identified in the response received at operation 504. In some embodiments, the user interface screen can be generated by a web browser that opens a URL specified by the digital supplement. The user interface screen can also be generated by another application that is launched to provide the digital supplement.

[0101] Figure 6 is a diagram of an example method 600 for identifying and presenting a digital supplement based on an image, according to implementations described herein. The method 600 may be performed, for example, by an application 122 of a client computing device 102 to identify and access a digital supplement based on a visual content query.

[0102] At operation 602, an image is captured. For example, the image may be captured by the camera component 112 of the client computing device 102. In some implementations, a sequence of images (ie, a video) may be captured by the camera component 112.

[0103] At operation 604, the image-based visual content query is transmitted to a server computing device, such as search server 152. Operation 604 can be similar to operation 502. In embodiments where a sequence of images is captured, the visual content query can include multiple images or a sequence of images. In some embodiments, the sequence of images can be streamed to the server computing device.

[0104] At operation 606, a response to the visual content query is received that identifies a plurality of digital supplements. Operation 606 may be similar to operation 504 previously described.

[0105] At operation 608, a user interface screen is displayed, which includes user-actuated controls to select a digital supplement from a plurality of digital supplements. For example, a digital supplement selection panel may be displayed. The digital supplement selection panel may include a plurality of user-actuated controls, each of which is associated with one of the plurality of digital supplements identified in the response. The digital supplement selection panel may arrange the user-actuated controls based on the sorting or ranking of the digital supplements provided by the server computing device. The digital supplement selection panel may arrange the user-actuated controls vertically, horizontally or otherwise. The user-actuated controls may be associated with the associated digital supplement or include information about the associated digital supplement, which the user may consider when deciding whether to select the digital supplement. For example, the information displayed may include one or more of the name, description, image, and publisher name of the digital supplement.

[0106] At operation 610, user input is received to select a digital supplement. The user input may be a click using a mouse or other device. The user input may also be a touch input from a stylus or finger. Another example of user input is near touch input (e.g., holding a finger or pointing device close to a touch screen). In some embodiments, the user input may also include gestures, head movements, eye movements, or voice input.

[0107] At operation 612, information is provided to a resource associated with the selected digital supplement. For example, information about the user of the client computing device may be transmitted to a server providing the digital supplement (if permission to provide the information has been provided). The information may also be provided to an application providing the digital supplement. Various types of information may be provided. For example, the information may include user information, such as a user name, user preferences, or location.

[0108] The information may also include information related to the visual content query, such as an image or an image sequence. The information may also include identifiers and / or locations of one or more supplementary anchors in the image. The information may be used to provide digital supplements to the user. For example, the size and positioning of the digitally supplemented AR content may be adjusted based on the image.

[0109] The information may be transmitted by the client computing device 102 directly to a resource associated with the digital supplement (e.g., a digital supplement server 172). In some implementations, the information is provided to the resource associated with the digital supplement by the search server 152 (e.g., so that the client computing device does not need to transmit as much data). In at least some of these implementations, the client computing device 102 may transmit selection information identifying the selected digital supplement to the search server 152. After receiving the selection and verifying that the user has authorized the information to be shared, the search server 152 may then transmit the information to the resource providing the digital supplement. The client computing device 102 may also prompt the user for permission to share the information. In some implementations, the search server 152 may determine the information to be transmitted to the resource based on the digital supplement data structure instance (which may be based on metadata associated with the digital supplement).

[0110] At operation 614 , the user interface is updated based on the selected digital supplement. Operation 614 may be similar to operation 506 .

[0111] Figures 7A-7C is a schematic diagram of a user interface screen displayed by an embodiment of a client computing device 102 to conduct a visual content search and display a digital supplement. Fig. 7A , user interface screen 700a is shown. User interface screen 700a includes image display panel 708 and information panel 730. In this example, image display panel 708 is displaying an image of a shelf full of wine bottles (e.g., what you might find in a store). Image display panel 708 also includes indicator 740 and indicator 742. Each of these indicators indicates that the wine bottle shown in the image below the indicator has been identified as a supplemental anchor (e.g., in this case, an identified product). Indicator 740 and indicator 742 are examples of user-actuable controls. Within information panel 730, an instruction is provided as "Tap on what you're interested in."

[0112] exist Figure 7B , user interface screen 700b is shown after the user has actuated indicator 740. After actuation, annotation 744 from the digital supplement is displayed. Annotation 744 includes information about the rating of the wine, which can help the user select a bottle of wine to purchase.

[0113] exist Figure 7C In FIG. 7 , another user interface screen 700c is shown after the user has actuated indicator 740. User interface screen 700c may be shown instead of or in addition to user interface screen 700b (e.g., after actuation of annotation 744, or if the user is in Figure 7B730 in the information panel). Figure 7C , an expanded information panel 732 is shown. The expanded information panel 732 is Fig. 7A and Figure 7B The information panel 730 in FIG. 7 occupies more of the user interface screen 700 c .

[0114] The extended information panel 732 includes a digital supplement selection panel 710 and a digital supplement content display panel 734. The digital supplement selection panel 710 includes a user-actuable control 712, a user-actuable control 714, and a user-actuable control 716 (only partially visible). In some embodiments, when a user swipes on the digital supplement selection panel 710, additional user-actuable controls may be displayed. The user-actuable controls of the digital supplement selection panel 710 may be arranged in a ranked order. The user-actuable control 712 is associated with a digital supplement for meal pairing. When the user-actuable control 712 is actuated, a digital supplement displaying food and meal pairing information for the selected wine may be displayed. The user-actuable control 714 is associated with a digital supplement for saving photos. When actuated, an application for saving photos may be activated and provided with an image. Other information may be saved with photos such as identified supplement anchors.

[0115] The digital supplemental content display panel 734 may display content from the digital supplement. The digital supplemental content display panel 734 may display a default digital supplement or a top-ranked digital supplement associated with the identified supplemental anchor. In this example, the digital supplemental content display panel 734 includes product information about the product associated with the selected supplemental anchor. In this case, the wine name, rating, origin, image, and reviews are provided.

[0116] Figures 8A-8C is a schematic diagram of a user interface screen displayed by an embodiment of client computing device 102 to conduct a visual content search and display a digital supplement. In this example, the visual content search is based on an image of a receipt.

[0117] exist Fig. 8A , user interface screen 800a is shown. User interface screen 800a includes image display panel 808 and information panel 830. In this example, image display panel 808 is displaying an image of a receipt from a restaurant. Image display panel 808 also includes indicator 840, indicator 842, annotation 844, and highlight overlay 846. In this case, indicator 840 is associated with the receipt as a document, and indicator 842 is associated with a specific restaurant named on the receipt. The identified receipt document and the identified restaurant name are both examples of supplemental anchors.

[0118] Annotation 844 is associated with a digital supplement that provides a tip calculator. In this example, an exemplary tip calculation is included on annotation 844 and is overlaid at an appropriate location on image display panel 808. In some embodiments, the digital supplement can be selected and displayed by default when the appropriate supplement anchor is identified. A highlight overlay 846 is overlaid on a portion of the receipt document that includes information used by the tip calculator digital supplement.

[0119] In this example, the items displayed in information panel 830 are related to a receipt as a document, just as if indicator 840 had been actuated. In some embodiments, the identified supplemental anchors are ranked based on possible relevance or user interest, which is based on, for example, the user's past actions, other user actions for similar images, confidence scores for the supplemental anchors, or the location or size of the portion of the image to which the supplemental anchors are related. Then, in at least some embodiments, information panel 830 can display items related to the highest ranked supplemental anchors. If, instead, indicator 842 is actuated, information panel 830 might include items about a particular restaurant.

[0120] Here, information panel 830 includes digital supplement selection panel 810. The digital supplement panel includes user-actuable controls 812, user-actuable controls 814, and user-actuable controls 816. In this example, user-actuable controls 812 are associated with a tip calculator digital supplement, user-actuable controls 814 are associated with a split bill digital supplement, and user-actuable controls 816 are associated with an expense report digital supplement. For example, upon actuation of user-actuable controls 812, user interface controls for adjusting parameters of a tip calculator (e.g., to adjust a percentage) may be displayed.

[0121] exist Figure 8B 8, a user interface screen 800b is shown after the user has actuated the user-actuable control 814. After actuation, an expanded information panel 832 is shown that includes items to help the user calculate how to split the bill. For example, the number of people who will split the bill can be entered to determine the amount each person should pay.

[0122] exist Figure 8C8, user interface screen 800c is shown after the user has actuated user-actuable control 816. After actuation, an expanded information panel 834 is shown that includes items to help the user store the receipt to an expense report. For example, the user can select an expense report that should be associated with the receipt (e.g., "Sydney trip 2018"). Once the expense report is selected, an image of the receipt can be uploaded to an expense report submission or management system. In some embodiments, the entire image shown on the image display panel 808 is uploaded. In some embodiments, a portion of the image is uploaded (e.g., the image is cropped to include only the receipt).

[0123] Fig. 9A and 9B is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device 102 for performing a visual content search and displaying a digital supplement. In this example, the visual content search is based on facial images.

[0124] exist Fig. 9A , user interface screen 900a is shown. User interface screen 900a includes an image display panel 908 and an information panel 930. In this example, image display panel 908 is displaying an image of a face. Here, a face is an example of a supplemental anchor. Information panel 930 includes a user-activatable control 912 for a digital supplement identified for a supplemental anchor in the image (i.e., a face). User-actuatable control 912 is associated with a digital supplement for trying on glasses.

[0125] exist Fig. 9B , user interface screen 900b is shown after the user has actuated the user-actuatable control 912. After actuation, an expanded information panel 932 is shown, which includes items to help the user visually try on the glasses on the face in the image. Here, multiple styles of glasses are displayed, and the user can select a pair to try on. Upon selecting a pair of glasses, AR content 960 is overlaid on the image display panel 908. Here, the AR content 960 corresponds to the selected glasses and is sized to match the face in the image. In some embodiments, when a digital supplement for trying on glasses is selected, the image shown in the image display panel 908 is transmitted to a server that provides the digital supplement so that the image can be analyzed to determine where and how to locate and resize the AR content 960 or recommend trying on glasses.

[0126] Figures 10A-10C is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device 102 for performing a visual content search and displaying a digital supplement. In this example, the visual content search is based on images of furniture in a catalog.

[0127] exist Fig. 10A, user interface screen 1000a is shown. User interface screen 1000a includes image display panel 1008. In this example, image display panel 1008 is displaying an image of a portion of a page of a furniture catalog. Image display panel also includes indicator 1040, indicator 1042, and indicator 1044. In this example, indicator 1040 is associated with a bed, indicator 1042 is associated with a decorative item, and indicator 1044 is associated with a rug. Images of beds, decorative items, and rugs in the catalog are examples of supplemental anchors.

[0128] exist Fig. 10B , user interface screen 1000b is shown after the user has selected indicator 1040 (e.g., by touching the screen at or near where indicator 1040 is displayed). User interface screen 1000b includes digital supplement selection panel 1010 and information panel 1030. Information panel 1030 includes information about the supplement anchor associated with the selected indicator (e.g., product name, description, and image).

[0129] Digital supplement selection panel 1010 includes user-actuable controls 1012 and user-actuable controls 1014. User-actuable controls 1012 are associated with a digital supplement that provides an at-home view. User-actuable controls 1014 are associated with another digital supplement (e.g., a digital supplement for posting to a social media site).

[0130] exist Fig. 10C , user interface screen 1000c is shown after actuation of a user-actuatable control 1012. User interface screen 1000c includes image display panel 1008, digital supplement selection panel 1010, and reduced information panel 1032. Reduced information panel 1032 may include a user-actuatable control that, when actuated, may cause an information panel to pop up and be displayed.

[0131] Here, the image display panel 1008 now displays an image of the room and includes AR content 1060. The AR content 1060 includes a 3D model of a bed associated with an indicator 1040 overlaid on the image panel. The user may be able to adjust the position of the AR content 1060 within the room to see how the bed fits in the room. In some embodiments, when a digital supplement is selected for the at-home view, the image shown in the image display panel 1008 is transmitted to a server that provides the digital supplement so that the image can be analyzed to determine where and how to position and resize the AR content 1060. In some embodiments, the AR content 1060 can be provided at a later time than the visual content query.

[0132] Figures 11A-11Cis a schematic diagram of a user interface screen displayed by an embodiment of the client computing device 102 for conducting various visual content searches within a store. In this example, the visual content search is based on images of products captured within the store.

[0133] exist Fig.11A , a user interface screen 1100a is shown. The user interface screen 1100a includes an image display panel 1108 and an information panel 1130. In this example, the image display panel 1108 is displaying an image captured within a store. The image display panel 1108 also includes an indicator 1140 associated with a vase. The vase displayed on the image display panel 1108 is an example of a supplement anchor. The information panel 1130 is displaying a digital supplement that includes product information about the vase and functionality to purchase the vase. The digital supplement may, for example, include a workflow to initiate a vase purchase. In this example, a digital supplement is identified based on the image content and the location of the client computing device, so that a digital supplement published by (or associated with) a store where the image was captured when the client computing device was in the store can be identified as a high-ranking result and provided to the visual content query. In some embodiments, if the location of the client computing device is changed, a different digital supplement will be provided for the same image.

[0134] exist Fig. 11B , user interface screen 1100b is shown. User interface screen 1100b includes image display panel 1108 and information panel 1130. In this example, image display panel 1108 is displaying another image captured within a store. Image display panel 1108 also includes indicator 1142 associated with a rug. The rug displayed on image display panel 1108 is an example of a supplement anchor. Information panel 1130 is displaying a digital supplement that includes product information about the rug and functionality to select a size and purchase the rug. As shown in Fig.11A In the present invention, a digital supplement is identified based on image content and a location of a client computing device.

[0135] exist Fig. 11C, user interface screen 1100c is shown. User interface screen 1100c includes an image display panel 1108 and an information panel 1130. In this example, image display panel 1108 is displaying another image captured within a store. Image display panel 1108 also includes an indicator 1144 associated with a vase. The vase displayed on image display panel 1108 is an example of an additional anchor. Information panel 1130 is displaying a digital supplement that includes product information about the vase. Information panel 1130 also includes a coupon indicator 1132 and functionality to redeem a coupon. Redeeming a coupon can include purchasing an item from a website associated with the store at a discounted price. In some embodiments, a coupon code is presented that can be used to secure a discount during checkout. As shown in Fig.11A and 11B In the present invention, a digital supplement is identified based on image content and a location of a client computing device.

[0136] Figures 12A-12C is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device 102 during various visual content searches. In this example, the visual content search is based on an image of a movie poster (eg, an image that might be captured at a movie theater).

[0137] exist Fig. 12A , user interface screen 1200a is shown. User interface screen 1200a includes image display panel 1208. In this example, image display panel 1208 is displaying an image of a movie poster. Image display panel 1208 also includes indicator 1240 associated with a movie poster identified in the image. A movie poster is an example of a supplement anchor. Indicator 1240 may include a user-actuatable control that, when actuated, will display a digital supplement or a menu to select a digital supplement.

[0138] exist Fig. 12B , user interface screen 1200b is shown. Image display panel 1208 also includes a preview digital supplement 1242 associated with a movie poster identified in the image. For example, the indicator 1240 ( Fig. 12A ) is followed by actuation of the preview digital supplement 1242. The preview digital supplement 1242 may overlay an image or video from a movie associated with the identified movie poster on the image of the movie poster.

[0139] exist Fig. 12C, user interface screen 1200c is shown. Image display panel 1208 also includes rating indicator 1244 and rating indicator 1246. Rating indicator 1244 and rating indicator 1246 may be generated by one or more digital supplements in response to a visual content query including a movie poster. The digital supplement may, for example, overlay rating information for a movie associated with a movie poster in the image. Rating indicator 1244 and rating indicator 1246 may include user-actuable controls that, when actuated, cause additional information about the rating and associated movie to be displayed.

[0140] Fig.13 An example of a computer device 1300 and a mobile computer device 1350 that can be used with the techniques described herein (to implement the client computing device 102, the search server 152, and the digital supplement server 172) is shown. The computing device 1300 includes a processor 1302, a memory 1304, a storage device 1306, a high-speed interface 1308 connected to the memory 1304 and a high-speed expansion port 1310, and a low-speed interface 1312 connected to a low-speed bus 1314 and the storage device 1306. Each of the components 1302, 1304, 1306, 1308, 1310, and 1312 is interconnected using various buses and can be mounted on a common motherboard or otherwise as appropriate. The processor 1302 can process instructions executed within the computing device 1300, including instructions stored in the memory 1304 or on the storage device 1306 to display graphical information for a GUI on an external input / output device (such as a display 1316 coupled to the high-speed interface 1308). In other embodiments, multiple processors and / or multiple buses and multiple memories and types of memories may be used as appropriate. In addition, multiple computing devices 1300 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

[0141] Memory 1304 stores information within computing device 1300. In one implementation, memory 1304 is one or more volatile memory units. In another implementation, memory 1304 is one or more non-volatile memory units. Memory 1304 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0142] Storage device 1306 can provide mass storage for computing device 1300. In one embodiment, storage device 1306 can be or include a computer readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a magnetic tape device, a flash memory or other similar solid-state memory device or array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. A computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as memory 1304, storage device 1306, or memory on processor 1302.

[0143] The high-speed controller 1308 manages bandwidth-intensive operations for the computing device 1300, while the low-speed controller 1312 manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one embodiment, the high-speed controller 1308 is coupled to the memory 1304, the display 1316 (e.g., through a graphics processor or accelerator), and to the high-speed expansion port 1310, which can accept various expansion cards (not shown). In this embodiment, the low-speed controller 1312 is coupled to the storage device 1306 and the low-speed expansion port 1314. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), is coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or a router, for example, through a network adapter.

[0144] As shown in the figure, computing device 1300 can be implemented in many different forms. For example, it can be implemented as a standard server 1320, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 1324. In addition, it can be implemented in a personal computer such as a laptop computer 1322. Alternatively, components from computing device 1300 can be combined with other components in a mobile device (not shown) such as device 1350. Each of such devices can contain one or more of computing devices 1300, 1350, and the entire system can be composed of multiple computing devices 1300, 1350 that communicate with each other.

[0145] Computing device 1350 includes a processor 1352, a memory 1364, an input / output device such as a display 1354, a communication interface 1366, and a transceiver 1368, among other components. Device 1350 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of components 1350, 1352, 1364, 1354, 1366, and 1368 are interconnected using various buses, and several components may be mounted on a common motherboard or otherwise as appropriate.

[0146] Processor 1352 may execute instructions within computing device 1350, including instructions stored in memory 1364. The processor may be implemented as a chipset of chips including separate and multiple analog and digital processors. For example, the processor may provide coordination for other components of device 1350, such as control of a user interface, applications executed by device 1350, and wireless communications performed by device 1350.

[0147] The processor 1352 can communicate with the user through a control interface 1358 and a display interface 1356 coupled to the display 1354. The display 1354 can be, for example, a TFT LCD (thin film transistor liquid crystal display) and an LED (light emitting diode) or OLED (organic light emitting diode) display or other appropriate display technology. The display interface 1356 may include appropriate circuits for driving the display 1354 to present graphics and other information to the user. The control interface 1358 may receive commands from the user and convert them for submission to the processor 1352. In addition, an external interface 1362 in communication with the processor 1352 may be provided to enable near-area communication of the device 1350 with other devices. The external interface 1362 may, for example, be provided for wired communication in some embodiments, or for wireless communication in other embodiments, and multiple interfaces may also be used.

[0148] The memory 1364 stores information within the computing device 1350. The memory 1364 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 1374 may also be provided and connected to the device 1350 via an expansion interface 1372, which may include, for example, a SIMM (Single In-line Memory Module) card interface. Such an expansion memory 1374 may provide additional storage space for the device 1350, or may also store applications or other information for the device 1350. Specifically, the expansion memory 1374 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Therefore, for example, the expansion memory 1374 may be provided as a security module for the device 1350, and may be programmed with instructions that allow the device 1350 to be used securely. In addition, security applications and additional information may be provided via a SIMM card, such as placing identification information on a SIMM card in an unbreakable manner.

[0149] As discussed below, the memory may include, for example, flash memory and / or NVRAM memory. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as memory 1364, expansion memory 1374, or memory on processor 1352, which may be received, for example, via transceiver 1368 or external interface 1362.

[0150] Device 1350 can communicate wirelessly through communication interface 1366, which may include digital signal processing circuitry when necessary. Communication interface 1366 may provide for communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc. Such communication may occur, for example, through radio frequency transceiver 1368. In addition, short-range communication may occur, such as using Bluetooth, Wi-Fi or other such transceivers (not shown). In addition, GPS (global positioning system) receiver module 1370 may provide additional navigation and location-related wireless data to device 1350, which may be used as appropriate by applications running on device 1350.

[0151] Device 1350 may also communicate audibly using audio codec 1360, which may receive spoken information from a user and convert it into usable digital information. Audio codec 1360 may likewise generate audible sounds for the user, such as through a speaker, for example, in a headset of device 1350. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 1350.

[0152] As shown in the figure, computing device 1350 can be implemented in many different forms. For example, it can be implemented as a cellular phone 1380. It can also be implemented as part of a smart phone 1382, a personal digital assistant, or other similar mobile device.

[0153] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor, which can be special purpose or general purpose, coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0154] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms "machine-readable medium," "computer-readable medium," refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0155] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (LED (light emitting diode), or OLED (organic LED), or LCD (liquid crystal display) monitor / screen) to display information to the user, and having a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including sound, voice, or tactile input.

[0156] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0157] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between a client and a server is generated by computer programs running on the respective computers and having a client-server relationship to each other.

[0158] In some embodiments, Fig.13 The computing device depicted in FIG. 10 may include sensors that interface with an AR headset / HMD device 1390 to generate an augmented environment for viewing inserted content within a physical space. For example, including Fig.13One or more sensors on the computing device 1350 or other computing device depicted in the AR headset 1390 can provide input to the AR headset 1390, or generally, can provide input to the AR space. Sensors can include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 1350 can use sensors to determine the absolute position and / or detected rotation of the computing device in the AR space, which can then be used as input to the AR space. For example, the computing device 1350 can be incorporated into the AR space as a virtual object such as a controller, a laser pointer, a keyboard, a weapon, etc. When incorporated into the AR space, the positioning of the computing device / virtual object by the user can allow the user to position the computing device so as to view the virtual object in the AR space in a certain way. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device like an actual laser pointer. The user can move the computing device left and right, up and down, in a circle, etc. and use the device in a similar manner to using a laser pointer. In some embodiments, the user can aim at a target location using a virtual laser pointer.

[0159] In some implementations, one or more input devices included in or connected to the computing device 1350 can be used as input to the AR space. The input devices can include, but are not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, headphones or earbuds with input capabilities, a game controller, or other connectable input devices. When the computing device is incorporated into the AR space, a user interacting with the input devices included on the computing device 1350 causes specific actions to occur in the AR space.

[0160] In some implementations, the touch screen of computing device 1350 may be rendered as a touch pad in the AR space. A user may interact with the touch screen of computing device 1350. For example, in AR headset 1390, the interactions are rendered as movements on a touch pad rendered in the AR space. The rendered movements may control virtual objects in the AR space.

[0161] In some embodiments, one or more output devices included on the computing device 1350 can provide output and / or feedback to the user of the AR headset 1390 in the AR space. The output and feedback can be visual, tactile, or audible. The output and / or feedback can include, but are not limited to, vibration, turning on, off, or flashing and / or flashing one or more lights or strobe lights, sounding an alarm, playing a ringtone, playing a song, and playing an audio file. The output device can include, but is not limited to, a vibration motor, a vibration coil, a piezoelectric device, an electrostatic device, a light emitting diode (LED), a strobe light, and a speaker.

[0162] In some embodiments, computing device 1350 may appear as another object in a computer-generated 3D environment. User interactions with computing device 1350 (e.g., rotating, shaking, touching a touch screen, sliding a finger on a touch screen) may be interpreted as interactions with objects in the AR space. In the example of a laser pointer in the AR space, computing device 1350 appears as a virtual laser pointer in the computer-generated 3D environment. As the user manipulates computing device 1350, the user in the AR space sees the movement of the laser pointer. The user receives feedback from the interaction with computing device 1350 in the AR environment on computing device 1350 or AR headset 1390. The user's interaction with the computing device may be transformed into interaction with a user interface generated for a controllable device in the AR environment.

[0163] In some implementations, computing device 1350 may include a touch screen. For example, a user may interact with the touch screen to interact with a user interface of a controllable device. For example, the touch screen may include user interface elements, such as sliders that may control properties of a controllable device.

[0164] Computing device 1300 is intended to represent various forms of digital computers and devices, including but not limited to laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 1350 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are exemplary only and are not meant to limit the embodiments of the inventions described and / or claimed herein.

[0165] A number of embodiments have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the specification.

[0166] In addition, the logic flows depicted in the accompanying drawings do not require the particular order shown, or the sequential order to achieve the desired results. In addition, other steps can be provided or steps can be removed from the described flow, and other components can be added to or removed from the described system. Therefore, other embodiments are within the scope of the appended claims.

[0167] Although certain features of the described embodiments have been described as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the embodiments. It should be understood that they are given only in an exemplary and non-limiting manner, and various changes in form and detail may be made. Except for mutually exclusive combinations, any part of the apparatus and / or method described herein may be combined in any combination. The embodiments described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different embodiments described.

Claims

1. A computer-implemented method comprising: Receiving image data by a search server; identifying at least one entity within the image data; storing in a database of the search server an instance of a data structure including the at least one entity identified within the image data; receiving a visual content query from a client computing device; detecting the at least one entity within the visual content query; detecting contextual information associated with the at least one entity identified within the visual content query; searching the database for the at least one entity detected within the visual content query; matching, by the search server, the at least one entity detected within the visual content query with the data structure instance comprising the at least one entity; identifying at least one digital supplement associated with the at least one entity based on the contextual information; as well as In response to the visual content query and the match, supplemental information including the at least one digital supplement associated with the at least one entity is transmitted by the search server to the client computing device.

2. The computer-implemented method of claim 1 , wherein: Transmitting the supplemental information associated with the at least one entity comprises transmitting at least one of: identification information associated with the at least one entity; location information associated with the at least one entity; one or more applications associated with the at least one entity; or One or more network-accessible resources associated with the at least one entity.

3. The computer-implemented method of claim 2, wherein: The supplemental information includes name, description, image and uniform resource locator.

4. The computer-implemented method of claim 1 , wherein: Storing the data structure instance includes storing the data structure instance including a digital supplement associated with the at least one entity identified within the image data.

5. The computer-implemented method of claim 4, wherein: Transmitting the supplemental information associated with the at least one entity to the client computing device includes transmitting information associated with a plurality of network-accessible resources to the client computing device.

6. The computer-implemented method of claim 4, wherein: Storing the data structure instance includes storing the data structure instance in the database including multiple other data structure instances, each of the multiple other data structure instances including at least one previously identified entity and at least one corresponding digital supplement associated with the corresponding at least one previously identified entity.

7. The computer-implemented method of claim 6, wherein: Transmitting the supplementary information includes: identifying a plurality of digital supplements including the at least one digital supplement associated with the at least one entity identified in the visual content query; determining a relevance score for each digital supplement in the plurality of digital supplements; and An ordered list of digital supplements is transmitted to the client computing device based on the relevance score of each digital supplement in the plurality of digital supplements.

8. The computer-implemented method of claim 7, wherein: Determining the relevance score includes determining the relevance score for each of the plurality of digital supplements associated with a corresponding plurality of data structure instances based on the context information.

9. The computer-implemented method of claim 8, wherein: Each of the plurality of data structure instances specifies context information, and wherein the matching comprises matching the context information associated with the at least one entity identified within the visual content query with the context information included in the plurality of data structure instances.

10. The computer-implemented method of claim 6, wherein: Transmitting said digital supplement comprises: A list of digital supplements is transmitted, the list including the digital supplement from the data structure instance associated with the at least one entity and a digital supplement from another data structure instance of the plurality of data structure instances.

11. A non-transitory computer-readable medium comprising instructions that, when executed by a processor of a computing system, cause the computing system to: Storing a plurality of data structure instances in a database of the computing system includes: receiving image data; identifying at least one entity within the image data; as well as storing a data structure instance in the database, the data structure instance including the at least one entity identified within the image data and supplemental information associated with the at least one entity, the database including a plurality of data structure instances; receiving a visual content query from a client computing device; detecting at least one entity within the visual content query; detecting contextual information associated with the at least one entity identified within the visual content query; searching the database for the at least one entity detected within the visual content query; matching the at least one entity detected within the visual content query to at least one entity included in one or more data structure instances among the plurality of data structure instances; identifying at least one digital supplement associated with the at least one entity based on the contextual information; as well as In response to the visual content query and the matching with the one or more of the plurality of data structure instances, the supplemental information including the at least one digital supplement associated with the at least one entity detected within the visual content query is transmitted.

12. The non-transitory computer readable medium of claim 11, wherein: The instructions cause the computing system to transmit the supplemental information to the client computing device to include at least one of: identification information associated with the at least one entity; location information associated with the at least one entity; application information associated with the at least one entity; or A network-accessible resource associated with the at least one entity detected within the visual content query.

13. The non-transitory computer readable medium of claim 11, wherein: The instructions cause the computing system to transmit the supplemental information to the client computing device to include a name, a description, an image, and a uniform resource locator associated with the at least one entity detected within the visual content query.

14. The non-transitory computer readable medium of claim 11, wherein: Each data structure instance includes supplemental information, which includes a corresponding digital supplement associated with at least one entity identified within the image data, and wherein the instructions cause the computing system to transmit the corresponding digital supplement included in the data structure instance that matches the at least one entity identified within the visual content query to the client computing device.

15. The non-transitory computer readable medium of claim 14, wherein: The instructions cause the computing system to: identifying a plurality of digital supplements including the at least one digital supplement associated with the at least one entity identified within the visual content query; determining a relevance score for each digital supplement in the plurality of digital supplements; as well as The ordered list of digital supplements is transmitted to the client computing device.

16. The non-transitory computer readable medium of claim 15, wherein: The instructions cause the computing system to determine the relevance score for each of the plurality of digital supplements based on the contextual information.

17. The non-transitory computer readable medium of claim 16, wherein: Each digital structure instance in the plurality of data structure instances specifies context information, and wherein the instructions cause the computing system to match the context information associated with the at least one entity identified within the visual content query with the context information included in the plurality of data structure instances.

18. The non-transitory computer readable medium of claim 16, wherein: The instructions cause the computing system to transmit a digital supplement list including the digital supplement from the data structure instance associated with the at least one entity and a digital supplement from another data structure instance of the plurality of data structure instances.

19. A computer-implemented method comprising: receiving, by the server, a visual content query from a client device, the visual content query comprising image data; detecting an entity within the image data; identifying a plurality of digital supplements associated with the entity detected within the image data; assigning a relevance score to each of the plurality of digital supplements based on the entity detected within the image data and at least one of a location associated with the client device, user profile data, or specifications associated with the client device; ranking the plurality of digital supplements based on the relevance score assigned to each of the plurality of digital supplements; generating a numerically supplemented ordered list based on the ranking; and In response to the visual content query, the ordered list of digital supplements is transmitted to the client device.

20. The computer-implemented method of claim 19, wherein: At least one digital supplement of the plurality of digital supplements includes a three-dimensional model corresponding to the entity detected within the image data.

21. The computer-implemented method of claim 20, wherein: Assigning the relevance score to each of the plurality of digital supplements includes assigning a higher relevance score to the at least one of the plurality of digital supplements that includes the three-dimensional model.

22. The computer-implemented method of claim 21, wherein: Assigning the higher relevance score includes assigning the higher relevance score to the at least one digital supplement of the plurality of digital supplements that includes the three-dimensional model based on the specification associated with the client device.

23. The computer-implemented method of claim 19, wherein: The detected entities include: detecting a plurality of entities within the image data; identifying a digital supplement associated with each of the plurality of entities; and A target entity is identified from the plurality of entities based on a cumulative relevance score of the plurality of numeric supplements associated with each entity of the plurality of entities.

24. The computer-implemented method of claim 23, wherein Ranking the plurality of digital supplements includes ranking the plurality of digital supplements associated with the target entity based on the relevance score assigned to each of the plurality of digital supplements associated with the target entity; and Generating the ordered list of digital supplements includes generating the ordered list of digital supplements based on the ranking of the plurality of digital supplements associated with the target entity.

25. A computer-implemented method comprising: executing the application by a processor of the computing device; capturing image data by an image sensor of the computing device; transmitting a query including the image data to a search server; receiving, by the application from the search server, a response to the query, the response identifying a digital supplement, the digital supplement comprising information related to a document identified in the image data based on at least one indicator detected in the image data; as well as A user interface is output by the application, the user interface including the information included in the digital supplement associated with the image data.

26. The computer-implemented method of claim 25, wherein: Outputting the user interface includes at least one of the following: outputting a first digital complement of a first user control, and in response to detecting selection of the first user control, triggering output of an annotation displayed as an overlay on the document captured in the image data, the annotation representing a first numerical calculation performed on a digital element detected from the image data; outputting a second digital complement including a second user control, and in response to detecting selection of the second user control, triggering outputting at least one request to at least one other user based on a second digital calculation performed on the digital element detected from the image data; outputting a third digital complement including a third user control, and in response to detecting selection of the third user control, triggering saving of the document to a specified location; or The output includes a fourth digital complement of a fourth user control, and in response to detecting a selection of the fourth user control, triggering submission of the at least one document to the designated recipient.